Correlate action code cache with master account + metadata API (CPL-351)

[GTC6244]

Implements CPL-351 by adding a CacheMetadataIndex secondary lookup alongside the IPFS action-code cache that correlates each cached entry to the master account (account wallet address) that executed it, tracking size, created/last-run timestamps, and run count. Recording happens best-effort inside lit_action so a failed on-chain wallet lookup never fails execution, and a moka eviction_listener keeps the index consistent when binaries are evicted, replaced, or removed. A new GET /core/v1/cache_metadata endpoint returns this metadata scoped to the caller's account — never the cached code/binaries themselves. Includes 7 unit tests for the index (recording, shared entries, eviction cleanup) and a regenerated k6/litApiServer.ts for the new route.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a secondary in-memory index in lit-api-server to correlate cached Lit Action code (keyed by IPFS id) with the caller’s master account, and exposes a new /core/v1/cache_metadata endpoint to retrieve cache metadata scoped to the authenticated account.

Changes:

• Introduces CacheMetadataIndex and wires it into Lit Action execution + IPFS cache eviction flow.
• Adds GET /cache_metadata endpoint and response models to return per-account cache metadata (no binaries).
• Regenerates k6/litApiServer.ts client types/route for the new endpoint.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
lit-api-server/src/main.rs	Creates and manages CacheMetadataIndex; attaches moka eviction listener to keep index consistent.
lit-api-server/src/core/v1/models/response.rs	Adds CacheMetadataResponse and CacheEntryMetadataItem API response models.
lit-api-server/src/core/v1/endpoints/mod.rs	Registers the new get_cache_metadata route in the v1 route list/spec.
lit-api-server/src/core/v1/endpoints/configuration.rs	Implements GET /cache_metadata endpoint (authenticated via ApiKey).
lit-api-server/src/core/v1/endpoints/actions.rs	Passes cache-metadata state into Lit Action execution path.
lit-api-server/src/core/mod.rs	Exposes the new cache_metadata module.
lit-api-server/src/core/core_features.rs	Records executions into the metadata index; adds core handler for returning per-account metadata.
lit-api-server/src/core/cache_metadata.rs	New index implementation + unit tests for recording/lookup/eviction cleanup.
k6/litApiServer.ts	Updates generated k6 client types and method for /cache_metadata.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[clawdbot-glitch003]

🤖 Code review (Claude)

This PR adds a CacheMetadataIndex (RwLock<HashMap> secondary index, Rocket-managed, shared across the restart loop) that correlates cached action-code entries with the master account wallet address that executed them, records executions best-effort inside lit_action, keeps the index consistent via a moka eviction_listener on the 1 GB ipfs_cache, and exposes it through a new authenticated GET /core/v1/cache_metadata endpoint (spec.json + k6 client regenerated). It also rewords a cargo-deny comment in rust-ci.yml.

Findings

1. RemovalCause::Replaced wipes live metadata — run_count/created_at are effectively always reset (correctness, major)

lit_action inserts into ipfs_cache unconditionally on every request, including when the entry is already cached (lit-api-server/src/core/core_features.rs:64-66, pre-existing). With the new listener in lit-api-server/src/main.rs:716-718, every such re-insert fires the eviction listener with RemovalCause::Replaced, and the listener calls remove_entry(&key) regardless of cause — deleting metadata for a key that is still in the cache.

Per re-run of an already-cached action the sequence is: insert → Replaced notification (immediate or queued, depending on moka's delivery) → remove_entry(ipfs_id) → record_execution(ipfs_id, ...) recreates the entry from scratch. Depending on notification timing you get either (a) run_count pinned at 1 and created_at == last_run_at forever, or (b) entries intermittently vanishing from GET /cache_metadata between runs. Either way the three fields the ticket exists to track (created/last-run/run-count) are wrong for any action run more than once. The unit tests don't catch this because they exercise the index directly, never through moka.

Fix: early-return in the listener when cause == RemovalCause::Replaced (the key still exists), and/or only insert into the cache on a miss (e.g. contains_key check or entry().or_insert_with). Worth an integration test that drives record_execution through an actual moka cache with the listener wired up.

2. Uncached on-chain RPC added to the hot action-execution path (performance)

crate::accounts::get_account_wallet_address (lit-api-server/src/accounts/mod.rs:799) does a raw getAccountWalletAddress contract call with no caching — unlike can_execute_action, which goes through blockchain_cache::try_get_with. The new call at lit-api-server/src/core/core_features.rs:66-77 is .awaited inline before the action client is built, so every POST /lit_action now pays an extra RPC round-trip (and RPC-provider load) purely for metadata bookkeeping. "Best-effort" here covers errors but not latency. Either route it through blockchain_cache (add a key-hash → wallet-address cache alongside wallet_derivation) or tokio::spawn the recording so it's off the request path.

3. Orphaned metadata when eviction races the recording (minor, related to #1)

record_execution runs after the cache insert; if a capacity eviction of that same key lands between the two (or the queued notification for it has already been drained), the metadata entry is recreated after its removal notification and then never cleaned up — remove_entry for it has already fired as a no-op. Bounded leak (one stale entry per raced key) but worth a comment or a periodic reconciliation against ipfs_cache.contains_key.

4. Invalid API key returns 400, not 401 (minor inconsistency)

get_cache_metadata maps a failed wallet resolution to ApiStatus::bad_request (lit-api-server/src/core/core_features.rs:496-501). The billing path deliberately types this as UnknownApiKey → 401 (lit-api-server/src/accounts/mod.rs:817-833, "so billing guards and endpoints can map it to 401 Unauthorized"). An unknown key on an authenticated read endpoint should follow the same convention.

Nits

• CacheEntrySnapshot.account_count (lit-api-server/src/core/cache_metadata.rs:227-229) is computed and asserted in tests but never surfaced in the API response — fine for privacy (it would leak how many other accounts run the same code), but if it's intentionally internal-only a comment saying so would help; otherwise drop it.
• entry_count duplicates entries.len() in the response — harmless, but clients can derive it.
• The .github/workflows/rust-ci.yml hunk is a pure comment reword with no behavior change — unrelated churn in a PR about cache metadata; consider dropping it or landing separately.
• ttl_seconds is hardwired to None with the reason encoded in three places (model doc, spec, mapping comment); fine, but if a TTL'd cache never materializes this field is dead weight in the API contract.

Verdict

Needs changes — finding #1 breaks the core deliverable (run counts / timestamps are reset on every re-execution or entries vanish entirely), and #2 adds an uncached RPC to the hottest path in the server. Both have small, contained fixes.

[GTC6244]

Responses to the Claude review — addressed in 7daccd78

Thanks for the review. All four findings addressed; details and one conscious deferral below.

#1 RemovalCause::Replaced wipes live metadata (major) — FIXED. Confirmed real: lit_action re-inserts on every request, and moka fires the listener with Replaced (which its own docs define as "the entry itself was not actually removed"), so my listener was deleting metadata for still-cached keys, resetting run_count/created_at. The listener now skips Replaced and only drops metadata on a genuine removal (Size/Expired/Explicit). record_execution's or_insert_with already preserved created_at and accumulated run_count, so the guard is the whole fix. Added a regression test (replaced_preserves_metadata_but_real_removal_drops_it) that drives the index through a real moka cache wired with the same listener guard — it asserts run_count == 2 and a preserved created_at across a re-insert, and would fail without the guard.

#2 Uncached on-chain RPC on the hot path — FIXED (latency). Moved the recording — including the uncached get_account_wallet_address call — into a tokio::spawn, so POST /lit_action no longer pays that round-trip in its response latency. I chose spawn over threading a new cache through blockchain_cache because this is best-effort bookkeeping and spawn is the contained fix. Note this removes the latency regression but not RPC volume; if provider load ever matters, the follow-up is a key-hash→wallet cache alongside wallet_derivation (happy to do it if you'd prefer, or to switch correlation to the local api_key_hash for zero RPC — the tradeoff is losing master-account rollup across sibling usage keys).

#3 Orphaned metadata on eviction race (minor) — DOCUMENTED, not coded around. With #1 fixed this is a bounded leak (one stale entry per raced key) and practically unreachable: the key was just inserted, so it's the MRU entry and can't be size-evicted unless it alone exceeds the 1 GB cap (impossible under max_code_length). I added a comment on record_execution rather than build periodic reconciliation, which would be over-engineering for an unreachable case.

#4 Invalid key → 400 instead of 401 (minor) — FIXED. get_cache_metadata now maps an unregistered key to 401 (matching the UnknownApiKey billing convention) and a genuine RPC/contract failure to 500, instead of a blanket 400.

Nits:

• account_count — kept, now with a doc comment stating it's internal-only and deliberately not in the API response (exposing how many other accounts run the same code would leak cross-account usage). It backs a correlation invariant in tests.
• entry_count — kept as a client convenience; harmless.
• rust-ci.yml churn — reverted; the file is now byte-identical to main (which already carries the same cargo-deny 0.20 fix after the re-merge).
• ttl_seconds: None — kept intentionally; it honors the ticket's "TTL, last run time, etc." field list and documents that this cache is capacity-bounded rather than time-expired.

Verification: build clean, clippy --all-features -- -D warnings clean, fmt --check clean, 8/8 cache_metadata tests pass. Response schema is unchanged, so spec.json/k6 client are unaffected.