From 7f983949653394545f21f87b8beb3053152de5d7 Mon Sep 17 00:00:00 2001 From: GTC6244 <95836911+GTC6244@users.noreply.github.com> Date: Fri, 10 Jul 2026 14:51:33 -0400 Subject: [PATCH 1/4] feat(cpl-267): document sovereign-mode cache staleness + listener-lag banner Each API-server instance serves reads from a per-instance cache refreshed by an event listener polling the chain ~every 10s, so after an on-chain write an instance didn't originate (notably ChainSecured wallet writes) its reads can be stale for up to one poll interval. Document this read-after-write window in the SDK sovereign-mode reference, the core_sdk.js header, and the dashboard user docs. Expose the listener's lag via a new account_event_listener_lag_seconds field on GET /health, and add a dashboard banner that warns when a connected instance reports more than 30s of lag. Co-Authored-By: Claude Opus 4.8 --- docs/management/account_modes.mdx | 49 +++++++++++++++ docs/management/dashboard.mdx | 23 +++++++ lit-api-server/src/account_events.rs | 73 +++++++++++++++++++++- lit-api-server/src/core/v1/health.rs | 38 ++++++++++- lit-static/core_sdk.js | 12 ++++ lit-static/dapps/dashboard/app.js | 4 ++ lit-static/dapps/dashboard/index.html | 4 ++ lit-static/dapps/dashboard/listener_lag.js | 71 +++++++++++++++++++++ lit-static/dapps/dashboard/styles.css | 16 +++++ 9 files changed, 287 insertions(+), 3 deletions(-) create mode 100644 lit-static/dapps/dashboard/listener_lag.js diff --git a/docs/management/account_modes.mdx b/docs/management/account_modes.mdx index e9064fc1..58f83a97 100644 --- a/docs/management/account_modes.mdx +++ b/docs/management/account_modes.mdx @@ -164,6 +164,55 @@ Base RPC works, e.g. `https://mainnet.base.org` or prefers the wallet's RPC, so this fallback only matters for the brief window before `connectSigner(signer)` runs. +## Read-after-write staleness + +Each API-server instance keeps a **per-instance cache** of on-chain +account and permission state. An event listener refreshes that cache by +polling the chain for account-mutation events roughly every **10 seconds** +(the poll interval). The node was deliberately built with a per-instance +listener rather than a single leader, so behind a load balancer there are +N independent views of chain state. + +The practical consequence: + +> After an on-chain write that a given instance did not originate, reads +> served by that instance may show stale data for up to ~10 seconds (one +> polling interval) until its listener catches up. Behind a load balancer, +> consecutive reads can land on different instances and briefly disagree. + +This window is reachable whenever a write happens **outside** the instance +serving your reads — most commonly: + +- A **ChainSecured (sovereign) wallet write** (the contract emits the event; + no API server originated it, so every instance learns of it only on its + next poll). +- A write made through a **different API-server instance** behind the load + balancer. +- A transaction sent **directly to the contract**. + +Which reads are affected: + +- **API mode** reads (`listApiKeys`, `listGroups`, `listWallets`, + `listActions`, ...) go through the server cache and are subject to this + window. +- **Sovereign-mode** reads call the contract directly via your RPC/signer + (no server round-trip) and reflect chain head immediately — they are + **not** subject to the cache window. + +If you need read-your-write certainty after a sovereign write that other +clients will read in API mode, either wait out one poll interval (~10s) or +read the contract directly. + + +A healthy instance keeps its lag at or below the poll interval. The +`GET /health` endpoint reports `account_event_listener_lag_seconds` +(unauthenticated; `null` until the listener's first poll). A value that +stays well above ~10s means that instance's listener is stalled or dead and +its cached reads may be substantially out of date until it recovers or the +process restarts. The Chipotle Dashboard polls this and shows a staleness +banner when an instance reports more than 30s of lag. + + ## Using the dashboard The Chipotle Dashboard offers both modes side-by-side on the login screen: diff --git a/docs/management/dashboard.mdx b/docs/management/dashboard.mdx index 04f676a2..8550bcac 100644 --- a/docs/management/dashboard.mdx +++ b/docs/management/dashboard.mdx @@ -74,6 +74,29 @@ async function main({ pkpId }) { ``` Choose the API key (account or usage key) to use for the request, then click **Execute**. The node runs the action and returns signatures, response, and logs. The key you use must be allowed to run that action (via the group and IPFS CID configuration). +## When data looks out of date + +The dashboard reads your account configuration (keys, groups, wallets, +actions) from the API server, which serves those reads from a short-lived +cache. That cache is refreshed by a listener that polls the chain about +every **10 seconds**. + +So right after a change lands on-chain — especially a **ChainSecured +(wallet-signed) write**, or a change made from another device or directly +against the contract — the dashboard can briefly show the previous state. +It catches up on its own within a few seconds; a manual refresh after a +moment will show the latest state. + + +If a server is having trouble keeping up, the dashboard shows an amber +banner near the top of the page: *"This server is catching up to recent +on-chain changes..."* It means reads may be more than 30 seconds out of +date. The banner clears automatically once the server catches up — no +action is needed on your part. Avoid making rapid back-to-back +configuration changes while it's visible, since the table you're looking at +may not yet reflect your last change. + + ## Daily Usage The dashboard is just your human-friendly configuration tool. Once your account and keys are set up to your liking, you can simply call the lit-action endpoint with your usage key each time you, your dApp or cron job needs to execute a lit action. So the only daily use step is diff --git a/lit-api-server/src/account_events.rs b/lit-api-server/src/account_events.rs index ad681325..89e977d7 100644 --- a/lit-api-server/src/account_events.rs +++ b/lit-api-server/src/account_events.rs @@ -20,10 +20,11 @@ use alloy::providers::Provider; use alloy::rpc::types::Filter; use alloy::sol_types::SolEvent; use std::collections::HashSet; -use std::time::Duration; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; /// Polling interval for checking new account-mutation events. -const EVENT_POLL_INTERVAL: Duration = Duration::from_secs(10); +pub const EVENT_POLL_INTERVAL: Duration = Duration::from_secs(10); /// Maximum number of consecutive startup failures before giving up. const MAX_LISTENER_RETRIES: u32 = 5; @@ -36,6 +37,44 @@ const DEAD_WARN_INTERVAL: Duration = Duration::from_secs(300); /// listener is invisible after its single startup-failure error scrolls away. const LISTENER_UP_GAUGE: &str = "account_event_listener.up"; +/// Wall-clock unix-seconds at which the listener last confirmed it was current +/// with the chain head (either after processing a batch of logs, or after +/// observing that no new blocks had been produced). `0` means it has not yet +/// completed a poll — i.e. the listener never started or is still on its first +/// iteration. Updated with [`Ordering::Relaxed`]: this is a monotonic-ish +/// liveness timestamp, not a synchronization point, and a few hundred +/// milliseconds of staleness in the reported lag is immaterial. +static LAST_CAUGHT_UP_UNIX_SECS: AtomicU64 = AtomicU64::new(0); + +fn now_unix_secs() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0) +} + +/// Record that the listener is current with the chain head as of now. +fn mark_caught_up() { + LAST_CAUGHT_UP_UNIX_SECS.store(now_unix_secs(), Ordering::Relaxed); +} + +/// Seconds since the account-event listener last confirmed it was current with +/// the chain head, or `None` if it has not yet completed a poll. +/// +/// Healthy values stay at or below [`EVENT_POLL_INTERVAL`] (~10s). A value that +/// climbs well above it means the listener is stalled (RPC errors, a slow +/// `eth_getLogs`) or has died — in which case on-chain account mutations are no +/// longer invalidating the per-instance permission cache, so reads served by +/// this instance may be stale. Exposed (unauthenticated) via `GET /health` as +/// `account_event_listener_lag_seconds` so clients/ops can surface staleness. +pub fn listener_lag_seconds() -> Option { + let last = LAST_CAUGHT_UP_UNIX_SECS.load(Ordering::Relaxed); + if last == 0 { + return None; + } + Some(now_unix_secs().saturating_sub(last)) +} + /// Expand `$mac!(EventType, |e| vec![..account hashes..])` once per `WritesFacet` /// account/permission mutation event. Used to build both the log filter's topic /// set and the decode dispatch from a single source of truth, so the two can @@ -230,6 +269,10 @@ async fn run_event_listener() -> anyhow::Result<()> { "Account event listener started" ); metrics::gauge!(LISTENER_UP_GAUGE).set(1.0); + // We've just read the chain head, so we're current as of now. Seeds the lag + // gauge so `/health` reports a real lag immediately rather than `null` until + // the first poll tick fires one interval later. + mark_caught_up(); let mut last_checked_block = start_block.saturating_sub(1); let mut interval = tokio::time::interval(EVENT_POLL_INTERVAL); @@ -247,6 +290,8 @@ async fn run_event_listener() -> anyhow::Result<()> { }; if latest_block <= last_checked_block { + // No new blocks since the last poll — already current with the head. + mark_caught_up(); continue; } @@ -265,6 +310,8 @@ async fn run_event_listener() -> anyhow::Result<()> { // Only advance once logs are successfully processed; on error we // retry the same range on the next tick. last_checked_block = latest_block; + // Processed everything up to the head we observed this tick. + mark_caught_up(); } Err(e) => { tracing::warn!( @@ -394,6 +441,28 @@ mod tests { assert_eq!(account_hashes_from_log(&removed), vec![account, usage]); } + #[test] + fn listener_lag_reports_none_until_first_poll_then_small() { + // The listener never starts in unit tests, and this static is touched by + // no other test, so we own it here. `0` (never polled) reads as None. + LAST_CAUGHT_UP_UNIX_SECS.store(0, Ordering::Relaxed); + assert_eq!(listener_lag_seconds(), None); + + // After a successful poll the lag is the elapsed wall-clock time, which + // is ~0 immediately after marking. + mark_caught_up(); + let lag = listener_lag_seconds().expect("lag is Some after a poll"); + assert!(lag <= 2, "fresh lag should be near zero, got {lag}"); + + // A stale timestamp surfaces as a large lag rather than wrapping. + LAST_CAUGHT_UP_UNIX_SECS.store(now_unix_secs().saturating_sub(120), Ordering::Relaxed); + let lag = listener_lag_seconds().expect("lag is Some"); + assert!( + lag >= 120, + "stale lag should reflect elapsed time, got {lag}" + ); + } + #[test] fn unknown_event_yields_no_hashes() { // A billing event emitted by the same contract is not a WritesFacet diff --git a/lit-api-server/src/core/v1/health.rs b/lit-api-server/src/core/v1/health.rs index 7b7b99ee..a393f52d 100644 --- a/lit-api-server/src/core/v1/health.rs +++ b/lit-api-server/src/core/v1/health.rs @@ -33,6 +33,14 @@ pub struct HealthResponse { pub lit_actions_reachable: bool, pub cpu_available: bool, pub billing_keys_present: bool, + /// Seconds the on-chain account-event listener is behind the chain head, or + /// `null` if it has not yet completed a poll. Healthy values sit at or below + /// the ~10s poll interval; a large value means this instance's permission + /// cache is no longer being invalidated by on-chain writes and its reads may + /// be stale. Informational only — does NOT affect the health status (a + /// lagging listener still serves traffic, just from a staler cache). Clients + /// (e.g. the dashboard) surface a staleness banner when this exceeds 30s. + pub account_event_listener_lag_seconds: Option, } pub fn routes() -> Vec { @@ -80,8 +88,10 @@ async fn health( let cpu_available = !cpu_monitor.is_overloaded(); let billing_keys_present = stripe_state.is_some(); + let account_event_listener_lag_seconds = crate::account_events::listener_lag_seconds(); - // billing_keys_present is informational only — does NOT affect health status. + // billing_keys_present and account_event_listener_lag_seconds are + // informational only — neither affects health status. let healthy = lit_actions_reachable && cpu_available; let status = if healthy { @@ -96,6 +106,7 @@ async fn health( lit_actions_reachable, cpu_available, billing_keys_present, + account_event_listener_lag_seconds, }), ) } @@ -163,6 +174,31 @@ mod tests { assert!(body.cpu_available); } + #[tokio::test] + async fn health_always_serializes_listener_lag_field() { + // The field is always present in the JSON (as a number or `null`) so + // clients can rely on its key. We assert on the raw body rather than the + // deserialized value because the lag is backed by a process-global the + // account_events unit tests also exercise — asserting a specific value + // here would race with them. (`Option` would silently deserialize a + // missing key as `None`, so a deserialize round-trip can't prove the key + // is emitted.) + let client = Client::tracked(build_rocket(false, None)) + .await + .expect("valid rocket"); + let body = client + .get("/health") + .dispatch() + .await + .into_string() + .await + .expect("body"); + assert!( + body.contains("account_event_listener_lag_seconds"), + "health body must always include the listener-lag key, got: {body}" + ); + } + #[tokio::test] async fn health_reports_lit_actions_unreachable_when_no_socket() { let client = Client::tracked(build_rocket(false, None)) diff --git a/lit-static/core_sdk.js b/lit-static/core_sdk.js index e673a161..bfa82f80 100644 --- a/lit-static/core_sdk.js +++ b/lit-static/core_sdk.js @@ -17,6 +17,18 @@ * `rpcUrl` or a signer that carries its own provider (any ethers v6 signer * with `signer.provider` set). The dashboard typically calls * `getNodeChainConfig()` first and passes the values in. + * + * Read-after-write staleness (API mode): server-mediated reads (e.g. + * `listApiKeys`, `listGroups`, `listWallets`, `listActions`) are served from a + * per-instance cache that an event listener refreshes by polling the chain + * about every 10s. After an on-chain write the server didn't originate — most + * commonly a ChainSecured (sovereign) wallet write, or a write made via another + * API-server instance — reads on a given instance can lag by up to that poll + * interval (~10s). Behind a load balancer, consecutive reads can hit different + * instances and briefly disagree. Sovereign-mode reads go straight to the chain + * via your RPC/signer and are not affected. The `/health` endpoint reports + * `account_event_listener_lag_seconds` if you need to detect a lagging instance. + * See docs/management/account_modes.mdx ("Read-after-write staleness"). */ import { ACCOUNT_CONFIG_VIEW_ABI } from './account_config_view_abi.js'; diff --git a/lit-static/dapps/dashboard/app.js b/lit-static/dapps/dashboard/app.js index 9d4fef96..8f2fdd1c 100644 --- a/lit-static/dapps/dashboard/app.js +++ b/lit-static/dapps/dashboard/app.js @@ -13,6 +13,7 @@ import { initActions, loadActions } from './actions.js'; import { initWallets, loadWallets } from './wallets.js'; import { initActionRunner } from './runner.js'; import { initAbout } from './about.js'; +import { startListenerLagMonitor } from './listener_lag.js'; // ----- Preload all tables (with error visibility) ----- @@ -284,6 +285,9 @@ setOnAuthReady(() => { refreshChangeOwnershipVisibility(); updateChainSecuredRpcUrlUI(); handleBillingReturn(); + // Watch the connected server's account-event listener and warn if its cache + // is lagging the chain (reads may be stale). Idempotent across re-auths. + startListenerLagMonitor(); }); // ----- Init ----- diff --git a/lit-static/dapps/dashboard/index.html b/lit-static/dapps/dashboard/index.html index cfc5b202..5d72010a 100644 --- a/lit-static/dapps/dashboard/index.html +++ b/lit-static/dapps/dashboard/index.html @@ -184,6 +184,10 @@

+ + +