From 3ff9e94e566a8c3d73ea78cf89583f7c2c2d0d83 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 7 Jul 2026 18:12:16 -0700 Subject: [PATCH 01/25] Spec: Fastly chunked-config GC to reclaim orphaned chunk entries on re-push --- .../specs/2026-07-07-fastly-chunk-gc.md | 248 ++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md new file mode 100644 index 00000000..9a1f97c1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -0,0 +1,248 @@ +# Fastly chunked-config GC: reclaim orphaned chunk entries on re-push + +## Motivation + +When a Fastly Config Store value exceeds the 8 000-character entry +limit, `config push` splits it into content-addressed chunk entries +plus a root pointer, per the chunked-storage design +(`crates/edgezero-adapter-fastly/src/chunked_config.rs`). Chunk keys +are content-addressed by the envelope hash: + +``` +.__edgezero_chunks.. +``` + +The push path is **upsert-only**. Both the cloud writer +(`crates/edgezero-adapter-fastly/src/cli/push_cloud.rs::write_entries`) +and the local writer +(`crates/edgezero-adapter-fastly/src/cli/push_local.rs::write_entries` +→ `provision_local::write_fastly_local_config_store`) insert-or-update +physical entries and never delete anything. + +Because a config change changes the envelope bytes, it changes +`envelope_sha256`, which changes **every** chunk key. The new push +writes a fresh chunk set and overwrites the root pointer in place, so +reads stay correct — the live pointer only ever references the current +chunk set. But the **previous** generation's chunk entries are left +behind, unreferenced and unreclaimed: + +- **Cloud**: orphan entries linger in the remote Config Store. +- **Local**: orphan keys linger in the `fastly.toml` + `[local_server.config_stores..contents]` table (upserted at + `provision_local.rs:559`; the existing code comment at + `provision_local.rs:528` notes chunks "become unreferenced" but + never deletes them). + +Every chunked re-push leaks one generation of chunks. This spec adds +a **best-effort, post-commit garbage-collection sweep** that deletes +the chunk entries the *previous* root pointer referenced and the *new* +pointer no longer does. + +## Scope + +- Applies to the Fastly adapter's `config push` writeback path only + (Stage 7), both cloud and `--local`. +- Deletes **chunk data entries** only. It NEVER deletes a root pointer + key: the pointer at `` is overwritten in place by the + existing upsert and remains the stable, live entry. +- Does not touch sibling logical keys (e.g. `app_config` vs + `app_config_staging`): the sweep is scoped per root key to the chunk + keys that root's own previous pointer referenced. + +## Terminology + +- **Root key**: the stable logical entry key (e.g. `app_config`). Holds + either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. +- **Previous pointer**: the value stored at the root key *before* this + push overwrites it. +- **Prior chunk keys**: `previous_pointer.chunks[].key` — the exact + chunk entries the previous generation referenced. Empty if the + previous value was a direct envelope, missing, or not a pointer. +- **New keep-set**: the physical keys this push writes for that root — + the new chunk keys plus the root key itself. +- **Orphans**: `prior_chunk_keys − new_keep_set`. In practice this is + all prior chunk keys, because a changed config changes the SHA and + therefore every chunk key; but the set-difference is the correct, + race-safe formulation (a re-push of *identical* bytes re-derives the + same keys and deletes nothing). + +## Invariants (MUST) + +1. **Nothing is deleted until the new root pointer is committed.** The + root pointer write is the atomic cutover: before it lands, the live + pointer still references the prior chunks; after it lands, the prior + chunks are unreachable. Deletes run strictly after the pointer + commits. + + Ordering per root key: + + 1. Write all new chunk entries (new-SHA keys) — upsert + 2. Write the new root pointer — upsert ← **commit point** + 3. Delete `prior_chunk_keys − new_keep_set` — best-effort + +2. **Delete only what the previous pointer referenced.** The delete-set + is derived from `previous_pointer.chunks[].key`, not from a prefix + scan or a store-wide enumeration. This guarantees the sweep can only + ever touch keys that were part of this root's own prior generation. + +3. **A key in the new keep-set is never deleted.** Even though a SHA + change makes overlap empty in practice, the sweep subtracts the + keep-set unconditionally so an identical-bytes re-push is a no-op. + +4. **GC failure never fails or blocks the push.** The push's success + criterion is unchanged: new chunks + new pointer committed. The + sweep is optimistic — any read/parse/delete failure degrades to a + warning folded into the returned status lines; the push still + reports success. A leaked chunk is harmless; a failed push is not. + +## Algorithm + +For each logical `(root_key, envelope_json)` entry in the push: + +``` +# --- before writing --- +prev = read_root_value(root_key) # may be absent +prior_keys = prior_chunk_keys(prev) # [] unless prev is a pointer +expanded = prepare_fastly_config_entries(root_key, envelope_json) +new_keys = { k for (k, _) in expanded } ∪ { root_key } + +# --- write (existing behaviour, unchanged) --- +push expanded # chunks first, root pointer last (commit point) + +# --- sweep (new, best-effort, only after push succeeds) --- +for k in prior_keys − new_keys: + try delete_entry(k) except e: warn("failed to delete orphan chunk `{k}`: {e}") +``` + +`prior_chunk_keys` reuses the existing pointer schema: parse the prior +value as a `FastlyChunkPointer`; if it parses and +`edgezero_kind == POINTER_KIND`, return `chunks[].key`; otherwise +return `[]`. A direct `BlobEnvelope` fails to parse as a pointer and +yields `[]` — correct, since a direct value has no chunks. This means +the **shrink-to-direct** case (config drops back under 8 000 chars) is +handled for free: the new value is direct, `new_keys = { root_key }`, +and every prior chunk key is swept. + +## Cloud path (`push_cloud.rs`) + +- **Read prior value**: reuse `fetch_remote_config_store_entry(store_id, + root_key)` (already present for chunk resolution). Returns + `Ok(Some(raw))`, `Ok(None)` (absent → no prior chunks), or `Err` + (degrade to warning, skip GC for that root). +- **Store id**: `write_entries` already resolves `resolved_id` via + `resolve_remote_config_store_id`. The prior-value read must move to + after that resolution and before the committer loop. Because the read + needs the store id, hoist `resolve_remote_config_store_id` above the + per-root prior-value reads (it currently sits just before the commit + loop — move it up; the dry-run branch returns before it, unchanged). +- **Delete helper**: new `delete_config_store_entry(store_id, key)` + shelling `fastly config-store-entry delete --store-id= + --key=`, mirroring the spawn/stderr handling of + `create_config_store_entry`. Treat a "not found" / "does not exist" / + "404" stderr as success (already gone). +- **Cost**: one extra `describe` per logical root before the push, plus + one `delete` per orphan after. No store-wide `list` call. + +## Local path (`push_local.rs` / `write_fastly_local_config_store`) + +The local `contents` table is held entirely in memory in a single +`DocumentMut`, so GC here is fully reliable and needs no round-trips: + +- Before the upsert loop at `provision_local.rs:559`, read each root + key's current value from `contents_tbl` (the prior pointer), compute + `prior_chunk_keys`, and after inserting the new physical entries, + `contents_tbl.remove(k)` for each orphan `k`. +- Because the whole file is rewritten atomically by the trailing + `fs::write`, steps 1–3 of the ordering invariant collapse into a + single write — there is no partial-commit window locally, so the + local sweep cannot leak or corrupt. +- The signature of `write_fastly_local_config_store` takes only + `entries` today. To compute `prior_chunk_keys` it must inspect the + pre-existing `contents_tbl` for each root key present in `entries`; + this is available in-function (the table is read before the upsert + loop). No new parameters are required — the prior values are read + from the table being edited. + +## Dry-run + +Both paths already have a dry-run branch that reports direct-vs-chunked +intent per key. Extend it: for each root key, report how many orphan +chunks *would* be deleted. The cloud dry-run reads the prior value +(same `fetch`) to count; if that read fails during dry-run, report +"unknown (prior read failed)" rather than erroring. Example lines: + +``` + would push `app_config` as chunked (12 chunks + 1 pointer, 84210B total) + would delete 9 orphan chunks from the previous generation of `app_config` +``` + +## Non-goals + +- **Reclaiming pre-existing leaks.** This sweep only deletes the + generation the *current* live pointer references. Chunks orphaned by + pushes that predate this feature (or by a prior sweep that partially + failed) are not enumerated and not reclaimed. Steady state going + forward: each push cleans the immediately-prior generation, so at most + one stale generation exists between pushes. A one-time full reclaim + (prefix-scan the store and delete unreferenced `__edgezero_chunks` + keys) is deferred to a possible future `config gc` command and is out + of scope here. +- **Transactional multi-key GC.** Each root key is swept independently; + there is no cross-key atomicity beyond what each path already + provides (per-entry for cloud, whole-file for local). + +## Testing + +Reuse the existing fake-`fastly` shim harness in `push_cloud.rs` tests +and the tempdir/`DocumentMut` harness in `provision_local.rs` / +`push_local.rs` tests. + +### `chunked_config.rs` unit tests (`prior_chunk_keys`) + +- Pointer value → returns its `chunks[].key` in order. +- Direct `BlobEnvelope` value → returns `[]`. +- Unparseable / unrelated JSON → returns `[]`. +- Pointer with non-matching `edgezero_kind` → returns `[]`. + +### Cloud (`push_cloud.rs`) + +- **Deletes prior, keeps new**: fake fastly serves a prior pointer for + the root; after push, assert `config-store-entry delete` is invoked + for each prior chunk key and NOT for any new chunk key or the root + key. +- **Shrink-to-direct**: prior value is a chunk pointer, new value is + direct (≤ 8 000 chars); assert all prior chunk keys are deleted and + the root key is upserted (not deleted). +- **First push (no prior)**: prior read returns not-found; assert zero + delete calls. +- **Identical re-push**: prior pointer and new pointer reference the + same keys; assert zero delete calls (keep-set subtraction). +- **Delete failure degrades to warning**: delete shell-out exits + non-zero with a non-"not found" stderr; assert the push still returns + success and the status includes a warning naming the failed chunk key. +- **Prior read failure degrades to warning**: `describe` on the root + errors (non-not-found); assert push succeeds, GC skipped, warning + present. +- **Ordering**: extend the existing argv-log fake to assert every + `delete` argv appears strictly after the root-pointer `update` argv. + +### Local (`push_local.rs` / `provision_local.rs`) + +- **Prunes orphan chunk keys from `contents`**: seed a `fastly.toml` + whose `contents` table holds a prior pointer + its chunk keys; push a + changed (still-chunked) config; assert the new chunk keys are present, + the prior chunk keys are absent, and sibling logical keys are + untouched. +- **Shrink-to-direct locally**: prior chunked → new direct; assert all + prior chunk keys removed and root key holds the direct envelope. +- **Sibling coexistence preserved**: a push of `app_config` must not + remove `app_config_staging` chunk keys (per Spec 12.7 coexistence). + +## Files touched + +| File | Change | +| --- | --- | +| `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(&str) -> Vec` + unit tests | +| `crates/edgezero-adapter-fastly/src/cli/push_cloud.rs` | Hoist store-id resolution; per-root prior read; `delete_config_store_entry`; post-commit sweep; dry-run counts; tests | +| `crates/edgezero-adapter-fastly/src/cli/push_local.rs` | Dry-run orphan counts; wire sweep into local write; tests | +| `crates/edgezero-adapter-fastly/src/cli/provision_local.rs` | `write_fastly_local_config_store` prunes orphan chunk keys after upsert; tests | From 77d65bde0680119e07000170c8c756e6984625f5 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 7 Jul 2026 19:18:32 -0700 Subject: [PATCH 02/25] Spec review v2: fix file map (cli.rs, not the unmerged split), validate+scope prior_chunk_keys, offline cloud dry-run, local root inference, warning semantics, invert stale no-GC test --- .../specs/2026-07-07-fastly-chunk-gc.md | 339 +++++++++++------- 1 file changed, 213 insertions(+), 126 deletions(-) diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 9a1f97c1..0336f65d 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -13,10 +13,9 @@ are content-addressed by the envelope hash: ``` The push path is **upsert-only**. Both the cloud writer -(`crates/edgezero-adapter-fastly/src/cli/push_cloud.rs::write_entries`) -and the local writer -(`crates/edgezero-adapter-fastly/src/cli/push_local.rs::write_entries` -→ `provision_local::write_fastly_local_config_store`) insert-or-update +(`FastlyCliAdapter::push_config_entries`, `cli.rs:349`) and the local +writer (`FastlyCliAdapter::push_config_entries_local`, `cli.rs:421` → +`write_fastly_local_config_store`, `cli.rs:926`) insert-or-update physical entries and never delete anything. Because a config change changes the envelope bytes, it changes @@ -28,16 +27,24 @@ behind, unreferenced and unreclaimed: - **Cloud**: orphan entries linger in the remote Config Store. - **Local**: orphan keys linger in the `fastly.toml` - `[local_server.config_stores..contents]` table (upserted at - `provision_local.rs:559`; the existing code comment at - `provision_local.rs:528` notes chunks "become unreferenced" but - never deletes them). + `[local_server.config_stores..contents]` table (upserted in + `write_fastly_local_config_store` at `cli.rs:926`, which reads the + doc, inserts each physical entry, and never removes stale keys). Every chunked re-push leaks one generation of chunks. This spec adds a **best-effort, post-commit garbage-collection sweep** that deletes the chunk entries the *previous* root pointer referenced and the *new* pointer no longer does. +> **Code-layout note.** This spec targets the layout on `main`, where +> the Fastly adapter is monolithic (`cli.rs` + `chunked_config.rs`). +> An in-flight branch (`feature/provision-local-impl`) splits `cli.rs` +> into `cli/push_cloud.rs`, `cli/push_local.rs`, and +> `cli/provision_local.rs`. If that branch merges first, the same +> changes land in those files instead — the functions and logic are +> identical, only the file boundaries move. Rebase the implementation +> onto whichever layout is current. + ## Scope - Applies to the Fastly adapter's `config push` writeback path only @@ -46,18 +53,24 @@ pointer no longer does. key: the pointer at `` is overwritten in place by the existing upsert and remains the stable, live entry. - Does not touch sibling logical keys (e.g. `app_config` vs - `app_config_staging`): the sweep is scoped per root key to the chunk - keys that root's own previous pointer referenced. + `app_config_staging`): the delete-set is scoped, per root key, to + chunk keys that match that root's own generated chunk prefix AND were + named by that root's previous pointer. ## Terminology - **Root key**: the stable logical entry key (e.g. `app_config`). Holds - either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. + either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. Root + keys are store-config ids and MUST NOT contain the reserved chunk + infix `.__edgezero_chunks.` (they never do — they are validated + identifiers, and `prepare_fastly_config_entries` only ever appends + the infix to form chunk keys). - **Previous pointer**: the value stored at the root key *before* this push overwrites it. -- **Prior chunk keys**: `previous_pointer.chunks[].key` — the exact - chunk entries the previous generation referenced. Empty if the - previous value was a direct envelope, missing, or not a pointer. +- **Prior chunk keys**: the keys named by `previous_pointer.chunks[]`, + *after* validation and prefix filtering (see `prior_chunk_keys`). + Empty if the previous value was a direct envelope, missing, or not a + valid v1 pointer. - **New keep-set**: the physical keys this push writes for that root — the new chunk keys plus the root key itself. - **Orphans**: `prior_chunk_keys − new_keep_set`. In practice this is @@ -80,10 +93,17 @@ pointer no longer does. 2. Write the new root pointer — upsert ← **commit point** 3. Delete `prior_chunk_keys − new_keep_set` — best-effort -2. **Delete only what the previous pointer referenced.** The delete-set - is derived from `previous_pointer.chunks[].key`, not from a prefix - scan or a store-wide enumeration. This guarantees the sweep can only - ever touch keys that were part of this root's own prior generation. + (Locally, steps 1–3 collapse into one atomic `fs::write` of the + whole `fastly.toml`, so there is no partial-commit window and the + local sweep can neither leak nor corrupt.) + +2. **Delete only validated, prefix-matched prior references.** The + delete-set is derived from `previous_pointer.chunks[].key`, filtered + to keys with the exact prefix `"{root_key}.__edgezero_chunks."`, and + only when the previous pointer is a valid v1 pointer. It is NOT + derived from a prefix scan or a store-wide enumeration. This bounds + the blast radius: a malformed or hand-edited pointer can never cause + GC to delete a key outside its own root's chunk namespace. 3. **A key in the new keep-set is never deleted.** Even though a SHA change makes overlap empty in practice, the sweep subtracts the @@ -95,6 +115,48 @@ pointer no longer does. warning folded into the returned status lines; the push still reports success. A leaked chunk is harmless; a failed push is not. +## `prior_chunk_keys` helper (`chunked_config.rs`) + +Add next to the existing `FastlyChunkPointer` schema +(`chunked_config.rs:47`): + +```rust +/// Validate a prior root value and return the chunk keys it referenced, +/// scoped to `root_key`'s own chunk namespace. Used only for GC. +/// +/// Returns: +/// - `Ok(keys)` — value is a valid v1 chunk pointer; `keys` are its +/// `chunks[].key` entries that match `"{root_key}.__edgezero_chunks."`. +/// - `Ok(vec![])` — value is a direct `BlobEnvelope`, absent, or not +/// pointer-shaped at all (normal: first push, or was-direct). Silent. +/// - `Err(msg)` — value IS pointer-kind (`edgezero_kind == POINTER_KIND`) +/// but fails validation: unsupported `version`, or a referenced key +/// falls outside this root's chunk prefix. Caller logs `msg` as a +/// warning and skips GC for this root (deletes nothing). +pub(crate) fn prior_chunk_keys( + root_key: &str, + raw: &str, +) -> Result, String>; +``` + +Rules: + +- Attempt to deserialize `raw` into `FastlyChunkPointer`. On failure + (missing fields — e.g. a direct `BlobEnvelope`, or unrelated JSON) → + `Ok(vec![])`, silent. +- If it deserializes but `edgezero_kind != POINTER_KIND` → `Ok(vec![])`, + silent (not our pointer). +- If `edgezero_kind == POINTER_KIND` but `version != 1` → `Err(...)`, + warn. +- Compute `prefix = format!("{root_key}{CHUNK_KEY_INFIX}")`. If every + `chunks[].key` starts with `prefix` → `Ok(matching keys)`. If any key + does NOT match the prefix → `Err(...)`, warn, and return no keys (do + not partially delete a suspicious pointer). + +Unit tests: valid pointer → its keys; direct envelope → `Ok([])`; +unrelated JSON → `Ok([])`; wrong `edgezero_kind` → `Ok([])`; `version` +≠ 1 → `Err`; a chunk key with a foreign prefix → `Err`. + ## Algorithm For each logical `(root_key, envelope_json)` entry in the push: @@ -102,7 +164,7 @@ For each logical `(root_key, envelope_json)` entry in the push: ``` # --- before writing --- prev = read_root_value(root_key) # may be absent -prior_keys = prior_chunk_keys(prev) # [] unless prev is a pointer +prior = prior_chunk_keys(root_key, prev) # Ok([]) unless prev is a valid v1 pointer expanded = prepare_fastly_config_entries(root_key, envelope_json) new_keys = { k for (k, _) in expanded } ∪ { root_key } @@ -110,71 +172,142 @@ new_keys = { k for (k, _) in expanded } ∪ { root_key } push expanded # chunks first, root pointer last (commit point) # --- sweep (new, best-effort, only after push succeeds) --- -for k in prior_keys − new_keys: - try delete_entry(k) except e: warn("failed to delete orphan chunk `{k}`: {e}") +match prior: + Err(msg): warn(msg) # suspicious pointer; skip GC + Ok(keys): + for k in keys − new_keys: + try delete_entry(k) except e: warn("failed to delete orphan chunk `{k}`: {e}") ``` -`prior_chunk_keys` reuses the existing pointer schema: parse the prior -value as a `FastlyChunkPointer`; if it parses and -`edgezero_kind == POINTER_KIND`, return `chunks[].key`; otherwise -return `[]`. A direct `BlobEnvelope` fails to parse as a pointer and -yields `[]` — correct, since a direct value has no chunks. This means -the **shrink-to-direct** case (config drops back under 8 000 chars) is -handled for free: the new value is direct, `new_keys = { root_key }`, -and every prior chunk key is swept. - -## Cloud path (`push_cloud.rs`) - -- **Read prior value**: reuse `fetch_remote_config_store_entry(store_id, - root_key)` (already present for chunk resolution). Returns - `Ok(Some(raw))`, `Ok(None)` (absent → no prior chunks), or `Err` - (degrade to warning, skip GC for that root). -- **Store id**: `write_entries` already resolves `resolved_id` via - `resolve_remote_config_store_id`. The prior-value read must move to - after that resolution and before the committer loop. Because the read - needs the store id, hoist `resolve_remote_config_store_id` above the - per-root prior-value reads (it currently sits just before the commit - loop — move it up; the dry-run branch returns before it, unchanged). -- **Delete helper**: new `delete_config_store_entry(store_id, key)` +The **shrink-to-direct** case (config drops back under 8 000 chars) is +handled for free: the new value is a direct envelope, +`new_keys = { root_key }`, and every prior chunk key is swept. + +## Cloud path (`push_config_entries`, `cli.rs:349`) + +- **Dry-run stays offline.** The `dry_run` early return (`cli.rs:385`) + is *before* `resolve_remote_config_store_id` (`cli.rs:410`) and must + stay that way — dry-run never resolves the store id and never fetches + remote state. It reports GC intent without a count, e.g. + `" would delete orphaned prior-generation chunks of `app_config` (count determined at push time)"`. + (Rationale: counting orphans needs the store id + a remote read, + which would make dry-run hit the network. Not worth it; the local + dry-run below gives an exact count offline for the common case.) +- **Read prior value (real push only).** After `resolve_remote_config_store_id` + succeeds, for each logical root call the existing + `fetch_remote_config_store_entry(store_id, root_key)` (`cli.rs:671`) + to get `prev`. `Ok(None)` → no prior chunks; `Err` → warn, skip GC + for that root. This read happens *before* the committer loop. +- **Delete helper.** New `delete_config_store_entry(store_id, key)` shelling `fastly config-store-entry delete --store-id= - --key=`, mirroring the spawn/stderr handling of - `create_config_store_entry`. Treat a "not found" / "does not exist" / - "404" stderr as success (already gone). + --key= --auto-yes`, mirroring the spawn/stderr handling of + `create_config_store_entry` (`cli.rs:1073`). `--auto-yes` suppresses + any interactive confirmation (non-interactive shell-out). Treat a + "not found" / "does not exist" / "404" stderr as success (already + gone). +- **Sweep after commit.** After `push_entries_with_committer` + (`cli.rs:1022`) returns `Ok`, run the per-root sweep. Fold delete + failures and prior-read/parse warnings into the returned status + `Vec` (the push still returns `Ok`). - **Cost**: one extra `describe` per logical root before the push, plus one `delete` per orphan after. No store-wide `list` call. -## Local path (`push_local.rs` / `write_fastly_local_config_store`) +## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) The local `contents` table is held entirely in memory in a single -`DocumentMut`, so GC here is fully reliable and needs no round-trips: - -- Before the upsert loop at `provision_local.rs:559`, read each root - key's current value from `contents_tbl` (the prior pointer), compute - `prior_chunk_keys`, and after inserting the new physical entries, - `contents_tbl.remove(k)` for each orphan `k`. -- Because the whole file is rewritten atomically by the trailing - `fs::write`, steps 1–3 of the ordering invariant collapse into a - single write — there is no partial-commit window locally, so the - local sweep cannot leak or corrupt. -- The signature of `write_fastly_local_config_store` takes only - `entries` today. To compute `prior_chunk_keys` it must inspect the - pre-existing `contents_tbl` for each root key present in `entries`; - this is available in-function (the table is read before the upsert - loop). No new parameters are required — the prior values are read - from the table being edited. - -## Dry-run - -Both paths already have a dry-run branch that reports direct-vs-chunked -intent per key. Extend it: for each root key, report how many orphan -chunks *would* be deleted. The cloud dry-run reads the prior value -(same `fetch`) to count; if that read fails during dry-run, report -"unknown (prior read failed)" rather than erroring. Example lines: +`DocumentMut`, so GC here is fully reliable and needs no round-trips or +new threaded metadata. The writer already receives the flattened +`physical_entries`; it recovers logical roots without extra structure: + +- **Root inference.** A physical entry is a **root** iff its key does + NOT contain the reserved infix `CHUNK_KEY_INFIX` + (`.__edgezero_chunks.`). Every chunk key contains it; every root key + (direct or pointer) does not. This holds because + `prepare_fastly_config_entries` is the only producer of these keys + and only appends the infix to chunk keys. (Guarded by the root-key + invariant in Terminology.) +- **Sweep inside the atomic write.** In `write_fastly_local_config_store` + (`cli.rs:926`), before the upsert loop, snapshot each root's *old* + value from the existing `contents_tbl`. After inserting the new + physical entries, for each inferred root compute + `prior_chunk_keys(root, old_value)` and `contents_tbl.remove(k)` for + each orphan `k` in `prior − new_keys`. All within the one + `DocumentMut` that the trailing `fs::write` persists. +- **Warnings.** `prior_chunk_keys` returning `Err` for a root → collect + the message; the local push returns it as an extra status line. A + removal is infallible on an in-memory table, so the only local + warnings are suspicious-pointer ones. +- **Dry-run (offline, exact count).** `push_config_entries_local`'s + dry-run (`cli.rs:459`) already reads no remote state. Extend it to + read the current `fastly.toml`, compute `prior_chunk_keys` per root, + and report the exact orphan count, e.g. + `" would delete 9 orphan chunks from the previous generation of `app_config`"`. + If the file is absent or a root has no prior pointer, report 0. + +## Existing tests that MUST change + +- `second_oversized_push_converges_runtime_on_new_envelope` + (`cli.rs:3317`). Today it asserts old A-generation chunks **remain** + in the local `contents` table after push B (loop at `cli.rs:3416` + asserting `chunks_b.contains(chunk_key)`, and the "no GC in v1" + comments at `cli.rs:3303`/`:3324`/`:3360`/`:3405`/`:3414`). With GC + this inverts: after push B, every A-chunk MUST be **absent** + (`!chunks_b.contains(...)`), B-chunks present, and the runtime read + still reconstructs envelope B. Update the assertions and rewrite the + stale "no GC" commentary. + +## New tests + +Reuse the fake-`fastly` shim + tempdir/`DocumentMut` harnesses already +in the `cli.rs` test module. + +### `chunked_config.rs` — `prior_chunk_keys` + +- Valid v1 pointer → its `chunks[].key` in order. +- Direct `BlobEnvelope` → `Ok([])`. +- Unrelated / unparseable JSON → `Ok([])`. +- `edgezero_kind` ≠ `POINTER_KIND` → `Ok([])`. +- `version` ≠ 1 → `Err`. +- A `chunks[].key` outside `"{root_key}.__edgezero_chunks."` → `Err`, + returns no keys. + +### Cloud (`push_config_entries`) -``` - would push `app_config` as chunked (12 chunks + 1 pointer, 84210B total) - would delete 9 orphan chunks from the previous generation of `app_config` -``` +- **Deletes prior, keeps new**: fake fastly serves a prior pointer for + the root; after push assert `config-store-entry delete --auto-yes` is + invoked for each prior chunk key and NOT for any new chunk key or the + root key. +- **Shrink-to-direct**: prior is a chunk pointer, new value is direct + (≤ 8 000 chars) → all prior chunk keys deleted, root key upserted + (not deleted). +- **First push (no prior)**: prior read returns not-found → zero delete + calls. +- **Identical re-push**: prior and new reference the same keys → zero + delete calls. +- **Delete failure degrades to warning**: delete exits non-zero with a + non-"not found" stderr → push returns `Ok`, status includes a warning + naming the failed chunk key. +- **Prior read failure degrades to warning**: `describe` on the root + errors (non-not-found) → push `Ok`, GC skipped, warning present. +- **Suspicious pointer degrades to warning**: prior value is + pointer-kind with `version` 2 → push `Ok`, no deletes, warning. +- **Ordering**: extend the argv-log fake to assert every `delete` argv + appears strictly after the root-pointer `update` argv. +- **Dry-run stays offline**: dry-run makes no `list`/`describe`/`delete` + calls and prints the no-count GC intent line. + +### Local (`push_config_entries_local` / `write_fastly_local_config_store`) + +- **Prunes orphan chunk keys**: seed a `fastly.toml` whose `contents` + holds a prior pointer + its chunks; push a changed (still-chunked) + config → new chunk keys present, prior chunk keys absent, sibling + logical keys untouched. (This is the inverted `cli.rs:3317` test.) +- **Shrink-to-direct locally**: prior chunked → new direct → all prior + chunk keys removed, root holds the direct envelope. +- **Sibling coexistence preserved**: a push of `app_config` must not + remove `app_config_staging` chunk keys (Spec 12.7 coexistence). +- **Dry-run exact count**: reports the correct orphan count and writes + nothing. ## Non-goals @@ -191,58 +324,12 @@ chunks *would* be deleted. The cloud dry-run reads the prior value there is no cross-key atomicity beyond what each path already provides (per-entry for cloud, whole-file for local). -## Testing - -Reuse the existing fake-`fastly` shim harness in `push_cloud.rs` tests -and the tempdir/`DocumentMut` harness in `provision_local.rs` / -`push_local.rs` tests. - -### `chunked_config.rs` unit tests (`prior_chunk_keys`) - -- Pointer value → returns its `chunks[].key` in order. -- Direct `BlobEnvelope` value → returns `[]`. -- Unparseable / unrelated JSON → returns `[]`. -- Pointer with non-matching `edgezero_kind` → returns `[]`. - -### Cloud (`push_cloud.rs`) - -- **Deletes prior, keeps new**: fake fastly serves a prior pointer for - the root; after push, assert `config-store-entry delete` is invoked - for each prior chunk key and NOT for any new chunk key or the root - key. -- **Shrink-to-direct**: prior value is a chunk pointer, new value is - direct (≤ 8 000 chars); assert all prior chunk keys are deleted and - the root key is upserted (not deleted). -- **First push (no prior)**: prior read returns not-found; assert zero - delete calls. -- **Identical re-push**: prior pointer and new pointer reference the - same keys; assert zero delete calls (keep-set subtraction). -- **Delete failure degrades to warning**: delete shell-out exits - non-zero with a non-"not found" stderr; assert the push still returns - success and the status includes a warning naming the failed chunk key. -- **Prior read failure degrades to warning**: `describe` on the root - errors (non-not-found); assert push succeeds, GC skipped, warning - present. -- **Ordering**: extend the existing argv-log fake to assert every - `delete` argv appears strictly after the root-pointer `update` argv. - -### Local (`push_local.rs` / `provision_local.rs`) - -- **Prunes orphan chunk keys from `contents`**: seed a `fastly.toml` - whose `contents` table holds a prior pointer + its chunk keys; push a - changed (still-chunked) config; assert the new chunk keys are present, - the prior chunk keys are absent, and sibling logical keys are - untouched. -- **Shrink-to-direct locally**: prior chunked → new direct; assert all - prior chunk keys removed and root key holds the direct envelope. -- **Sibling coexistence preserved**: a push of `app_config` must not - remove `app_config_staging` chunk keys (per Spec 12.7 coexistence). - ## Files touched | File | Change | | --- | --- | -| `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(&str) -> Vec` + unit tests | -| `crates/edgezero-adapter-fastly/src/cli/push_cloud.rs` | Hoist store-id resolution; per-root prior read; `delete_config_store_entry`; post-commit sweep; dry-run counts; tests | -| `crates/edgezero-adapter-fastly/src/cli/push_local.rs` | Dry-run orphan counts; wire sweep into local write; tests | -| `crates/edgezero-adapter-fastly/src/cli/provision_local.rs` | `write_fastly_local_config_store` prunes orphan chunk keys after upsert; tests | +| `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(root_key, raw) -> Result, String>` (validates v1 + prefix-scopes) + unit tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries`, `:349`) | Per-root prior read via `fetch_remote_config_store_entry`; `delete_config_store_entry` helper (`--auto-yes`); post-commit sweep; offline dry-run GC-intent line; tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Dry-run exact orphan counts; tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | Infer roots (no chunk infix), snapshot old values, prune orphans in the same atomic write | +| `crates/edgezero-adapter-fastly/src/cli.rs` (test `:3317`) | Invert: assert old chunks are deleted after re-push; drop "no GC in v1" commentary | From 533c6444c47ab885df9046ed82ad6c7ec10599f3 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 7 Jul 2026 22:06:08 -0700 Subject: [PATCH 03/25] Spec review v3: thread logical roots (--key is free-form, infix inference unsound); Value-first prior_chunk_keys so invalid pointer-kind warns; drop 'atomic' overclaim; define local dry-run degrade semantics; state cloud GC runs only after full commit --- .../specs/2026-07-07-fastly-chunk-gc.md | 142 ++++++++++++------ 1 file changed, 92 insertions(+), 50 deletions(-) diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 0336f65d..b9b83800 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -60,11 +60,15 @@ pointer no longer does. ## Terminology - **Root key**: the stable logical entry key (e.g. `app_config`). Holds - either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. Root - keys are store-config ids and MUST NOT contain the reserved chunk - infix `.__edgezero_chunks.` (they never do — they are validated - identifiers, and `prepare_fastly_config_entries` only ever appends - the infix to form chunk keys). + either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. The + root key is the logical store-config id, or the operator's + `--key ` — a free-form `Option` (`args.rs:311`, + used directly in `config.rs:297`). Because `--key` is **not** + validated today, GC does NOT infer roots by string-matching the chunk + infix; it threads the logical root keys explicitly (see Local path). + As defence-in-depth, the implementation SHOULD also reject a `--key` + containing the reserved infix `.__edgezero_chunks.` at push time, + since such a key would collide with the generated chunk namespace. - **Previous pointer**: the value stored at the root key *before* this push overwrites it. - **Prior chunk keys**: the keys named by `previous_pointer.chunks[]`, @@ -93,9 +97,13 @@ pointer no longer does. 2. Write the new root pointer — upsert ← **commit point** 3. Delete `prior_chunk_keys − new_keep_set` — best-effort - (Locally, steps 1–3 collapse into one atomic `fs::write` of the - whole `fastly.toml`, so there is no partial-commit window and the - local sweep can neither leak nor corrupt.) + (Locally, steps 1–3 happen as one in-memory `DocumentMut` rewrite + followed by the existing single `fs::write` of `fastly.toml` + (`cli.rs:999`). That write is a plain overwrite, NOT a durable + atomic rename — but from GC's perspective there is no partial-commit + window: the new entries and the orphan removals land in the same + rewrite, so a reader never sees new chunks without the new pointer, + nor a swept chunk that is still referenced.) 2. **Delete only validated, prefix-matched prior references.** The delete-set is derived from `previous_pointer.chunks[].key`, filtered @@ -139,23 +147,29 @@ pub(crate) fn prior_chunk_keys( ) -> Result, String>; ``` -Rules: - -- Attempt to deserialize `raw` into `FastlyChunkPointer`. On failure - (missing fields — e.g. a direct `BlobEnvelope`, or unrelated JSON) → - `Ok(vec![])`, silent. -- If it deserializes but `edgezero_kind != POINTER_KIND` → `Ok(vec![])`, - silent (not our pointer). -- If `edgezero_kind == POINTER_KIND` but `version != 1` → `Err(...)`, - warn. -- Compute `prefix = format!("{root_key}{CHUNK_KEY_INFIX}")`. If every - `chunks[].key` starts with `prefix` → `Ok(matching keys)`. If any key - does NOT match the prefix → `Err(...)`, warn, and return no keys (do - not partially delete a suspicious pointer). +Rules — parse as `serde_json::Value` **first**, so a pointer-kind value +with missing or invalid fields still reaches the warning path instead +of being silently dropped by a failed struct deserialize (e.g. +`{"edgezero_kind":"fastly_config_chunks","version":2}` has no `chunks` +array and would fail a direct `FastlyChunkPointer` deserialize): + +1. Parse `raw` as `serde_json::Value`. If that fails, or the top-level + `edgezero_kind` field is absent or `!= POINTER_KIND` → `Ok(vec![])`, + silent. (Direct `BlobEnvelope`, unrelated JSON, or a first push.) +2. The value IS pointer-kind. From here every failure is `Err(...)` + (warn, delete nothing): + - required fields missing / wrong types (no `chunks` array, + `version` not an integer, …), or + - `version != 1`, or + - any `chunks[].key` does NOT start with + `format!("{root_key}{CHUNK_KEY_INFIX}")`. +3. Otherwise → `Ok(keys)`, the `chunks[].key` values (all prefix-matched + by step 2). Unit tests: valid pointer → its keys; direct envelope → `Ok([])`; -unrelated JSON → `Ok([])`; wrong `edgezero_kind` → `Ok([])`; `version` -≠ 1 → `Err`; a chunk key with a foreign prefix → `Err`. +unrelated JSON → `Ok([])`; wrong `edgezero_kind` → `Ok([])`; pointer-kind +with missing `chunks` and `version:2` → `Err` (NOT silently `Ok([])`); +`version` ≠ 1 → `Err`; a chunk key with a foreign prefix → `Err`. ## Algorithm @@ -205,44 +219,65 @@ handled for free: the new value is a direct envelope, any interactive confirmation (non-interactive shell-out). Treat a "not found" / "does not exist" / "404" stderr as success (already gone). -- **Sweep after commit.** After `push_entries_with_committer` - (`cli.rs:1022`) returns `Ok`, run the per-root sweep. Fold delete - failures and prior-read/parse warnings into the returned status +- **Sweep after the whole commit.** The cloud writer flattens every + logical root's physical entries into one `physical_entries` vec + (`cli.rs:380`) and commits them in a single `push_entries_with_committer` + loop (`cli.rs:411`, `:1022`). Prior-value reads for every root happen + *before* that loop; the per-root sweep runs only *after* it returns + `Ok` — i.e. after ALL roots' chunks + pointers are committed. Fold + delete failures and prior-read/parse warnings into the returned status `Vec` (the push still returns `Ok`). +- **Partial-commit failure ⇒ no GC.** If the committer fails partway, + `push_config_entries` returns `Err` and no sweep runs for any root. + Safe by construction: nothing is deleted, so the store is left with + (at worst) a mix of old and new chunks, each still referenced by + whichever pointer actually committed. Reclamation waits for the next + successful push. Leak-safe over corrupt-safe, per invariant 4. - **Cost**: one extra `describe` per logical root before the push, plus one `delete` per orphan after. No store-wide `list` call. ## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) The local `contents` table is held entirely in memory in a single -`DocumentMut`, so GC here is fully reliable and needs no round-trips or -new threaded metadata. The writer already receives the flattened -`physical_entries`; it recovers logical roots without extra structure: - -- **Root inference.** A physical entry is a **root** iff its key does - NOT contain the reserved infix `CHUNK_KEY_INFIX` - (`.__edgezero_chunks.`). Every chunk key contains it; every root key - (direct or pointer) does not. This holds because - `prepare_fastly_config_entries` is the only producer of these keys - and only appends the infix to chunk keys. (Guarded by the root-key - invariant in Terminology.) -- **Sweep inside the atomic write.** In `write_fastly_local_config_store` - (`cli.rs:926`), before the upsert loop, snapshot each root's *old* - value from the existing `contents_tbl`. After inserting the new - physical entries, for each inferred root compute +`DocumentMut`, so GC here is fully reliable and needs no round-trips. +To stay sound against a free-form `--key`, the logical root keys are +threaded explicitly rather than inferred from the flattened entries: + +- **Thread logical roots (no inference).** `push_config_entries_local` + already holds the logical `entries: &[(String, String)]`. Pass the + logical root keys into `write_fastly_local_config_store` via a new + parameter (e.g. `roots: &[&str]`) alongside `physical_entries`. No + string-matching on the chunk infix — a `--key` that happens to + contain `.__edgezero_chunks.` cannot mislead root detection. +- **Sweep inside the single rewrite.** In `write_fastly_local_config_store` + (`cli.rs:926`), before the upsert loop, snapshot each threaded root's + *old* value from the existing `contents_tbl`. After inserting the new + physical entries, for each root compute `prior_chunk_keys(root, old_value)` and `contents_tbl.remove(k)` for each orphan `k` in `prior − new_keys`. All within the one - `DocumentMut` that the trailing `fs::write` persists. + `DocumentMut` that the trailing `fs::write` (`cli.rs:999`) persists — + one in-memory rewrite, then one write. - **Warnings.** `prior_chunk_keys` returning `Err` for a root → collect the message; the local push returns it as an extra status line. A removal is infallible on an in-memory table, so the only local warnings are suspicious-pointer ones. -- **Dry-run (offline, exact count).** `push_config_entries_local`'s - dry-run (`cli.rs:459`) already reads no remote state. Extend it to - read the current `fastly.toml`, compute `prior_chunk_keys` per root, - and report the exact orphan count, e.g. +- **Dry-run (offline, best-effort count).** `push_config_entries_local`'s + dry-run (`cli.rs:459`) reads no remote state. Extend it to read the + current `fastly.toml`, compute `prior_chunk_keys` per root, and report + the orphan count, e.g. `" would delete 9 orphan chunks from the previous generation of `app_config`"`. - If the file is absent or a root has no prior pointer, report 0. + Error semantics — the dry-run MUST NOT newly fail where it succeeds + today (the current dry-run does not read the file at all), so GC + counting degrades rather than erroring: + - File absent, or a root has no prior pointer / a direct prior value + → report `0`. + - File present but unreadable, malformed TOML, `contents` not a table, + or a root's value not a string → report + `"unknown (could not read prior state)"` for that root and continue. + (The real push still fails fatally on malformed TOML via the writer + at `cli.rs:938`; only the dry-run *count* degrades.) + - Prior value is pointer-kind-but-invalid (`prior_chunk_keys` → `Err`) + → report `"unknown (suspicious prior pointer)"`. ## Existing tests that MUST change @@ -306,8 +341,14 @@ in the `cli.rs` test module. chunk keys removed, root holds the direct envelope. - **Sibling coexistence preserved**: a push of `app_config` must not remove `app_config_staging` chunk keys (Spec 12.7 coexistence). -- **Dry-run exact count**: reports the correct orphan count and writes +- **Free-form `--key` with the infix**: threading roots (not inference) + means a root key containing `.__edgezero_chunks.` is still handled + correctly — its own prior chunks are swept, siblings untouched. +- **Dry-run count**: reports the correct orphan count and writes nothing. +- **Dry-run degrades, never fails**: a malformed `fastly.toml` makes the + dry-run report `unknown` for the GC count while still printing the + direct-vs-chunked intent lines and returning `Ok`. ## Non-goals @@ -330,6 +371,7 @@ in the `cli.rs` test module. | --- | --- | | `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(root_key, raw) -> Result, String>` (validates v1 + prefix-scopes) + unit tests | | `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries`, `:349`) | Per-root prior read via `fetch_remote_config_store_entry`; `delete_config_store_entry` helper (`--auto-yes`); post-commit sweep; offline dry-run GC-intent line; tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Dry-run exact orphan counts; tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | Infer roots (no chunk infix), snapshot old values, prune orphans in the same atomic write | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Thread logical roots to the writer; best-effort dry-run orphan counts (degrade to "unknown", never newly fail); tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | New `roots: &[&str]` param; snapshot old values, prune orphans in the same in-memory rewrite before the existing `fs::write` (`:999`) | | `crates/edgezero-adapter-fastly/src/cli.rs` (test `:3317`) | Invert: assert old chunks are deleted after re-push; drop "no GC in v1" commentary | +| `crates/edgezero-cli/src/config.rs` (`:297`) / `args.rs` (`:311`) — optional, defence-in-depth | Reject a `--key` containing `.__edgezero_chunks.` at push time (collides with the chunk namespace) | From d484fa54f8b585127e2d5de6f69236db3eb9f31a Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 7 Jul 2026 22:46:24 -0700 Subject: [PATCH 04/25] Add Fastly chunk-GC implementation plan, reconciled with spec v3 Value-first prior_chunk_keys (pointer-kind-but-malformed warns), thread logical roots into write_fastly_local_config_store via roots: &[&str] (no infix inference, since --key is free-form), best-effort local dry-run counts, post-commit-only cloud sweep. Task-by-task with TDD steps for subagent-driven-development. --- .../plans/2026-07-07-fastly-chunk-gc.md | 791 ++++++++++++++++++ 1 file changed, 791 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md new file mode 100644 index 00000000..f8cec88f --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -0,0 +1,791 @@ +# Fastly Chunk GC Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reclaim obsolete Fastly chunked config-store entries after a successful re-push without changing push success semantics. + +**Architecture:** Add a root-scoped prior-pointer inspector in `chunked_config.rs`, then wire cloud and local Fastly push paths in `cli.rs` to snapshot prior chunk keys before writing and sweep them only after the new root entry is committed. Cloud GC is best-effort and warning-only; local GC happens inside the same `DocumentMut` rewrite used for `fastly.toml`. + +**Tech Stack:** Rust 2021, `serde_json`, `toml_edit`, existing Fastly CLI shell-out helpers, existing fake-`fastly` test shims, `cargo test -p edgezero-adapter-fastly --features cli`. + +--- + +## File Structure + +- Modify `crates/edgezero-adapter-fastly/src/chunked_config.rs` + - Add `prior_chunk_keys(root_key, raw) -> Result, String>` next to `FastlyChunkPointer`. + - Add focused unit tests in the existing `#[cfg(test)]` module. + +- Modify `crates/edgezero-adapter-fastly/src/cli.rs` + - Import `prior_chunk_keys` and `CHUNK_KEY_INFIX`. + - Add small private GC metadata helpers (`FastlyConfigGcPlan`, `new_key_set_for_root`, `orphan_chunk_keys`) near the config-push helpers. Roots are sourced from logical `entries`/threaded `roots`, never inferred from physical keys. + - Extend `push_config_entries` with offline dry-run intent, prior root read, post-commit cloud sweep (only after the whole flattened commit returns `Ok`), and warning status lines. + - Add `delete_config_store_entry(store_id, key)`. + - Extend `push_config_entries_local` dry-run with best-effort offline orphan counts (degrade to `unknown`, never newly fail). + - Extend `write_fastly_local_config_store` with a `roots: &[&str]` param; prune prior root chunks in the same in-memory rewrite before the (non-atomic) `fs::write` and return warning lines from suspicious prior pointers. + - Add/adjust tests in the existing `cli.rs` test module. + +Optional (defence-in-depth, spec "Files touched"): reject a `--key` +containing `.__edgezero_chunks.` at push time in `edgezero-cli` +(`config.rs:297` / `args.rs:311`), since such a key collides with the +generated chunk namespace. Not required for GC correctness (roots are +threaded, not inferred); skip if it widens scope. + +--- + +### Task 1: Add Root-Scoped Prior Pointer Inspection + +**Files:** +- Modify: `crates/edgezero-adapter-fastly/src/chunked_config.rs:47` +- Test: `crates/edgezero-adapter-fastly/src/chunked_config.rs:309` + +- [ ] **Step 1: Add failing tests for `prior_chunk_keys`** + +Add tests near the existing chunked-config unit tests: + +```rust +#[test] +fn prior_chunk_keys_returns_valid_v1_pointer_keys() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + + let keys = prior_chunk_keys("app_config", pointer_json).expect("valid pointer"); + + let expected: Vec = entries[..entries.len().saturating_sub(1)] + .iter() + .map(|(key, _)| key.clone()) + .collect(); + assert_eq!(keys, expected); +} + +#[test] +fn prior_chunk_keys_returns_empty_for_direct_envelope() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT); + assert_eq!(prior_chunk_keys("app_config", &envelope).unwrap(), Vec::::new()); +} + +#[test] +fn prior_chunk_keys_returns_empty_for_unrelated_json() { + assert_eq!( + prior_chunk_keys("app_config", r#"{"hello":"world"}"#).unwrap(), + Vec::::new() + ); +} + +#[test] +fn prior_chunk_keys_returns_empty_for_wrong_kind() { + let raw = r#"{"edgezero_kind":"other","version":1,"chunks":[],"data_sha256":"","envelope_len":0,"envelope_sha256":""}"#; + assert_eq!(prior_chunk_keys("app_config", raw).unwrap(), Vec::::new()); +} + +#[test] +fn prior_chunk_keys_rejects_unsupported_pointer_version() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + let mut pointer: FastlyChunkPointer = serde_json::from_str(pointer_json).unwrap(); + pointer.version = 2; + let raw = serde_json::to_string(&pointer).unwrap(); + + let err = prior_chunk_keys("app_config", &raw).expect_err("version 2 should warn"); + assert!(err.contains("unsupported version"), "{err}"); +} + +#[test] +fn prior_chunk_keys_rejects_foreign_chunk_prefix() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + let mut pointer: FastlyChunkPointer = serde_json::from_str(pointer_json).unwrap(); + pointer.chunks[0].key = pointer.chunks[0].key.replacen("app_config", "app_config_staging", 1); + let raw = serde_json::to_string(&pointer).unwrap(); + + let err = prior_chunk_keys("app_config", &raw).expect_err("foreign chunk should warn"); + assert!(err.contains("outside"), "{err}"); +} + +// Regression for the Value-first rule: a value that IS pointer-kind but +// is missing required fields (`chunks`, `data_sha256`, …) must WARN, not +// be silently dropped by a failed struct deserialize. +#[test] +fn prior_chunk_keys_warns_on_pointer_kind_with_missing_fields() { + let raw = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#; + let err = prior_chunk_keys("app_config", raw) + .expect_err("pointer-kind but malformed must warn, not Ok([])"); + assert!(!err.is_empty(), "{err}"); +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli prior_chunk_keys +``` + +Expected: fail because `prior_chunk_keys` is not defined. + +- [ ] **Step 3: Implement `prior_chunk_keys`** + +Add near the other public helpers: + +Parse as `serde_json::Value` **first** so a pointer-kind value with +missing/invalid fields reaches the warning path (spec: "Rules — parse as +`serde_json::Value` first"). Deserializing straight into +`FastlyChunkPointer` would make `{"edgezero_kind":"...","version":2}` +(no `chunks`) fail parsing and be silently returned as `Ok([])`. + +```rust +#[cfg(any(feature = "cli", test))] +pub(crate) fn prior_chunk_keys(root_key: &str, raw: &str) -> Result, String> { + // 1. Parse loosely. Not-JSON, or not our pointer kind => silent. + let value: serde_json::Value = match serde_json::from_str(raw) { + Ok(value) => value, + Err(_) => return Ok(Vec::new()), + }; + if value.get("edgezero_kind").and_then(serde_json::Value::as_str) != Some(POINTER_KIND) { + // Direct BlobEnvelope, unrelated JSON, or first push. + return Ok(Vec::new()); + } + + // 2. It IS pointer-kind: from here every failure WARNS (Err), never silent. + let pointer: FastlyChunkPointer = serde_json::from_value(value).map_err(|err| { + format!("prior chunk pointer at `{root_key}` is malformed: {err}; skipping chunk GC") + })?; + if pointer.version != 1 { + return Err(format!( + "prior chunk pointer at `{root_key}` has unsupported version {}; skipping chunk GC", + pointer.version + )); + } + + // 3. Prefix-scope every referenced key to this root's own namespace. + let prefix = format!("{root_key}{CHUNK_KEY_INFIX}"); + let mut keys = Vec::with_capacity(pointer.chunks.len()); + for chunk_ref in pointer.chunks { + if !chunk_ref.key.starts_with(&prefix) { + return Err(format!( + "prior chunk pointer at `{root_key}` references chunk key `{}` outside expected prefix `{prefix}`; skipping chunk GC", + chunk_ref.key + )); + } + keys.push(chunk_ref.key); + } + Ok(keys) +} +``` + +- [ ] **Step 4: Run scoped tests** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli prior_chunk_keys +``` + +Expected: all `prior_chunk_keys` tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add crates/edgezero-adapter-fastly/src/chunked_config.rs +git commit -m "test(fastly): cover prior chunk key extraction" +``` + +--- + +### Task 2: Add Shared GC Metadata Helpers in Fastly CLI + +**Files:** +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:8` +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:1004` +- Test: compile through later tasks + +- [ ] **Step 1: Update imports** + +Change the Fastly chunked config import: + +```rust +use crate::chunked_config::{ + prior_chunk_keys, prepare_fastly_config_entries, resolve_fastly_config_value, CHUNK_KEY_INFIX, +}; +``` + +- [ ] **Step 2: Add small metadata structs/helpers near config-push helpers** + +Add before `push_entries_with_committer`. Note there is **no** +`root_keys_from_physical_entries` inference helper: `--key` is a +free-form `Option` (`args.rs:311`), so a logical root key may +itself contain `.__edgezero_chunks.`. Root keys are always sourced from +the caller's **logical** `entries` (cloud) or the threaded `roots` param +(local) — never string-matched out of the flattened physical entries. + +```rust +#[derive(Debug)] +struct FastlyConfigGcPlan { + root_key: String, + prior_keys: Result, String>, + new_keys: std::collections::HashSet, +} + +fn new_key_set_for_root(root_key: &str, entries: &[(String, String)]) -> std::collections::HashSet { + entries + .iter() + .filter_map(|(key, _)| { + if key == root_key || key.starts_with(&format!("{root_key}{CHUNK_KEY_INFIX}")) { + Some(key.clone()) + } else { + None + } + }) + .collect() +} + +fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { + match &plan.prior_keys { + Ok(prior) => Ok(prior + .iter() + .filter(|key| !plan.new_keys.contains(*key)) + .cloned() + .collect()), + Err(err) => Err(err.clone()), + } +} +``` + +Keep helpers private. If `clippy` complains about line length or type imports, pull `HashSet` into the top-level imports instead. + +- [ ] **Step 3: Run compile check** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli --no-run +``` + +Expected: compile succeeds. + +- [ ] **Step 4: Commit** + +```bash +git add crates/edgezero-adapter-fastly/src/cli.rs +git commit -m "refactor(fastly): add chunk gc planning helpers" +``` + +--- + +### Task 3: Implement Local Fastly TOML GC + +**Files:** +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:421` +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:926` +- Test: `crates/edgezero-adapter-fastly/src/cli.rs:2991` +- Test: `crates/edgezero-adapter-fastly/src/cli.rs:3296` + +- [ ] **Step 1: Rewrite the second oversized local push test to fail** + +In `second_oversized_push_converges_runtime_on_new_envelope`, invert the old-chunk assertion: + +```rust +for chunk_key in &chunks_a { + assert!( + !chunks_b.contains(chunk_key), + "old A-chunk `{chunk_key}` must be pruned after push B; B-set={chunks_b:?}" + ); +} +``` + +Update stale comments that say old chunks remain or no GC exists. + +- [ ] **Step 2: Add shrink-to-direct local test** + +Add a test near the local push tests: + +```rust +#[cfg(unix)] +#[test] +fn push_config_entries_local_prunes_prior_chunks_when_value_shrinks_to_direct() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let chunked = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), chunked)], + &AdapterPushContext::new(), + false, + ) + .expect("first push"); + + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), direct.clone())], + &AdapterPushContext::new(), + false, + ) + .expect("second push"); + + let after = fs::read_to_string(&fastly_toml).expect("read"); + let doc: toml_edit::DocumentMut = after.parse().expect("parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents"); + + assert_eq!(contents.get(TEST_CONFIG_ID).and_then(toml_edit::Item::as_str), Some(direct.as_str())); + assert!( + !contents.iter().any(|(key, _)| key.contains(CHUNK_KEY_INFIX)), + "prior chunks must be removed: {after}" + ); +} +``` + +- [ ] **Step 3: Extend local dry-run test to expect orphan count** + +Seed a prior chunked push before dry-run, keep `original`, then assert: + +```rust +assert!(combined.contains("would delete")); +assert!(combined.contains("orphan chunks")); +``` + +Also keep `assert_eq!(after, original)`. + +- [ ] **Step 4: Run local tests to verify failures** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli push_config_entries_local -- --nocapture +cargo test -p edgezero-adapter-fastly --features cli second_oversized_push_converges_runtime_on_new_envelope -- --nocapture +``` + +Expected: failures showing old chunks remain and dry-run lacks GC count. + +- [ ] **Step 5: Implement local in-memory pruning** + +Change `write_fastly_local_config_store` to (a) accept the **logical +root keys** explicitly via a new `roots: &[&str]` param and (b) return +warning lines, while preserving its error behavior: + +```rust +fn write_fastly_local_config_store( + path: &Path, + platform_name: &str, + entries: &[(String, String)], + roots: &[&str], +) -> Result, String> +``` + +Update ALL existing call sites to pass `roots`. The adapter caller +(`push_config_entries_local`) passes the logical entry keys +(`&entries.iter().map(|(k, _)| k.as_str()).collect::>()`). +Setup-only writes in the unit-test module pass the store id they seed +(e.g. `&[TEST_CONFIG_ID]`), or `&[]` where the test does not exercise +GC. Existing direct callers that only `.expect("write")` can ignore the +returned `Vec`. + +Inside the function: + +1. Snapshot prior state for the **threaded roots** before mutation + (no inference — the roots are given): + +```rust +let mut plans = Vec::with_capacity(roots.len()); +for root_key in roots { + let prior_keys = contents_tbl + .get(root_key) + .and_then(toml_edit::Item::as_str) + .map_or_else(|| Ok(Vec::new()), |raw| prior_chunk_keys(root_key, raw)); + plans.push(FastlyConfigGcPlan { + root_key: (*root_key).to_owned(), + prior_keys, + new_keys: new_key_set_for_root(root_key, entries), + }); +} +``` + +2. Keep the existing upsert loop. + +3. After the upsert loop, remove orphans and collect warnings: + +```rust +let mut warnings = Vec::new(); +for plan in &plans { + match orphan_chunk_keys(plan) { + Ok(orphan_keys) => { + for key in orphan_keys { + contents_tbl.remove(&key); + } + } + Err(err) => warnings.push(format!("warning: {err}")), + } +} +``` + +Return `Ok(warnings)` after `fs::write`. Suspicious prior pointers must not fail the local write and must not produce partial deletes. + +In `push_config_entries_local`, capture the warning vector and append it after the existing success line: + +```rust +let roots: Vec<&str> = entries.iter().map(|(key, _)| key.as_str()).collect(); +let warnings = write_fastly_local_config_store(&fastly_path, name, &physical_entries, &roots)?; +let mut out = vec![format!( + "wrote {} physical entries ({} logical) to `[local_server.config_stores.{name}.contents]` in {} (logical id `{logical}`); restart `fastly compute serve` to pick up changes", + physical_entries.len(), + entries.len(), + fastly_path.display() +)]; +out.extend(warnings); +Ok(out) +``` + +- [ ] **Step 6: Add local dry-run count helper** + +Add a private helper that reads `fastly.toml` and returns counts per root: + +```rust +fn local_orphan_counts_for_dry_run( + path: &Path, + platform_name: &str, + entries: &[(String, String)], // logical entries; roots are their keys +) -> Vec<(String, Result)> { + // Roots are the logical entry keys (NOT inferred from physical keys). + // Missing file, missing/absent contents table, or a root with no prior + // pointer / a direct prior value => Ok(0) per root. + // Unreadable file, malformed TOML, contents-not-a-table, or a root + // value that is not a string => Err("could not read prior state"). + // prior_chunk_keys(root, old_raw) returning Err (suspicious pointer) + // => Err(). + // Use prior_chunk_keys(root, old_raw), new_key_set_for_root, and orphan_chunk_keys. +} +``` + +This helper MUST NOT fail the dry-run: it returns a per-root `Result` +that the caller renders as a count or an `unknown (...)` line. The +current dry-run reads no file at all, so introducing this read must not +turn a previously-succeeding dry-run into an error. Use exact +`toml_edit` table access, not string scanning. Keep this helper small; +if it gets large, split only into `local_contents_table` and count +computation. + +- [ ] **Step 7: Wire local dry-run count output** + +In `push_config_entries_local` dry-run, after each direct/chunked line add: + +```rust +out.push(format!( + " would delete {count} orphan chunks from the previous generation of `{key}`" +)); +``` + +For an `Err`, emit: + +```rust +out.push(format!( + " would delete unknown orphan chunks from the previous generation of `{key}` ({err})" +)); +``` + +- [ ] **Step 8: Run local scoped tests** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli push_config_entries_local -- --nocapture +cargo test -p edgezero-adapter-fastly --features cli second_oversized_push_converges_runtime_on_new_envelope -- --nocapture +cargo test -p edgezero-adapter-fastly --features cli read_config_entry_local -- --nocapture +``` + +Expected: local push/read tests pass. + +- [ ] **Step 9: Commit** + +```bash +git add crates/edgezero-adapter-fastly/src/cli.rs +git commit -m "feat(fastly): prune local config chunks on re-push" +``` + +--- + +### Task 4: Implement Cloud Fastly GC + +**Files:** +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:349` +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:671` +- Modify: `crates/edgezero-adapter-fastly/src/cli.rs:1073` +- Test: `crates/edgezero-adapter-fastly/src/cli.rs:2440` +- Test: `crates/edgezero-adapter-fastly/src/cli.rs:2760` + +- [ ] **Step 1: Add or replace command-aware fake Fastly helper** + +Add a Unix-only helper in the test module that: + +- Responds to `config-store list --json`. +- Responds to `config-store-entry describe --key=` from a key-response map. +- Accepts `config-store-entry update --upsert --stdin` and logs `update `. +- Accepts `config-store-entry delete --key= --auto-yes` and logs `delete `. +- Can be configured so specific delete keys fail with stderr. + +Prefer one operation per log line, for example: + +```text +list +describe app_config +update app_config.__edgezero_chunks..0 +update app_config +delete app_config.__edgezero_chunks..0 +``` + +- [ ] **Step 2: Add failing cloud tests** + +Add tests for: + +- Prior chunks are deleted and new/root keys are not. +- Shrink-to-direct deletes all prior chunks. +- No prior value produces zero deletes. +- Identical re-push produces zero deletes. +- Delete failure returns `Ok` with a warning line naming the chunk. +- Prior read failure returns `Ok` with a warning and no deletes. +- Suspicious pointer version returns `Ok` with a warning and no deletes. +- Delete operations occur after root update. +- Dry-run stays offline and emits no-count GC intent. + +Use `path_mutation_guard()` for tests that prepend a fake `fastly`. + +- [ ] **Step 3: Run cloud tests to verify failures** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli push_config_entries_ -- --nocapture +``` + +Expected: new GC tests fail because no deletes or warning lines exist yet. + +- [ ] **Step 4: Add `delete_config_store_entry`** + +Add near `create_config_store_entry`: + +```rust +fn delete_config_store_entry(store_id: &str, key: &str) -> Result<(), String> { + let store_arg = format!("--store-id={store_id}"); + let key_arg = format!("--key={key}"); + let output = Command::new("fastly") + .args([ + "config-store-entry", + "delete", + store_arg.as_str(), + key_arg.as_str(), + "--auto-yes", + ]) + .output() + .map_err(|err| { + if err.kind() == ErrorKind::NotFound { + format!("`fastly` not found on PATH; {FASTLY_INSTALL_HINT}") + } else { + format!("failed to spawn `fastly`: {err}") + } + })?; + if output.status.success() { + return Ok(()); + } + let stderr = String::from_utf8_lossy(&output.stderr); + let lower = stderr.to_ascii_lowercase(); + if lower.contains("not found") || lower.contains("does not exist") || lower.contains("404") { + return Ok(()); + } + Err(format!( + "`fastly config-store-entry delete --store-id={store_id} --key={key} --auto-yes` exited with status {}\nstderr: {}", + output.status, + stderr.trim() + )) +} +``` + +- [ ] **Step 5: Build GC plans before cloud commit** + +In `push_config_entries`, after `resolved_id`: + +```rust +let mut gc_plans = Vec::with_capacity(entries.len()); +let mut warnings = Vec::new(); +for (root_key, _) in entries { + let prior_keys = match fetch_remote_config_store_entry(&resolved_id, root_key) { + Ok(Some(raw)) => prior_chunk_keys(root_key, &raw), + Ok(None) => Ok(Vec::new()), + Err(err) => { + Err(format!( + "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" + )) + } + }; + gc_plans.push(FastlyConfigGcPlan { + root_key: root_key.clone(), + prior_keys, + new_keys: new_key_set_for_root(root_key, &physical_entries), + }); +} +``` + +If `prior_chunk_keys` returns `Err`, push a warning and keep the `Err` in the plan so deletes are skipped. + +- [ ] **Step 6: Sweep after successful commit** + +Keep the existing `push_entries_with_committer` call. Only after it returns `Ok`, iterate plans: + +```rust +for plan in &gc_plans { + match orphan_chunk_keys(plan) { + Ok(keys) => { + for key in keys { + if let Err(err) = delete_config_store_entry(&resolved_id, &key) { + warnings.push(format!( + "warning: failed to delete orphan chunk `{key}` for `{}`: {err}", + plan.root_key + )); + } + } + } + Err(err) => warnings.push(format!( + "warning: {err}" + )), + } +} +``` + +Return the existing success line plus warning lines. + +- [ ] **Step 7: Keep cloud dry-run offline** + +In the `dry_run` branch, after each direct/chunked line add: + +```rust +out.push(format!( + " would delete orphaned prior-generation chunks of `{key}` (count determined at push time)" +)); +``` + +Do not move `resolve_remote_config_store_id` above the dry-run branch. + +- [ ] **Step 8: Run cloud scoped tests** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli push_config_entries_ -- --nocapture +``` + +Expected: cloud push tests pass. + +- [ ] **Step 9: Commit** + +```bash +git add crates/edgezero-adapter-fastly/src/cli.rs +git commit -m "feat(fastly): delete prior remote config chunks after push" +``` + +--- + +### Task 5: Integration Cleanup and Verification + +**Files:** +- Modify as needed: `crates/edgezero-adapter-fastly/src/cli.rs` +- Modify as needed: `crates/edgezero-adapter-fastly/src/chunked_config.rs` + +- [ ] **Step 1: Run Fastly adapter tests** + +Run: + +```bash +cargo test -p edgezero-adapter-fastly --features cli +``` + +Expected: all Fastly adapter CLI-feature tests pass. + +- [ ] **Step 2: Run full workspace tests required after code changes** + +Run: + +```bash +cargo test --workspace --all-targets +``` + +Expected: all workspace tests pass. + +- [ ] **Step 3: Run formatting check** + +Run: + +```bash +cargo fmt --all -- --check +``` + +Expected: no formatting diffs. + +- [ ] **Step 4: Run clippy** + +Run: + +```bash +cargo clippy --workspace --all-targets --all-features -- -D warnings +``` + +Expected: no warnings. + +- [ ] **Step 5: Run feature compile check** + +Run: + +```bash +cargo check --workspace --all-targets --features "fastly cloudflare spin" +``` + +Expected: compile succeeds. + +- [ ] **Step 6: Run Spin target check** + +Run: + +```bash +cargo check -p edgezero-adapter-spin --target wasm32-wasip2 --features spin +``` + +Expected: compile succeeds. + +- [ ] **Step 7: Commit final cleanup** + +```bash +git add crates/edgezero-adapter-fastly/src/chunked_config.rs crates/edgezero-adapter-fastly/src/cli.rs +git commit -m "test(fastly): verify chunk gc behavior" +``` + +--- + +## Notes for Implementers + +- Do not delete root keys. Only delete keys returned by `prior_chunk_keys(root_key, raw)` after subtracting the root's new keep-set. +- Source root keys from the caller's logical `entries` (cloud) or the threaded `roots: &[&str]` param (local). Never infer roots by string-matching `CHUNK_KEY_INFIX` on physical keys — `--key` is free-form and may itself contain the infix. +- `prior_chunk_keys` parses `serde_json::Value` first: non-pointer-kind values are silent `Ok([])`; a value that IS pointer-kind but malformed/invalid must `Err` (warn), not be silently skipped. +- Do not make cloud dry-run resolve the store id or call `fastly`. +- Do not change `edgezero_adapter::registry::Adapter`; raw prior pointer reads are Fastly-specific. +- Do not use prefix scans of the remote store. The only remote reads are per-root `describe` calls. +- Use exact `toml_edit::Table` key operations for local dotted chunk keys. +- If a push commit fails, do not sweep anything. The old root pointer may still be live. +- If GC fails, return `Ok` with warning status lines. The push success criterion remains new entries committed. From 93a5886a25140134f84cdd74b13ae3908b7eb6cb Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 7 Jul 2026 23:29:00 -0700 Subject: [PATCH 05/25] Self-review fixes (spec+plan): expand new_keys in local dry-run so identical re-push counts 0 (was over-counting); enumerate all 10 writer call sites + roots args; forbid --all on delete; reword failed-delete warnings as informational (inert, future config gc); note sequential-spawn latency + approximate line numbers --- .../plans/2026-07-07-fastly-chunk-gc.md | 74 ++++++++++++++----- .../specs/2026-07-07-fastly-chunk-gc.md | 39 ++++++++-- 2 files changed, 88 insertions(+), 25 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index f8cec88f..1f814c47 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -367,6 +367,14 @@ assert!(combined.contains("orphan chunks")); Also keep `assert_eq!(after, original)`. +Add a second dry-run test for the identical-bytes re-push (regression +for MEDIUM-A / the expand-`new_keys` rule): seed a chunked config, then +dry-run the SAME bytes and assert the count is `0`: + +```rust +assert!(combined.contains("would delete 0 orphan chunks")); +``` + - [ ] **Step 4: Run local tests to verify failures** Run: @@ -393,13 +401,27 @@ fn write_fastly_local_config_store( ) -> Result, String> ``` -Update ALL existing call sites to pass `roots`. The adapter caller -(`push_config_entries_local`) passes the logical entry keys -(`&entries.iter().map(|(k, _)| k.as_str()).collect::>()`). -Setup-only writes in the unit-test module pass the store id they seed -(e.g. `&[TEST_CONFIG_ID]`), or `&[]` where the test does not exercise -GC. Existing direct callers that only `.expect("write")` can ignore the -returned `Vec`. +Update ALL existing call sites to pass `roots`. There are **10** (line +numbers approximate — grep `write_fastly_local_config_store` to confirm): + +| Site | Kind | `roots` to pass | +| --- | --- | --- | +| `cli.rs:483` | prod (`push_config_entries_local`) | logical entry keys (see caller snippet below) | +| `cli.rs:1838` | test — minimal-file block | `&[TEST_CONFIG_ID]` | +| `cli.rs:1867`, `:1873` | test — replaces-block-on-re-push | `&[TEST_CONFIG_ID]` (both writes) | +| `cli.rs:1902` | test — preserves-unrelated-blocks | `&[TEST_CONFIG_ID]` | +| `cli.rs:1931` | test — creates-file-when-missing | `&[TEST_CONFIG_ID]` | +| `cli.rs:2366` | test | the store id it seeds, or `&[]` if GC is irrelevant | +| `cli.rs:3243` | test | the store id it seeds, or `&[]` | +| `cli.rs:3275` | test — local chunked roundtrip | `&[TEST_CONFIG_ID]` | + +Pass `&[]` only where the test does not exercise GC and the extra prune +would perturb its assertions; otherwise pass the seeded root(s). The +adapter caller (`push_config_entries_local`) passes the logical entry +keys (see snippet below). Existing direct callers that only +`.expect("write")` can ignore the returned `Vec` (the return +type changes from `Result<(), _>` to `Result, _>`, which is +source-compatible with `.expect(...)`). Inside the function: @@ -467,23 +489,32 @@ fn local_orphan_counts_for_dry_run( entries: &[(String, String)], // logical entries; roots are their keys ) -> Vec<(String, Result)> { // Roots are the logical entry keys (NOT inferred from physical keys). + // For each logical (root_key, body): + // new_keys = keys of prepare_fastly_config_entries(root_key, body) ∪ {root_key} + // — MUST expand here; using the logical key alone would over-count an + // identical-bytes re-push (new chunk keys would be missing from + // new_keys, so every prior chunk would look like an orphan). + // old_raw = contents table value at root_key (if present) + // count = orphan_chunk_keys({prior_chunk_keys(root, old_raw), new_keys}).len() // Missing file, missing/absent contents table, or a root with no prior // pointer / a direct prior value => Ok(0) per root. // Unreadable file, malformed TOML, contents-not-a-table, or a root // value that is not a string => Err("could not read prior state"). // prior_chunk_keys(root, old_raw) returning Err (suspicious pointer) - // => Err(). - // Use prior_chunk_keys(root, old_raw), new_key_set_for_root, and orphan_chunk_keys. + // => Err("suspicious prior pointer"). } ``` This helper MUST NOT fail the dry-run: it returns a per-root `Result` that the caller renders as a count or an `unknown (...)` line. The current dry-run reads no file at all, so introducing this read must not -turn a previously-succeeding dry-run into an error. Use exact -`toml_edit` table access, not string scanning. Keep this helper small; -if it gets large, split only into `local_contents_table` and count -computation. +turn a previously-succeeding dry-run into an error. Note it expands each +logical entry via `prepare_fastly_config_entries` to derive the true +`new_keys` — the same expansion the real push performs — so the count +matches what the push would actually delete (0 for an identical +re-push). Use exact `toml_edit` table access, not string scanning. Keep +this helper small; if it gets large, split only into +`local_contents_table` and count computation. - [ ] **Step 7: Wire local dry-run count output** @@ -495,11 +526,12 @@ out.push(format!( )); ``` -For an `Err`, emit: +For an `Err`, emit (wording matches the spec's local dry-run bullet — +`{reason}` is `could not read prior state` or `suspicious prior pointer`): ```rust out.push(format!( - " would delete unknown orphan chunks from the previous generation of `{key}` ({err})" + " would delete an unknown number of orphan chunks from the previous generation of `{key}` (unknown: {reason})" )); ``` @@ -581,7 +613,9 @@ Expected: new GC tests fail because no deletes or warning lines exist yet. - [ ] **Step 4: Add `delete_config_store_entry`** -Add near `create_config_store_entry`: +Add near `create_config_store_entry`. NOTE: only ever pass `--key`. +The delete subcommand also accepts `-a/--all`, which wipes **every** +entry in the store — never construct that flag here. ```rust fn delete_config_store_entry(store_id: &str, key: &str) -> Result<(), String> { @@ -656,8 +690,12 @@ for plan in &gc_plans { Ok(keys) => { for key in keys { if let Err(err) = delete_config_store_entry(&resolved_id, &key) { + // Informational: the push already succeeded and the new + // pointer is live, so this orphan is inert, just not + // reclaimed. It will be cleaned by a future `config gc` + // command; do NOT tell the operator to retry the push. warnings.push(format!( - "warning: failed to delete orphan chunk `{key}` for `{}`: {err}", + "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be removed by a future `config gc`", plan.root_key )); } @@ -789,3 +827,5 @@ git commit -m "test(fastly): verify chunk gc behavior" - Use exact `toml_edit::Table` key operations for local dotted chunk keys. - If a push commit fails, do not sweep anything. The old root pointer may still be live. - If GC fails, return `Ok` with warning status lines. The push success criterion remains new entries committed. +- Cloud GC spawns one `fastly` process per orphan, sequentially. A large prior generation (e.g. 50 chunks) means ~50 sequential shell-outs after the push — a known wall-clock cost. Fastly has no bulk delete-by-key (`--all` is store-wide and unusable), so batching is not available; do not add parallel spawns in v1 without measuring. +- Line numbers in this plan are approximate anchors against `main`; `grep` the named function/test before editing, since prior edits shift offsets. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index b9b83800..4a272155 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -216,9 +216,18 @@ handled for free: the new value is a direct envelope, shelling `fastly config-store-entry delete --store-id= --key= --auto-yes`, mirroring the spawn/stderr handling of `create_config_store_entry` (`cli.rs:1073`). `--auto-yes` suppresses - any interactive confirmation (non-interactive shell-out). Treat a + any interactive confirmation (non-interactive shell-out). Only ever + pass `--key`: the subcommand also accepts `-a/--all`, which deletes + **every** entry in the store — never construct that flag. Treat a "not found" / "does not exist" / "404" stderr as success (already gone). +- **A failed delete is informational, not actionable.** The push has + already succeeded and the new pointer is live, so an un-deleted orphan + is inert. Word the warning accordingly (e.g. "could not reclaim orphan + chunk … ; it is inert and will be removed by a future `config gc`") — + do NOT imply the operator should retry the push. There is no reclaim + command in v1, so such an orphan persists as a pre-existing leak (see + Non-goals) until that future tool ships. - **Sweep after the whole commit.** The cloud writer flattens every logical root's physical entries into one `physical_entries` vec (`cli.rs:380`) and commits them in a single `push_entries_with_committer` @@ -234,7 +243,11 @@ handled for free: the new value is a direct envelope, whichever pointer actually committed. Reclamation waits for the next successful push. Leak-safe over corrupt-safe, per invariant 4. - **Cost**: one extra `describe` per logical root before the push, plus - one `delete` per orphan after. No store-wide `list` call. + one `delete` per orphan after. No store-wide `list` call. Deletes are + sequential `fastly` subprocess spawns — a large prior generation + (e.g. 50 chunks) adds ~50 sequential shell-outs of post-push latency. + Fastly exposes no bulk delete-by-key (`--all` is store-wide), so v1 + accepts this cost rather than parallelising spawns. ## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) @@ -263,8 +276,14 @@ threaded explicitly rather than inferred from the flattened entries: warnings are suspicious-pointer ones. - **Dry-run (offline, best-effort count).** `push_config_entries_local`'s dry-run (`cli.rs:459`) reads no remote state. Extend it to read the - current `fastly.toml`, compute `prior_chunk_keys` per root, and report - the orphan count, e.g. + current `fastly.toml` and, per root, compute the orphan count as + `prior_chunk_keys(root, old_value) − new_keys`, where `new_keys` is + the **expanded** key set (`prepare_fastly_config_entries(root, body)` + keys ∪ `{root}`) — the same expansion the real push does. Expanding + is required: using the logical key alone would over-count an + identical-bytes re-push (the new chunk keys would be missing from + `new_keys`, making every prior chunk look like an orphan; the correct + answer is `0`). Report e.g. `" would delete 9 orphan chunks from the previous generation of `app_config`"`. Error semantics — the dry-run MUST NOT newly fail where it succeeds today (the current dry-run does not read the file at all), so GC @@ -273,11 +292,12 @@ threaded explicitly rather than inferred from the flattened entries: → report `0`. - File present but unreadable, malformed TOML, `contents` not a table, or a root's value not a string → report - `"unknown (could not read prior state)"` for that root and continue. - (The real push still fails fatally on malformed TOML via the writer - at `cli.rs:938`; only the dry-run *count* degrades.) + `"would delete an unknown number of orphan chunks from the previous generation of `app_config` (unknown: could not read prior state)"` + for that root and continue. (The real push still fails fatally on + malformed TOML via the writer at `cli.rs:938`; only the dry-run + *count* degrades.) - Prior value is pointer-kind-but-invalid (`prior_chunk_keys` → `Err`) - → report `"unknown (suspicious prior pointer)"`. + → report the same line with `(unknown: suspicious prior pointer)`. ## Existing tests that MUST change @@ -346,6 +366,9 @@ in the `cli.rs` test module. correctly — its own prior chunks are swept, siblings untouched. - **Dry-run count**: reports the correct orphan count and writes nothing. +- **Dry-run identical re-push counts 0**: seed a chunked config, then + dry-run the SAME bytes → reports `0` orphans (regression for computing + `new_keys` from the expanded, not logical, entries). - **Dry-run degrades, never fails**: a malformed `fastly.toml` makes the dry-run report `unknown` for the GC count while still printing the direct-vs-chunked intent lines and returning `Ok`. From 1089b5bcc319907fdcc5ace155aaa85430ad4e2f Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 07:41:41 -0700 Subject: [PATCH 06/25] Review v4 (spec+plan): close cloud GC concurrency hole via post-commit root read-back guard (invariant 5); build keep-sets from per-root expand_root instead of prefix-scanning flattened entries; make reserved-infix --key rejection mandatory at the Fastly adapter boundary + flip the infix test to expect rejection; add local suspicious-pointer real-push test and cloud concurrency-guard test --- .../plans/2026-07-07-fastly-chunk-gc.md | 229 ++++++++++++------ .../specs/2026-07-07-fastly-chunk-gc.md | 124 +++++++--- 2 files changed, 246 insertions(+), 107 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index 1f814c47..70c3a462 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -18,18 +18,22 @@ - Modify `crates/edgezero-adapter-fastly/src/cli.rs` - Import `prior_chunk_keys` and `CHUNK_KEY_INFIX`. - - Add small private GC metadata helpers (`FastlyConfigGcPlan`, `new_key_set_for_root`, `orphan_chunk_keys`) near the config-push helpers. Roots are sourced from logical `entries`/threaded `roots`, never inferred from physical keys. + - Add small private GC helpers (`FastlyConfigGcPlan`, `expand_root`, `orphan_chunk_keys`, `reject_reserved_root_keys`) near the config-push helpers. Per-root keep-sets come from `expand_root` (that root's own expansion), never prefix-scanned from the flattened physical set. - Extend `push_config_entries` with offline dry-run intent, prior root read, post-commit cloud sweep (only after the whole flattened commit returns `Ok`), and warning status lines. - Add `delete_config_store_entry(store_id, key)`. - Extend `push_config_entries_local` dry-run with best-effort offline orphan counts (degrade to `unknown`, never newly fail). - - Extend `write_fastly_local_config_store` with a `roots: &[&str]` param; prune prior root chunks in the same in-memory rewrite before the (non-atomic) `fs::write` and return warning lines from suspicious prior pointers. + - Extend `write_fastly_local_config_store` with a `gc_roots: &[(String, HashSet)]` param (exact per-root keep-sets; empty ⇒ no GC); prune prior root chunks in the same in-memory rewrite before the (non-atomic) `fs::write` and return warning lines from suspicious prior pointers. - Add/adjust tests in the existing `cli.rs` test module. -Optional (defence-in-depth, spec "Files touched"): reject a `--key` -containing `.__edgezero_chunks.` at push time in `edgezero-cli` -(`config.rs:297` / `args.rs:311`), since such a key collides with the -generated chunk namespace. Not required for GC correctness (roots are -threaded, not inferred); skip if it widens scope. +**Mandatory** reserved-key rejection: both `push_config_entries` and +`push_config_entries_local` call `reject_reserved_root_keys(entries)?` +immediately after the empty-entries check. A logical key containing +`.__edgezero_chunks.` is rejected with an error (it collides with the +generated chunk namespace). This lives at the Fastly adapter boundary — +not in generic `edgezero-cli` — because the infix is a Fastly concept. +It is a hard error, not a warning: there is no valid reason for such a +key, and allowing it would let a push write into another key's chunk +namespace. --- @@ -215,32 +219,35 @@ use crate::chunked_config::{ - [ ] **Step 2: Add small metadata structs/helpers near config-push helpers** -Add before `push_entries_with_committer`. Note there is **no** -`root_keys_from_physical_entries` inference helper: `--key` is a -free-form `Option` (`args.rs:311`), so a logical root key may -itself contain `.__edgezero_chunks.`. Root keys are always sourced from -the caller's **logical** `entries` (cloud) or the threaded `roots` param -(local) — never string-matched out of the flattened physical entries. +Add before `push_entries_with_committer`. There is **no** prefix-scan +helper: `--key` is free-form (`args.rs:311`), so `new_keys` must NOT be +derived by string-matching flattened physical entries (that reintroduces +inference). Instead, expand each logical root **on its own** and take +that root's exact keys. ```rust +use std::collections::HashSet; + #[derive(Debug)] struct FastlyConfigGcPlan { root_key: String, prior_keys: Result, String>, - new_keys: std::collections::HashSet, + new_keys: HashSet, + /// Exactly the value this push wrote at `root_key` (the root pointer + /// for a chunked push, or the direct envelope). Cloud uses it as a + /// read-back concurrency guard before deleting; local ignores it. + new_root_value: String, } -fn new_key_set_for_root(root_key: &str, entries: &[(String, String)]) -> std::collections::HashSet { - entries - .iter() - .filter_map(|(key, _)| { - if key == root_key || key.starts_with(&format!("{root_key}{CHUNK_KEY_INFIX}")) { - Some(key.clone()) - } else { - None - } - }) - .collect() +/// Expand ONE logical (root_key, body) into its physical entries, the +/// exact keep-set for that root, and the value written at the root key. +/// No cross-root prefix scanning. +fn expand_root(root_key: &str, body: &str) -> Result<(Vec<(String, String)>, HashSet, String), String> { + let expanded = prepare_fastly_config_entries(root_key, body)?; + let new_keys: HashSet = expanded.iter().map(|(k, _)| k.clone()).collect(); + // prepare_* always emits the root entry LAST (root pointer or direct value). + let new_root_value = expanded.last().map(|(_, v)| v.clone()).unwrap_or_default(); + Ok((expanded, new_keys, new_root_value)) } fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { @@ -253,9 +260,24 @@ fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { Err(err) => Err(err.clone()), } } + +/// Enforced at the adapter boundary because `--key` is free-form: a key +/// containing the reserved chunk infix would collide with chunk storage. +fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> { + for (key, _) in entries { + if key.contains(CHUNK_KEY_INFIX) { + return Err(format!( + "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different --key" + )); + } + } + Ok(()) +} ``` -Keep helpers private. If `clippy` complains about line length or type imports, pull `HashSet` into the top-level imports instead. +Both `push_config_entries` and `push_config_entries_local` MUST call +`reject_reserved_root_keys(entries)?` immediately after the empty-entries +check, before any expansion or I/O. Keep helpers private. - [ ] **Step 3: Run compile check** @@ -375,6 +397,22 @@ dry-run the SAME bytes and assert the count is `0`: assert!(combined.contains("would delete 0 orphan chunks")); ``` +- [ ] **Step 3c: Add local suspicious-pointer real-push test (spec §"local")** + +Seed `fastly.toml` so the root holds a pointer-kind-but-invalid value +(e.g. `{"edgezero_kind":"fastly_config_chunks","version":2}`), then run a +REAL (non-dry-run) local push of a new config. Assert: +- the returned status `Vec` contains a suspicious-pointer warning, +- no chunk keys were removed as a side effect (the writer must not delete + when `prior_chunk_keys` returns `Err`), +- the new value is written to the root key. + +- [ ] **Step 3d: Add reserved-key rejection test (local)** + +Assert `push_config_entries_local` with a logical key containing +`.__edgezero_chunks.` returns `Err` before touching `fastly.toml` (file +unchanged / not created). + - [ ] **Step 4: Run local tests to verify failures** Run: @@ -388,57 +426,58 @@ Expected: failures showing old chunks remain and dry-run lacks GC count. - [ ] **Step 5: Implement local in-memory pruning** -Change `write_fastly_local_config_store` to (a) accept the **logical -root keys** explicitly via a new `roots: &[&str]` param and (b) return -warning lines, while preserving its error behavior: +Change `write_fastly_local_config_store` to (a) accept the exact per-root +keep-sets via a new `gc_roots: &[(String, HashSet)]` param +(each `(root_key, new_keys)` from `expand_root` — NOT prefix-scanned) and +(b) return warning lines, while preserving its error behavior: ```rust fn write_fastly_local_config_store( path: &Path, platform_name: &str, entries: &[(String, String)], - roots: &[&str], + gc_roots: &[(String, std::collections::HashSet)], ) -> Result, String> ``` -Update ALL existing call sites to pass `roots`. There are **10** (line +Update ALL existing call sites to pass `gc_roots`. There are **10** (line numbers approximate — grep `write_fastly_local_config_store` to confirm): -| Site | Kind | `roots` to pass | +| Site | Kind | `gc_roots` to pass | | --- | --- | --- | -| `cli.rs:483` | prod (`push_config_entries_local`) | logical entry keys (see caller snippet below) | -| `cli.rs:1838` | test — minimal-file block | `&[TEST_CONFIG_ID]` | -| `cli.rs:1867`, `:1873` | test — replaces-block-on-re-push | `&[TEST_CONFIG_ID]` (both writes) | -| `cli.rs:1902` | test — preserves-unrelated-blocks | `&[TEST_CONFIG_ID]` | -| `cli.rs:1931` | test — creates-file-when-missing | `&[TEST_CONFIG_ID]` | -| `cli.rs:2366` | test | the store id it seeds, or `&[]` if GC is irrelevant | -| `cli.rs:3243` | test | the store id it seeds, or `&[]` | -| `cli.rs:3275` | test — local chunked roundtrip | `&[TEST_CONFIG_ID]` | - -Pass `&[]` only where the test does not exercise GC and the extra prune -would perturb its assertions; otherwise pass the seeded root(s). The -adapter caller (`push_config_entries_local`) passes the logical entry -keys (see snippet below). Existing direct callers that only -`.expect("write")` can ignore the returned `Vec` (the return +| `cli.rs:483` | prod (`push_config_entries_local`) | `&gc_roots` built via `expand_root` (see caller snippet below) | +| `cli.rs:1838` | test — minimal-file block | `&[]` (setup-only; no prior generation to prune) | +| `cli.rs:1867`, `:1873` | test — replaces-block-on-re-push | `&[]` (both writes; asserts value replacement, not GC) | +| `cli.rs:1902` | test — preserves-unrelated-blocks | `&[]` | +| `cli.rs:1931` | test — creates-file-when-missing | `&[]` | +| `cli.rs:2366` | test | `&[]` unless it asserts GC | +| `cli.rs:3243` | test | `&[]` unless it asserts GC | +| `cli.rs:3275` | test — local chunked roundtrip | `&[]` (read-back roundtrip, not GC) | + +Setup-only writes pass `&[]` (empty ⇒ the writer skips the snapshot/prune +loop entirely, so behaviour is unchanged for those tests). Only tests +that assert pruning build a real `gc_roots`. Existing direct callers that +only `.expect("write")` can ignore the returned `Vec` (return type changes from `Result<(), _>` to `Result, _>`, which is source-compatible with `.expect(...)`). Inside the function: -1. Snapshot prior state for the **threaded roots** before mutation - (no inference — the roots are given): +1. Snapshot prior state using the **given exact keep-sets** before + mutation (no inference, no prefix scan): ```rust -let mut plans = Vec::with_capacity(roots.len()); -for root_key in roots { +let mut plans = Vec::with_capacity(gc_roots.len()); +for (root_key, new_keys) in gc_roots { let prior_keys = contents_tbl .get(root_key) .and_then(toml_edit::Item::as_str) .map_or_else(|| Ok(Vec::new()), |raw| prior_chunk_keys(root_key, raw)); plans.push(FastlyConfigGcPlan { - root_key: (*root_key).to_owned(), + root_key: root_key.clone(), prior_keys, - new_keys: new_key_set_for_root(root_key, entries), + new_keys: new_keys.clone(), + new_root_value: String::new(), // unused locally (no remote concurrency) }); } ``` @@ -466,8 +505,16 @@ Return `Ok(warnings)` after `fs::write`. Suspicious prior pointers must not fail In `push_config_entries_local`, capture the warning vector and append it after the existing success line: ```rust -let roots: Vec<&str> = entries.iter().map(|(key, _)| key.as_str()).collect(); -let warnings = write_fastly_local_config_store(&fastly_path, name, &physical_entries, &roots)?; +// Expand each logical root once: flatten for the write, keep exact per-root +// keep-sets for GC (no prefix scan of the flattened set). +let mut physical_entries: Vec<(String, String)> = Vec::new(); +let mut gc_roots: Vec<(String, std::collections::HashSet)> = Vec::with_capacity(entries.len()); +for (root_key, body) in entries { + let (expanded, new_keys, _new_root) = expand_root(root_key, body)?; + physical_entries.extend(expanded); + gc_roots.push((root_key.clone(), new_keys)); +} +let warnings = write_fastly_local_config_store(&fastly_path, name, &physical_entries, &gc_roots)?; let mut out = vec![format!( "wrote {} physical entries ({} logical) to `[local_server.config_stores.{name}.contents]` in {} (logical id `{logical}`); restart `fastly compute serve` to pick up changes", physical_entries.len(), @@ -571,6 +618,11 @@ Add a Unix-only helper in the test module that: - Responds to `config-store list --json`. - Responds to `config-store-entry describe --key=` from a key-response map. +- For the concurrency-guard test, can serve the root `describe` + **sequentially** — first the prior pointer (pre-commit read), then a + different value (post-commit read-back) — e.g. via an invocation + counter file so successive `describe ` calls return different + bodies. - Accepts `config-store-entry update --upsert --stdin` and logs `update `. - Accepts `config-store-entry delete --key= --auto-yes` and logs `delete `. - Can be configured so specific delete keys fail with stderr. @@ -589,7 +641,10 @@ delete app_config.__edgezero_chunks..0 Add tests for: -- Prior chunks are deleted and new/root keys are not. +- Prior chunks are deleted and new/root keys are not. (The fake serves the + root `describe` twice: the prior pointer pre-commit, then the + newly-written root value on read-back, so the concurrency guard passes + and deletes proceed.) - Shrink-to-direct deletes all prior chunks. - No prior value produces zero deletes. - Identical re-push produces zero deletes. @@ -597,8 +652,18 @@ Add tests for: - Prior read failure returns `Ok` with a warning and no deletes. - Suspicious pointer version returns `Ok` with a warning and no deletes. - Delete operations occur after root update. +- Concurrency guard: when the post-commit root re-read differs from what + this push wrote (fake `fastly` returns a different root value on the + second `describe`), GC is skipped with a "root changed" warning and + **no** deletes are issued. +- Reserved key: `--key` containing `.__edgezero_chunks.` is rejected + before any `fastly` invocation (no `list`/`describe`/`update`/`delete`). - Dry-run stays offline and emits no-count GC intent. +Note the concurrency test needs the fake `fastly` to serve the root +`describe` TWICE with different values: the prior pointer (pre-commit +read) and then a mismatching value (post-commit read-back). + Use `path_mutation_guard()` for tests that prepend a fake `fastly`. - [ ] **Step 3: Run cloud tests to verify failures** @@ -660,40 +725,49 @@ In `push_config_entries`, after `resolved_id`: ```rust let mut gc_plans = Vec::with_capacity(entries.len()); let mut warnings = Vec::new(); -for (root_key, _) in entries { +for (root_key, body) in entries { + // Exact per-root keep-set + the value we will write (no prefix scan). + let (_, new_keys, new_root_value) = expand_root(root_key, body)?; let prior_keys = match fetch_remote_config_store_entry(&resolved_id, root_key) { Ok(Some(raw)) => prior_chunk_keys(root_key, &raw), Ok(None) => Ok(Vec::new()), - Err(err) => { - Err(format!( - "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" - )) - } + Err(err) => Err(format!( + "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" + )), }; gc_plans.push(FastlyConfigGcPlan { root_key: root_key.clone(), prior_keys, - new_keys: new_key_set_for_root(root_key, &physical_entries), + new_keys, + new_root_value, }); } ``` -If `prior_chunk_keys` returns `Err`, push a warning and keep the `Err` in the plan so deletes are skipped. +If `prior_chunk_keys` returns `Err`, keep the `Err` in the plan so +deletes are skipped (the sweep in Step 6 turns it into a warning). - [ ] **Step 6: Sweep after successful commit** -Keep the existing `push_entries_with_committer` call. Only after it returns `Ok`, iterate plans: +Keep the existing `push_entries_with_committer` call. Only after it +returns `Ok`, iterate plans. Before deleting a root's orphans, **re-read +the raw root and confirm it still equals exactly what this push wrote** +(`plan.new_root_value`). This guards against a concurrent push that +overwrote the root between our commit and our sweep — e.g. reverting it +to the prior pointer, which would make the "orphans" live again: ```rust for plan in &gc_plans { - match orphan_chunk_keys(plan) { - Ok(keys) => { + let keys = match orphan_chunk_keys(plan) { + Ok(keys) if !keys.is_empty() => keys, + Ok(_) => continue, // nothing to reclaim + Err(err) => { warnings.push(format!("warning: {err}")); continue } + }; + // Concurrency guard: only sweep if the root still holds our write. + match fetch_remote_config_store_entry(&resolved_id, &plan.root_key) { + Ok(Some(current)) if current == plan.new_root_value => { for key in keys { if let Err(err) = delete_config_store_entry(&resolved_id, &key) { - // Informational: the push already succeeded and the new - // pointer is live, so this orphan is inert, just not - // reclaimed. It will be cleaned by a future `config gc` - // command; do NOT tell the operator to retry the push. warnings.push(format!( "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be removed by a future `config gc`", plan.root_key @@ -701,14 +775,23 @@ for plan in &gc_plans { } } } + Ok(_) => warnings.push(format!( + "note: skipped chunk GC for `{}`: root changed since this push wrote it (concurrent push?); orphans left for a future `config gc`", + plan.root_key + )), Err(err) => warnings.push(format!( - "warning: {err}" + "note: skipped chunk GC for `{}`: could not re-read root before sweep ({err}); orphans left for a future `config gc`", + plan.root_key )), } } ``` -Return the existing success line plus warning lines. +This narrows but does not fully close the window (a writer could still +intervene between the read-back and an individual delete) — Fastly has no +compare-and-delete. That residual is acceptable under invariant 4: +GC is best-effort and the guard eliminates the realistic revert-to-prior +corruption. Return the existing success line plus warning lines. - [ ] **Step 7: Keep cloud dry-run offline** @@ -819,7 +902,7 @@ git commit -m "test(fastly): verify chunk gc behavior" ## Notes for Implementers - Do not delete root keys. Only delete keys returned by `prior_chunk_keys(root_key, raw)` after subtracting the root's new keep-set. -- Source root keys from the caller's logical `entries` (cloud) or the threaded `roots: &[&str]` param (local). Never infer roots by string-matching `CHUNK_KEY_INFIX` on physical keys — `--key` is free-form and may itself contain the infix. +- Source root keys and keep-sets from the caller's logical `entries` via `expand_root` (each root expanded on its own); pass exact per-root keep-sets to the local writer as `gc_roots`. Never infer roots or keep-sets by string-matching `CHUNK_KEY_INFIX` on the flattened physical set — `--key` is free-form and may itself contain the infix (and is separately rejected via `reject_reserved_root_keys`). - `prior_chunk_keys` parses `serde_json::Value` first: non-pointer-kind values are silent `Ok([])`; a value that IS pointer-kind but malformed/invalid must `Err` (warn), not be silently skipped. - Do not make cloud dry-run resolve the store id or call `fastly`. - Do not change `edgezero_adapter::registry::Adapter`; raw prior pointer reads are Fastly-specific. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 4a272155..83b37ec3 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -63,12 +63,16 @@ pointer no longer does. either a direct `BlobEnvelope` or a chunk pointer. Never GC'd. The root key is the logical store-config id, or the operator's `--key ` — a free-form `Option` (`args.rs:311`, - used directly in `config.rs:297`). Because `--key` is **not** - validated today, GC does NOT infer roots by string-matching the chunk - infix; it threads the logical root keys explicitly (see Local path). - As defence-in-depth, the implementation SHOULD also reject a `--key` - containing the reserved infix `.__edgezero_chunks.` at push time, - since such a key would collide with the generated chunk namespace. + used directly in `config.rs:297`). GC never infers roots by + string-matching the chunk infix; roots come from the caller's logical + entries. In addition, the Fastly adapter **MUST reject** (hard error, + not warning) any logical key containing the reserved infix + `.__edgezero_chunks.`, at the top of both `push_config_entries` and + `push_config_entries_local`, before any expansion or I/O. Such a key + collides with the generated chunk namespace and has no valid use; a + push must not be able to write into another key's chunk space. The + check lives at the Fastly adapter boundary (the infix is a + Fastly-specific concept), not in generic `edgezero-cli`. - **Previous pointer**: the value stored at the root key *before* this push overwrites it. - **Prior chunk keys**: the keys named by `previous_pointer.chunks[]`, @@ -76,7 +80,12 @@ pointer no longer does. Empty if the previous value was a direct envelope, missing, or not a valid v1 pointer. - **New keep-set**: the physical keys this push writes for that root — - the new chunk keys plus the root key itself. + the new chunk keys plus the root key itself. It is built by expanding + **that root's own** `(root_key, body)` via + `prepare_fastly_config_entries` and taking those keys — NOT by + prefix-scanning the flattened multi-root physical set (which would + reintroduce infix inference and mis-handle shared prefixes or a + free-form key). - **Orphans**: `prior_chunk_keys − new_keep_set`. In practice this is all prior chunk keys, because a changed config changes the SHA and therefore every chunk key; but the set-difference is the correct, @@ -123,6 +132,21 @@ pointer no longer does. warning folded into the returned status lines; the push still reports success. A leaked chunk is harmless; a failed push is not. +5. **Cloud deletes are guarded by a post-commit root read-back.** A + concurrent push could overwrite the root between our commit and our + sweep — e.g. revert it to the prior pointer, making the "orphans" + live again. So before deleting a root's orphans, cloud GC re-reads + the raw root and proceeds only if it still equals *exactly* the value + this push wrote (`new_root_value`). If it differs (or the re-read + fails), GC is skipped for that root with a warning. This narrows but + does not fully close the window (no compare-and-delete exists in + Fastly; a writer could still intervene between the read-back and an + individual delete) — acceptable under invariant 4, since the guard + eliminates the realistic revert-to-prior corruption and the residual + only risks a leak, never a live delete in the common case. The local + path holds the whole file in one rewrite and has no equivalent + remote-concurrency window. + ## `prior_chunk_keys` helper (`chunked_config.rs`) Add next to the existing `FastlyChunkPointer` schema @@ -173,24 +197,33 @@ with missing `chunks` and `version:2` → `Err` (NOT silently `Ok([])`); ## Algorithm -For each logical `(root_key, envelope_json)` entry in the push: +# --- validate first (both paths), before any expansion or I/O --- +reject the whole push if ANY logical key contains CHUNK_KEY_INFIX # hard error + +For each logical `(root_key, body)` entry in the push: ``` # --- before writing --- -prev = read_root_value(root_key) # may be absent -prior = prior_chunk_keys(root_key, prev) # Ok([]) unless prev is a valid v1 pointer -expanded = prepare_fastly_config_entries(root_key, envelope_json) -new_keys = { k for (k, _) in expanded } ∪ { root_key } +prev = read_root_value(root_key) # may be absent +prior = prior_chunk_keys(root_key, prev) # Ok([]) unless prev is a valid v1 pointer +expanded = prepare_fastly_config_entries(root_key, body) # THIS root only +new_keys = { k for (k, _) in expanded } # includes root_key (its last entry) +new_root_value = value of expanded.last() # exactly what we write at root_key # --- write (existing behaviour, unchanged) --- push expanded # chunks first, root pointer last (commit point) -# --- sweep (new, best-effort, only after push succeeds) --- +# --- sweep (best-effort, only after the WHOLE push succeeds) --- match prior: Err(msg): warn(msg) # suspicious pointer; skip GC Ok(keys): - for k in keys − new_keys: - try delete_entry(k) except e: warn("failed to delete orphan chunk `{k}`: {e}") + orphans = keys − new_keys + if orphans not empty: + # cloud only — read-back concurrency guard (invariant 5): + if read_root_value(root_key) != new_root_value: + warn("root `{root_key}` changed since this push wrote it; skipping GC"); continue + for k in orphans: + try delete_entry(k) except e: warn("could not reclaim orphan `{k}`: {e}") ``` The **shrink-to-direct** case (config drops back under 8 000 chars) is @@ -236,6 +269,13 @@ handled for free: the new value is a direct envelope, `Ok` — i.e. after ALL roots' chunks + pointers are committed. Fold delete failures and prior-read/parse warnings into the returned status `Vec` (the push still returns `Ok`). +- **Read-back concurrency guard (invariant 5).** Each root's GC record + carries `new_root_value` — the exact value written at the root key + (its new pointer, or the direct envelope; i.e. the last expanded + entry's value). In the post-commit sweep, before deleting a root's + orphans, re-read the raw root via `fetch_remote_config_store_entry` + and delete only if it equals `new_root_value`; otherwise warn ("root + changed since this push wrote it") and skip that root. - **Partial-commit failure ⇒ no GC.** If the committer fails partway, `push_config_entries` returns `Err` and no sweep runs for any root. Safe by construction: nothing is deleted, so the store is left with @@ -253,19 +293,22 @@ handled for free: the new value is a direct envelope, The local `contents` table is held entirely in memory in a single `DocumentMut`, so GC here is fully reliable and needs no round-trips. -To stay sound against a free-form `--key`, the logical root keys are -threaded explicitly rather than inferred from the flattened entries: - -- **Thread logical roots (no inference).** `push_config_entries_local` - already holds the logical `entries: &[(String, String)]`. Pass the - logical root keys into `write_fastly_local_config_store` via a new - parameter (e.g. `roots: &[&str]`) alongside `physical_entries`. No - string-matching on the chunk infix — a `--key` that happens to - contain `.__edgezero_chunks.` cannot mislead root detection. +Per-root keep-sets are computed from each root's own expansion and +passed to the writer explicitly — never inferred from the flattened set: + +- **Pass exact per-root keep-sets (no inference).** + `push_config_entries_local` expands each logical `(root_key, body)` via + `prepare_fastly_config_entries` (once, reused for `physical_entries`) + and passes `gc_roots: &[(String, HashSet)]` — each root with + its own exact new-key set — into `write_fastly_local_config_store` + alongside `physical_entries`. No string-matching on the chunk infix; a + `--key` containing `.__edgezero_chunks.` is separately rejected up + front (Terminology). An empty `gc_roots` means "no GC" (setup-only + writers pass `&[]`). - **Sweep inside the single rewrite.** In `write_fastly_local_config_store` - (`cli.rs:926`), before the upsert loop, snapshot each threaded root's - *old* value from the existing `contents_tbl`. After inserting the new - physical entries, for each root compute + (`cli.rs:926`), before the upsert loop, snapshot each root's *old* + value from the existing `contents_tbl`. After inserting the new + physical entries, for each `(root, new_keys)` compute `prior_chunk_keys(root, old_value)` and `contents_tbl.remove(k)` for each orphan `k` in `prior − new_keys`. All within the one `DocumentMut` that the trailing `fs::write` (`cli.rs:999`) persists — @@ -346,6 +389,15 @@ in the `cli.rs` test module. errors (non-not-found) → push `Ok`, GC skipped, warning present. - **Suspicious pointer degrades to warning**: prior value is pointer-kind with `version` 2 → push `Ok`, no deletes, warning. +- **Concurrency guard skips on changed root**: the fake serves the root + `describe` twice — prior pointer pre-commit, then a *different* value + on the post-commit read-back → GC skipped for that root with a "root + changed" warning and **no** `delete` calls. (The happy-path + delete test conversely serves the newly-written value on read-back so + the guard passes.) +- **Reserved key rejected**: `push_config_entries` with a logical key + containing `.__edgezero_chunks.` returns `Err` before any `fastly` + invocation (no `list`/`describe`/`update`/`delete`). - **Ordering**: extend the argv-log fake to assert every `delete` argv appears strictly after the root-pointer `update` argv. - **Dry-run stays offline**: dry-run makes no `list`/`describe`/`delete` @@ -361,9 +413,13 @@ in the `cli.rs` test module. chunk keys removed, root holds the direct envelope. - **Sibling coexistence preserved**: a push of `app_config` must not remove `app_config_staging` chunk keys (Spec 12.7 coexistence). -- **Free-form `--key` with the infix**: threading roots (not inference) - means a root key containing `.__edgezero_chunks.` is still handled - correctly — its own prior chunks are swept, siblings untouched. +- **Suspicious prior pointer (real push)**: seed the root with a + pointer-kind-but-invalid value (e.g. `version: 2`); a real local push + of a new config returns a suspicious-pointer warning, deletes no chunk + keys as a side effect, and still writes the new value. +- **Reserved key rejected**: `push_config_entries_local` with a logical + key containing `.__edgezero_chunks.` returns `Err` before touching + `fastly.toml` (file unchanged / not created). - **Dry-run count**: reports the correct orphan count and writes nothing. - **Dry-run identical re-push counts 0**: seed a chunked config, then @@ -393,8 +449,8 @@ in the `cli.rs` test module. | File | Change | | --- | --- | | `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(root_key, raw) -> Result, String>` (validates v1 + prefix-scopes) + unit tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries`, `:349`) | Per-root prior read via `fetch_remote_config_store_entry`; `delete_config_store_entry` helper (`--auto-yes`); post-commit sweep; offline dry-run GC-intent line; tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Thread logical roots to the writer; best-effort dry-run orphan counts (degrade to "unknown", never newly fail); tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | New `roots: &[&str]` param; snapshot old values, prune orphans in the same in-memory rewrite before the existing `fs::write` (`:999`) | +| `crates/edgezero-adapter-fastly/src/cli.rs` (helpers) | `reject_reserved_root_keys`, `expand_root` (exact per-root keep-set + `new_root_value`), `orphan_chunk_keys`, `delete_config_store_entry` (`--key --auto-yes`, never `--all`) | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries`, `:349`) | Reject reserved keys; per-root prior read via `fetch_remote_config_store_entry`; post-commit sweep with read-back concurrency guard; offline dry-run GC-intent line; tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Reject reserved keys; pass exact per-root keep-sets to the writer; best-effort dry-run orphan counts (degrade to "unknown", never newly fail); tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | New `gc_roots: &[(String, HashSet)]` param (empty ⇒ no GC); snapshot old values, prune orphans in the same in-memory rewrite before the existing `fs::write` (`:999`); update all 10 call sites | | `crates/edgezero-adapter-fastly/src/cli.rs` (test `:3317`) | Invert: assert old chunks are deleted after re-push; drop "no GC in v1" commentary | -| `crates/edgezero-cli/src/config.rs` (`:297`) / `args.rs` (`:311`) — optional, defence-in-depth | Reject a `--key` containing `.__edgezero_chunks.` at push time (collides with the chunk namespace) | From 98bc46867e6347170a2e451b939bdbd63c012e7e Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 08:00:16 -0700 Subject: [PATCH 07/25] Review v5 (spec+plan): document cloud GC as single-writer/best-effort (drop 'race-safe' overclaim, add Concurrency model section + plan precondition gate); correct cost note for the post-commit read-back describe; add dry-run suspicious-pointer test --- .../plans/2026-07-07-fastly-chunk-gc.md | 14 +++- .../specs/2026-07-07-fastly-chunk-gc.md | 76 ++++++++++++++----- 2 files changed, 68 insertions(+), 22 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index 70c3a462..40f38486 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -8,6 +8,8 @@ **Tech Stack:** Rust 2021, `serde_json`, `toml_edit`, existing Fastly CLI shell-out helpers, existing fake-`fastly` test shims, `cargo test -p edgezero-adapter-fastly --features cli`. +**Precondition (gate):** Cloud GC is **single-writer, best-effort** — it is safe only when pushes to a given config store are serialized. The post-commit read-back guard narrows but does not close the concurrent-push window (Fastly has no compare-and-delete). Do NOT start this plan unless the team accepts that concurrent cloud pushes are unsupported for GC purposes. If strict concurrent-push safety is required, stop and redesign around an offline, lease-guarded `config gc` (the deferred non-goal) instead. See the spec's "Concurrency model". + --- ## File Structure @@ -397,6 +399,15 @@ dry-run the SAME bytes and assert the count is `0`: assert!(combined.contains("would delete 0 orphan chunks")); ``` +Add a third dry-run test for a suspicious prior pointer: seed the root +with a pointer-kind-but-invalid value (e.g. `version: 2`), dry-run a new +config, and assert the degraded line plus no write: + +```rust +assert!(combined.contains("unknown: suspicious prior pointer")); +// and the file is unchanged +``` + - [ ] **Step 3c: Add local suspicious-pointer real-push test (spec §"local")** Seed `fastly.toml` so the root holds a pointer-kind-but-invalid value @@ -910,5 +921,6 @@ git commit -m "test(fastly): verify chunk gc behavior" - Use exact `toml_edit::Table` key operations for local dotted chunk keys. - If a push commit fails, do not sweep anything. The old root pointer may still be live. - If GC fails, return `Ok` with warning status lines. The push success criterion remains new entries committed. -- Cloud GC spawns one `fastly` process per orphan, sequentially. A large prior generation (e.g. 50 chunks) means ~50 sequential shell-outs after the push — a known wall-clock cost. Fastly has no bulk delete-by-key (`--all` is store-wide and unusable), so batching is not available; do not add parallel spawns in v1 without measuring. +- Cloud GC assumes a **single writer** per config store (serialized pushes). The post-commit read-back guard removes the common revert-before-read-back race but NOT the read-back-then-revert interleaving — Fastly has no compare-and-delete. This is best-effort by design (spec "Concurrency model"); do not represent cloud GC as strictly concurrent-safe. +- Cloud GC costs: one pre-push `describe` per root, plus one post-commit read-back `describe` per root that has orphans, plus one `fastly` `delete` process per orphan (sequential). A 50-chunk prior generation ≈ 50 sequential deletes. Fastly has no bulk delete-by-key (`--all` is store-wide and unusable); do not add parallel spawns in v1 without measuring. - Line numbers in this plan are approximate anchors against `main`; `grep` the named function/test before editing, since prior edits shift offsets. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 83b37ec3..ea7cd6b3 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -89,8 +89,9 @@ pointer no longer does. - **Orphans**: `prior_chunk_keys − new_keep_set`. In practice this is all prior chunk keys, because a changed config changes the SHA and therefore every chunk key; but the set-difference is the correct, - race-safe formulation (a re-push of *identical* bytes re-derives the - same keys and deletes nothing). + idempotent formulation (a re-push of *identical* bytes re-derives the + same keys and deletes nothing). "Idempotent" here is about re-pushing + the same bytes, NOT concurrency safety — see the Concurrency model. ## Invariants (MUST) @@ -132,21 +133,47 @@ pointer no longer does. warning folded into the returned status lines; the push still reports success. A leaked chunk is harmless; a failed push is not. -5. **Cloud deletes are guarded by a post-commit root read-back.** A - concurrent push could overwrite the root between our commit and our - sweep — e.g. revert it to the prior pointer, making the "orphans" - live again. So before deleting a root's orphans, cloud GC re-reads - the raw root and proceeds only if it still equals *exactly* the value - this push wrote (`new_root_value`). If it differs (or the re-read - fails), GC is skipped for that root with a warning. This narrows but - does not fully close the window (no compare-and-delete exists in - Fastly; a writer could still intervene between the read-back and an - individual delete) — acceptable under invariant 4, since the guard - eliminates the realistic revert-to-prior corruption and the residual - only risks a leak, never a live delete in the common case. The local - path holds the whole file in one rewrite and has no equivalent +5. **Cloud deletes are guarded by a post-commit root read-back + (mitigation, not a guarantee).** A concurrent push could overwrite + the root between our commit and our sweep — e.g. revert it to the + prior pointer, making the "orphans" live again. So before deleting a + root's orphans, cloud GC re-reads the raw root and proceeds only if + it still equals *exactly* the value this push wrote (`new_root_value`). + If it differs (or the re-read fails), GC is skipped for that root with + a warning. This shrinks the window but does NOT close it: Fastly has + no compare-and-delete, so an interleaving push can still restore the + prior pointer *after* our read-back passes but *before* our deletes + run, causing a live delete (see Concurrency model). The local path + holds the whole file in one in-memory rewrite and has no equivalent remote-concurrency window. +## Concurrency model + +**Cloud GC assumes a single writer per config store** — i.e. pushes to a +given store are serialized (one operator, or a CI pipeline that does not +run overlapping `config push` jobs against the same store). Under that +assumption the read-back guard (invariant 5) makes cloud GC safe: after +we write our root and confirm it on read-back, nothing else changes it, +so its orphans are genuinely dead. + +Under **concurrent** writers, cloud GC is **best-effort and not +strictly safe**. Fastly Config Store exposes no compare-and-delete or +lock, so there is a residual interleaving — push A commits B and its +read-back passes, then push C reverts the root to the prior pointer A, +then A's deletes run and remove chunks that are live again for C. The +read-back guard removes the *common* revert-before-read-back case but +cannot remove this one. The impact is bounded to a specific config +store's chunk data; it never affects other stores or the root pointer +itself, and the immediately-following read will surface the integrity +error (missing/again-orphaned chunk) rather than serving wrong data. + +If strict concurrent-push safety is ever required, the design must +change: stop deleting inline and move all reclamation to an offline, +lease-guarded `config gc` command (the deferred non-goal), or introduce +an external lock around push. That is out of scope for v1, which ships +GC as a single-writer, best-effort convenience. **Do not begin +implementation unless the team accepts this single-writer assumption.** + ## `prior_chunk_keys` helper (`chunked_config.rs`) Add next to the existing `FastlyChunkPointer` schema @@ -282,12 +309,15 @@ handled for free: the new value is a direct envelope, (at worst) a mix of old and new chunks, each still referenced by whichever pointer actually committed. Reclamation waits for the next successful push. Leak-safe over corrupt-safe, per invariant 4. -- **Cost**: one extra `describe` per logical root before the push, plus - one `delete` per orphan after. No store-wide `list` call. Deletes are - sequential `fastly` subprocess spawns — a large prior generation - (e.g. 50 chunks) adds ~50 sequential shell-outs of post-push latency. - Fastly exposes no bulk delete-by-key (`--all` is store-wide), so v1 - accepts this cost rather than parallelising spawns. +- **Cost**: per logical root, one `describe` before the push; then, for + each root that actually has orphans, a second `describe` after commit + (the read-back guard); then one `delete` per orphan. A root with no + prior chunks costs only the first `describe` and no read-back. No + store-wide `list` call. Deletes are sequential `fastly` subprocess + spawns — a large prior generation (e.g. 50 chunks) adds ~50 sequential + shell-outs of post-push latency. Fastly exposes no bulk delete-by-key + (`--all` is store-wide), so v1 accepts this cost rather than + parallelising spawns. ## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) @@ -428,6 +458,10 @@ in the `cli.rs` test module. - **Dry-run degrades, never fails**: a malformed `fastly.toml` makes the dry-run report `unknown` for the GC count while still printing the direct-vs-chunked intent lines and returning `Ok`. +- **Dry-run suspicious prior pointer**: seed the root with a + pointer-kind-but-invalid value; the dry-run reports + `(unknown: suspicious prior pointer)` for that root and still returns + `Ok` without writing. ## Non-goals From b8eb07be884cdc4186fa5cf7d7f887ae7b7c9e32 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 08:48:09 -0700 Subject: [PATCH 08/25] Reframe cloud concurrency as last-writer-wins (per decision) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concurrent cloud pushes are SUPPORTED: root pointer is upsert so the last write wins on the value. GC obeys LWW via the post-commit read-back guard — a push reclaims prior chunks only while it is still the last writer of the root, else it yields (never deletes the winner's live chunks). Removes the single-writer assumption and the blocking 'do not implement' gate; keeps the honest best-effort residual-window note. No code. --- .../plans/2026-07-07-fastly-chunk-gc.md | 4 +- .../specs/2026-07-07-fastly-chunk-gc.md | 86 ++++++++++--------- 2 files changed, 47 insertions(+), 43 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index 40f38486..7121c07f 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -8,7 +8,7 @@ **Tech Stack:** Rust 2021, `serde_json`, `toml_edit`, existing Fastly CLI shell-out helpers, existing fake-`fastly` test shims, `cargo test -p edgezero-adapter-fastly --features cli`. -**Precondition (gate):** Cloud GC is **single-writer, best-effort** — it is safe only when pushes to a given config store are serialized. The post-commit read-back guard narrows but does not close the concurrent-push window (Fastly has no compare-and-delete). Do NOT start this plan unless the team accepts that concurrent cloud pushes are unsupported for GC purposes. If strict concurrent-push safety is required, stop and redesign around an offline, lease-guarded `config gc` (the deferred non-goal) instead. See the spec's "Concurrency model". +**Concurrency model — last-writer-wins:** Concurrent cloud pushes are **supported**. The root pointer is `update --upsert`, so the last write wins on the config value. GC obeys LWW via the post-commit read-back guard: a push reclaims prior chunks only while it is still the last writer of the root (read-back == what it wrote), otherwise it **yields** and deletes nothing — so a superseded push never deletes the winner's live chunks. The guard narrows but does not fully close the window (Fastly has no compare-and-delete), making GC best-effort under LWW, not transactional. No blocking precondition. See the spec's "Concurrency model: last-writer-wins". --- @@ -921,6 +921,6 @@ git commit -m "test(fastly): verify chunk gc behavior" - Use exact `toml_edit::Table` key operations for local dotted chunk keys. - If a push commit fails, do not sweep anything. The old root pointer may still be live. - If GC fails, return `Ok` with warning status lines. The push success criterion remains new entries committed. -- Cloud GC assumes a **single writer** per config store (serialized pushes). The post-commit read-back guard removes the common revert-before-read-back race but NOT the read-back-then-revert interleaving — Fastly has no compare-and-delete. This is best-effort by design (spec "Concurrency model"); do not represent cloud GC as strictly concurrent-safe. +- Cloud GC follows **last-writer-wins** (spec "Concurrency model: last-writer-wins"): concurrent pushes are supported; the last root-pointer write wins on the value. A push sweeps prior chunks only while the post-commit read-back shows it is still the last writer, else it yields. The guard removes the common race but not the read-back-then-supersede interleaving (no compare-and-delete) — best-effort, not transactional; don't represent it as strictly concurrent-safe. - Cloud GC costs: one pre-push `describe` per root, plus one post-commit read-back `describe` per root that has orphans, plus one `fastly` `delete` process per orphan (sequential). A 50-chunk prior generation ≈ 50 sequential deletes. Fastly has no bulk delete-by-key (`--all` is store-wide and unusable); do not add parallel spawns in v1 without measuring. - Line numbers in this plan are approximate anchors against `main`; `grep` the named function/test before editing, since prior edits shift offsets. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index ea7cd6b3..ba2581b6 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -91,7 +91,8 @@ pointer no longer does. therefore every chunk key; but the set-difference is the correct, idempotent formulation (a re-push of *identical* bytes re-derives the same keys and deletes nothing). "Idempotent" here is about re-pushing - the same bytes, NOT concurrency safety — see the Concurrency model. + the same bytes, NOT concurrency safety — see "Concurrency model: + last-writer-wins". ## Invariants (MUST) @@ -133,46 +134,49 @@ pointer no longer does. warning folded into the returned status lines; the push still reports success. A leaked chunk is harmless; a failed push is not. -5. **Cloud deletes are guarded by a post-commit root read-back - (mitigation, not a guarantee).** A concurrent push could overwrite - the root between our commit and our sweep — e.g. revert it to the - prior pointer, making the "orphans" live again. So before deleting a - root's orphans, cloud GC re-reads the raw root and proceeds only if - it still equals *exactly* the value this push wrote (`new_root_value`). - If it differs (or the re-read fails), GC is skipped for that root with - a warning. This shrinks the window but does NOT close it: Fastly has - no compare-and-delete, so an interleaving push can still restore the - prior pointer *after* our read-back passes but *before* our deletes - run, causing a live delete (see Concurrency model). The local path - holds the whole file in one in-memory rewrite and has no equivalent - remote-concurrency window. - -## Concurrency model - -**Cloud GC assumes a single writer per config store** — i.e. pushes to a -given store are serialized (one operator, or a CI pipeline that does not -run overlapping `config push` jobs against the same store). Under that -assumption the read-back guard (invariant 5) makes cloud GC safe: after -we write our root and confirm it on read-back, nothing else changes it, -so its orphans are genuinely dead. - -Under **concurrent** writers, cloud GC is **best-effort and not -strictly safe**. Fastly Config Store exposes no compare-and-delete or -lock, so there is a residual interleaving — push A commits B and its -read-back passes, then push C reverts the root to the prior pointer A, -then A's deletes run and remove chunks that are live again for C. The -read-back guard removes the *common* revert-before-read-back case but -cannot remove this one. The impact is bounded to a specific config -store's chunk data; it never affects other stores or the root pointer -itself, and the immediately-following read will surface the integrity -error (missing/again-orphaned chunk) rather than serving wrong data. - -If strict concurrent-push safety is ever required, the design must -change: stop deleting inline and move all reclamation to an offline, -lease-guarded `config gc` command (the deferred non-goal), or introduce -an external lock around push. That is out of scope for v1, which ships -GC as a single-writer, best-effort convenience. **Do not begin -implementation unless the team accepts this single-writer assumption.** +5. **Cloud deletes obey last-writer-wins via a post-commit read-back.** + A push reclaims a root's prior chunks only while it is still the last + writer of that root. After committing, cloud GC re-reads the raw root + and deletes only if it still equals *exactly* what this push wrote + (`new_root_value`); if a newer push has superseded it (root differs) + or the re-read fails, GC **yields** — deletes nothing, warns, moves + on. This keeps a superseded push from deleting the winner's live + chunks. The guard narrows but does not fully close the window (Fastly + has no compare-and-delete); see "Concurrency model: last-writer-wins" + for the residual and its bounded impact. The local path holds the + whole file in one in-memory rewrite and has no remote-concurrency + window. + +## Concurrency model: last-writer-wins + +The config value follows **last-writer-wins (LWW)**. The root key is a +single Config Store entry and `update --upsert` means the push whose +root pointer lands *last* defines the live config. Concurrent pushes to +the same store are **supported** under this semantic — no single-writer +assumption, no lock, no blocking precondition. + +GC layers onto LWW with one rule: **a push reclaims prior-generation +chunks only while it is still the last writer of the root.** That is +exactly what the invariant-5 read-back checks — after committing, the +push re-reads the root and sweeps only if it still equals what the push +wrote (`new_root_value`). If a newer push has superseded it (the root +now differs), the push **yields**: it deletes nothing and lets the new +last writer own reclamation. This is what keeps LWW honest — a +superseded (losing) push never deletes the *winner's* live chunks, so +the winner's config stays readable. + +Best-effort boundary (stated honestly): the read-back narrows but does +not fully close the window, because Fastly exposes no compare-and-delete. +A newer push can still supersede us *between* our read-back and an +individual delete, so in a rare interleaving a losing push can delete a +chunk the new winner references. The blast radius is one store's chunk +data — never another store, never the root pointer. The next read of +that config then surfaces an integrity error (missing chunk) rather than +serving wrong data, and re-pushing the winning config restores it +(chunks are written before the pointer). GC is therefore a best-effort +reclaimer under LWW, not a transactional one. If that residual ever +matters, the deferred `config gc` non-goal is the path to fully +lease-guarded reclamation. ## `prior_chunk_keys` helper (`chunked_config.rs`) From 3cc25af437d8773b45f1e369d9c486bb57632607 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 09:08:34 -0700 Subject: [PATCH 09/25] Plan/spec tighten-ups (non-blocking): non-vacuous local suspicious-pointer test (seed real chunk keys, assert they survive); expand_root errors on empty instead of silent default; reserved-key error wording drops --key assumption --- .../plans/2026-07-07-fastly-chunk-gc.md | 19 +++++++++++++------ .../specs/2026-07-07-fastly-chunk-gc.md | 8 +++++--- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index 7121c07f..6796dbdc 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -248,7 +248,11 @@ fn expand_root(root_key: &str, body: &str) -> Result<(Vec<(String, String)>, Has let expanded = prepare_fastly_config_entries(root_key, body)?; let new_keys: HashSet = expanded.iter().map(|(k, _)| k.clone()).collect(); // prepare_* always emits the root entry LAST (root pointer or direct value). - let new_root_value = expanded.last().map(|(_, v)| v.clone()).unwrap_or_default(); + // Make the invariant explicit rather than silently defaulting to "". + let new_root_value = expanded + .last() + .map(|(_, v)| v.clone()) + .ok_or_else(|| format!("internal: no physical entries produced for root `{root_key}`"))?; Ok((expanded, new_keys, new_root_value)) } @@ -269,7 +273,7 @@ fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> for (key, _) in entries { if key.contains(CHUNK_KEY_INFIX) { return Err(format!( - "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different --key" + "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" )); } } @@ -411,11 +415,14 @@ assert!(combined.contains("unknown: suspicious prior pointer")); - [ ] **Step 3c: Add local suspicious-pointer real-push test (spec §"local")** Seed `fastly.toml` so the root holds a pointer-kind-but-invalid value -(e.g. `{"edgezero_kind":"fastly_config_chunks","version":2}`), then run a -REAL (non-dry-run) local push of a new config. Assert: +(e.g. `{"edgezero_kind":"fastly_config_chunks","version":2}`) **and** +seed one or more real chunk-like entries under the root's namespace +(keys containing `.__edgezero_chunks.`) so the "no deletes" assertion is +non-vacuous. Then run a REAL (non-dry-run) local push of a new config. +Assert: - the returned status `Vec` contains a suspicious-pointer warning, -- no chunk keys were removed as a side effect (the writer must not delete - when `prior_chunk_keys` returns `Err`), +- **the pre-seeded chunk-like keys are still present** (the writer must + not prune when `prior_chunk_keys` returns `Err`), - the new value is written to the root key. - [ ] **Step 3d: Add reserved-key rejection test (local)** diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index ba2581b6..4638527c 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -448,9 +448,11 @@ in the `cli.rs` test module. - **Sibling coexistence preserved**: a push of `app_config` must not remove `app_config_staging` chunk keys (Spec 12.7 coexistence). - **Suspicious prior pointer (real push)**: seed the root with a - pointer-kind-but-invalid value (e.g. `version: 2`); a real local push - of a new config returns a suspicious-pointer warning, deletes no chunk - keys as a side effect, and still writes the new value. + pointer-kind-but-invalid value (e.g. `version: 2`) **and** pre-seed + real chunk-like keys under the root namespace (so "no deletes" is + non-vacuous); a real local push of a new config returns a + suspicious-pointer warning, leaves the pre-seeded chunk keys present, + and still writes the new value. - **Reserved key rejected**: `push_config_entries_local` with a logical key containing `.__edgezero_chunks.` returns `Err` before touching `fastly.toml` (file unchanged / not created). From 67ac0ddce5ff235252836d4abb1cd1d3b041e26c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 09:14:12 -0700 Subject: [PATCH 10/25] feat(fastly): add prior_chunk_keys for chunk GC (Value-first, v1-validated, prefix-scoped) + unit tests --- .../src/chunked_config.rs | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/crates/edgezero-adapter-fastly/src/chunked_config.rs b/crates/edgezero-adapter-fastly/src/chunked_config.rs index 6c8b0937..fbe610fc 100644 --- a/crates/edgezero-adapter-fastly/src/chunked_config.rs +++ b/crates/edgezero-adapter-fastly/src/chunked_config.rs @@ -302,6 +302,66 @@ where Ok(reconstructed) } +/// Validate a prior root value and return the chunk keys it referenced, +/// scoped to `root_key`'s own chunk namespace. Used only for chunk GC on +/// re-push (Stage 7 writeback). +/// +/// Parse as `serde_json::Value` FIRST so a pointer-kind value with +/// missing/invalid fields still reaches the warning path instead of being +/// silently dropped by a failed struct deserialize. +/// +/// Returns: +/// - `Ok(keys)` -- value is a valid v1 chunk pointer; `keys` are its +/// `chunks[].key` entries (all confirmed to match +/// `"{root_key}{CHUNK_KEY_INFIX}"`). +/// - `Ok(vec![])` -- value is a direct `BlobEnvelope`, absent, or not +/// pointer-shaped at all (normal: first push, or was-direct). Silent. +/// - `Err(msg)` -- value IS pointer-kind (`edgezero_kind == POINTER_KIND`) +/// but fails validation: malformed, unsupported `version`, or a +/// referenced key falls outside this root's chunk prefix. Callers log +/// `msg` as a warning and skip GC for this root (delete nothing). +#[cfg(any(feature = "cli", test))] +pub(crate) fn prior_chunk_keys(root_key: &str, raw: &str) -> Result, String> { + // 1. Parse loosely. Not-JSON, or not our pointer kind => silent. + let value: serde_json::Value = match serde_json::from_str(raw) { + Ok(value) => value, + Err(_) => return Ok(Vec::new()), + }; + if value + .get("edgezero_kind") + .and_then(serde_json::Value::as_str) + != Some(POINTER_KIND) + { + // Direct BlobEnvelope, unrelated JSON, or first push. + return Ok(Vec::new()); + } + + // 2. It IS pointer-kind: from here every failure WARNS (Err), never silent. + let pointer: FastlyChunkPointer = serde_json::from_value(value).map_err(|err| { + format!("prior chunk pointer at `{root_key}` is malformed: {err}; skipping chunk GC") + })?; + if pointer.version != 1 { + return Err(format!( + "prior chunk pointer at `{root_key}` has unsupported version {}; skipping chunk GC", + pointer.version + )); + } + + // 3. Prefix-scope every referenced key to this root's own namespace. + let prefix = format!("{root_key}{CHUNK_KEY_INFIX}"); + let mut keys = Vec::with_capacity(pointer.chunks.len()); + for chunk_ref in pointer.chunks { + if !chunk_ref.key.starts_with(&prefix) { + return Err(format!( + "prior chunk pointer at `{root_key}` references chunk key `{}` outside expected prefix `{prefix}`; skipping chunk GC", + chunk_ref.key + )); + } + keys.push(chunk_ref.key); + } + Ok(keys) +} + // --------------------------------------------------------------------------- // Unit tests // --------------------------------------------------------------------------- @@ -657,4 +717,87 @@ mod tests { ); assert!(err.contains("my_key"), "error must name root key: {err}"); } + + // ---- prior_chunk_keys tests ---- + + #[test] + fn prior_chunk_keys_returns_valid_v1_pointer_keys() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + + let keys = prior_chunk_keys("app_config", pointer_json).expect("valid pointer"); + + let expected: Vec = entries[..entries.len().saturating_sub(1)] + .iter() + .map(|(key, _)| key.clone()) + .collect(); + assert_eq!(keys, expected); + } + + #[test] + fn prior_chunk_keys_returns_empty_for_direct_envelope() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT); + assert_eq!( + prior_chunk_keys("app_config", &envelope).unwrap(), + Vec::::new() + ); + } + + #[test] + fn prior_chunk_keys_returns_empty_for_unrelated_json() { + assert_eq!( + prior_chunk_keys("app_config", r#"{"hello":"world"}"#).unwrap(), + Vec::::new() + ); + } + + #[test] + fn prior_chunk_keys_returns_empty_for_wrong_kind() { + let raw = r#"{"edgezero_kind":"other","version":1,"chunks":[],"data_sha256":"","envelope_len":0,"envelope_sha256":""}"#; + assert_eq!( + prior_chunk_keys("app_config", raw).unwrap(), + Vec::::new() + ); + } + + #[test] + fn prior_chunk_keys_rejects_unsupported_pointer_version() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + let mut pointer: FastlyChunkPointer = serde_json::from_str(pointer_json).unwrap(); + pointer.version = 2; + let raw = serde_json::to_string(&pointer).unwrap(); + + let err = prior_chunk_keys("app_config", &raw).expect_err("version 2 should warn"); + assert!(err.contains("unsupported version"), "{err}"); + } + + #[test] + fn prior_chunk_keys_rejects_foreign_chunk_prefix() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + let mut pointer: FastlyChunkPointer = serde_json::from_str(pointer_json).unwrap(); + pointer.chunks[0].key = + pointer.chunks[0] + .key + .replacen("app_config", "app_config_staging", 1); + let raw = serde_json::to_string(&pointer).unwrap(); + + let err = prior_chunk_keys("app_config", &raw).expect_err("foreign chunk should warn"); + assert!(err.contains("outside"), "{err}"); + } + + // Regression for the Value-first rule: a value that IS pointer-kind but + // is missing required fields (`chunks`, `data_sha256`, …) must WARN, not + // be silently dropped by a failed struct deserialize. + #[test] + fn prior_chunk_keys_warns_on_pointer_kind_with_missing_fields() { + let raw = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#; + let err = prior_chunk_keys("app_config", raw) + .expect_err("pointer-kind but malformed must warn, not Ok([])"); + assert!(!err.is_empty(), "{err}"); + } } From 653b2203d7ab2f51e151303b89a8381ec75d9fea Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 09:18:43 -0700 Subject: [PATCH 11/25] feat(fastly): add chunk-GC planning helpers (expand_root, orphan_chunk_keys, reject_reserved_root_keys, FastlyConfigGcPlan) + unit tests Wired into push paths in the following commits; transient dead-code warnings until then. --- crates/edgezero-adapter-fastly/src/cli.rs | 142 +++++++++++++++++++++- 1 file changed, 141 insertions(+), 1 deletion(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index a3de1cba..701bdfb4 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -1,3 +1,4 @@ +use std::collections::HashSet; use std::env; use std::fs; use std::io::{ErrorKind, Write as _}; @@ -5,7 +6,9 @@ use std::path::{Path, PathBuf}; use std::process::Command; use std::process::Stdio; -use crate::chunked_config::{prepare_fastly_config_entries, resolve_fastly_config_value}; +use crate::chunked_config::{ + prepare_fastly_config_entries, prior_chunk_keys, resolve_fastly_config_value, CHUNK_KEY_INFIX, +}; use ctor::ctor; use edgezero_adapter::cli_support::{ find_manifest_upwards, find_workspace_root, path_distance, read_package_name, run_native_cli, @@ -1001,6 +1004,70 @@ fn write_fastly_local_config_store( Ok(()) } +// ------------------------------------------------------------------- +// chunk GC helpers (Stage 7 re-push reclamation) +// ------------------------------------------------------------------- + +/// Per-root plan for reclaiming a prior generation's chunk entries. +struct FastlyConfigGcPlan { + root_key: String, + /// Prior chunk keys to consider deleting, or a warning to surface + /// (suspicious prior pointer) that skips GC for this root. + prior_keys: Result, String>, + /// Exact keep-set this push writes for the root (chunk keys + root key). + new_keys: HashSet, + /// Exactly the value written at `root_key` (its new pointer, or the + /// direct envelope). Cloud uses it as a read-back concurrency guard + /// (last-writer-wins); local ignores it. + new_root_value: String, +} + +/// Expand ONE logical `(root_key, body)` into its physical entries, the +/// exact keep-set for that root, and the value written at the root key. +/// No cross-root prefix scanning (a free-form `--key` can't mislead it). +fn expand_root( + root_key: &str, + body: &str, +) -> Result<(Vec<(String, String)>, HashSet, String), String> { + let expanded = prepare_fastly_config_entries(root_key, body)?; + let new_keys: HashSet = expanded.iter().map(|(k, _)| k.clone()).collect(); + // prepare_* always emits the root entry LAST (root pointer or direct + // value). Make the invariant explicit rather than silently defaulting. + let new_root_value = expanded + .last() + .map(|(_, v)| v.clone()) + .ok_or_else(|| format!("internal: no physical entries produced for root `{root_key}`"))?; + Ok((expanded, new_keys, new_root_value)) +} + +/// Orphans = prior chunk keys not in the new keep-set. Propagates a +/// suspicious-pointer `Err` so the caller can warn and skip GC. +fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { + match &plan.prior_keys { + Ok(prior) => Ok(prior + .iter() + .filter(|key| !plan.new_keys.contains(*key)) + .cloned() + .collect()), + Err(err) => Err(err.clone()), + } +} + +/// Reject logical keys that collide with the reserved chunk namespace. +/// `--key` is free-form, so this is enforced at the Fastly adapter +/// boundary: such a key would let a push write into another key's chunk +/// space, and could not be reclaimed correctly. +fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> { + for (key, _) in entries { + if key.contains(CHUNK_KEY_INFIX) { + return Err(format!( + "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" + )); + } + } + Ok(()) +} + // ------------------------------------------------------------------- // `config push` helpers // ------------------------------------------------------------------- @@ -1406,6 +1473,79 @@ mod tests { const TEST_CONFIG_ID: &str = "app_config"; const TEST_SECRET_ID: &str = "default"; + // ---- chunk GC helpers ---- + + #[test] + fn reject_reserved_root_keys_accepts_clean_keys() { + let entries = vec![ + ("app_config".to_owned(), "{}".to_owned()), + ("app_config_staging".to_owned(), "{}".to_owned()), + ]; + assert!(reject_reserved_root_keys(&entries).is_ok()); + } + + #[test] + fn reject_reserved_root_keys_rejects_infix_key() { + let bad = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); + let entries = vec![(bad.clone(), "{}".to_owned())]; + let err = reject_reserved_root_keys(&entries).expect_err("reserved infix must reject"); + assert!(err.contains(&bad), "error names the key: {err}"); + assert!(err.contains("reserved"), "error explains why: {err}"); + } + + #[test] + fn orphan_chunk_keys_subtracts_new_keys() { + let mut new_keys = std::collections::HashSet::new(); + new_keys.insert("keep".to_owned()); + let plan = FastlyConfigGcPlan { + root_key: "app_config".to_owned(), + prior_keys: Ok(vec![ + "gone1".to_owned(), + "keep".to_owned(), + "gone2".to_owned(), + ]), + new_keys, + new_root_value: String::new(), + }; + let orphans = orphan_chunk_keys(&plan).expect("ok"); + assert_eq!(orphans, vec!["gone1".to_owned(), "gone2".to_owned()]); + } + + #[test] + fn orphan_chunk_keys_propagates_prior_err() { + let plan = FastlyConfigGcPlan { + root_key: "app_config".to_owned(), + prior_keys: Err("suspicious".to_owned()), + new_keys: std::collections::HashSet::new(), + new_root_value: String::new(), + }; + assert!(orphan_chunk_keys(&plan).is_err()); + } + + #[test] + fn expand_root_direct_value_has_single_entry() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); + assert_eq!(expanded.len(), 1); + assert_eq!(new_root_value, envelope); + assert!(new_keys.contains(TEST_CONFIG_ID)); + assert_eq!(new_keys.len(), 1); + } + + #[test] + fn expand_root_chunked_value_carries_pointer_as_root_value() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); + assert!(expanded.len() >= 2, "chunks + pointer"); + let (last_key, last_val) = expanded.last().unwrap(); + assert_eq!(last_key, TEST_CONFIG_ID); + assert_eq!(&new_root_value, last_val); + assert!(new_keys.contains(TEST_CONFIG_ID)); + assert_eq!(new_keys.len(), expanded.len()); + } + /// RAII guard: prepends a directory to `$PATH` and restores the original /// value on drop. #[cfg(unix)] From e8853b96486f63d7ccb0082c2fa71274221cf86c Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 09:30:29 -0700 Subject: [PATCH 12/25] feat(fastly): prune prior config chunks on local re-push write_fastly_local_config_store takes exact per-root keep-sets (gc_roots) and prunes orphaned chunk keys in the same in-memory rewrite; suspicious prior pointers warn and delete nothing. push_config_entries_local rejects reserved keys, threads per-root expansion, and reports best-effort orphan counts in dry-run. Inverts the stale no-GC test; adds shrink-to-direct, suspicious-pointer, reserved-key, and dry-run count/identical/unknown tests. --- crates/edgezero-adapter-fastly/src/cli.rs | 458 ++++++++++++++++++++-- 1 file changed, 427 insertions(+), 31 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 701bdfb4..699d6f9c 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -453,19 +453,25 @@ impl Adapter for FastlyCliAdapter { fastly_path.display() )]); } - // Expand logical entries into physical entries (chunks + pointer). + // Reject reserved keys before any expansion or I/O. + reject_reserved_root_keys(entries)?; + // Expand each logical root once: flatten for the write, keep the + // exact per-root keep-set for GC (no prefix scan of the flattened set). let mut physical_entries: Vec<(String, String)> = Vec::new(); + let mut gc_roots: Vec<(String, HashSet)> = Vec::with_capacity(entries.len()); for (key, body) in entries { - let expanded = prepare_fastly_config_entries(key, body)?; + let (expanded, new_keys, _new_root) = expand_root(key, body)?; physical_entries.extend(expanded); + gc_roots.push((key.clone(), new_keys)); } if dry_run { - let mut out = Vec::with_capacity(entries.len().saturating_add(1)); + let counts = local_orphan_counts_for_dry_run(&fastly_path, name, entries); + let mut out = Vec::with_capacity(entries.len().saturating_mul(2).saturating_add(1)); out.push(format!( "would edit `[local_server.config_stores.{name}.contents]` in {} (logical id `{logical}`) with entries:", fastly_path.display(), )); - for (key, body) in entries { + for (idx, (key, body)) in entries.iter().enumerate() { let expanded = prepare_fastly_config_entries(key, body) .unwrap_or_else(|_| vec![(key.clone(), body.clone())]); if expanded.len() == 1 { @@ -480,16 +486,28 @@ impl Adapter for FastlyCliAdapter { body.len() )); } + match counts.get(idx).map(|(_, count)| count) { + Some(Ok(n)) => out.push(format!( + " would delete {n} orphan chunks from the previous generation of `{key}`" + )), + Some(Err(reason)) => out.push(format!( + " would delete an unknown number of orphan chunks from the previous generation of `{key}` (unknown: {reason})" + )), + None => {} + } } return Ok(out); } - write_fastly_local_config_store(&fastly_path, name, &physical_entries)?; - Ok(vec![format!( + let warnings = + write_fastly_local_config_store(&fastly_path, name, &physical_entries, &gc_roots)?; + let mut out = vec![format!( "wrote {} physical entries ({} logical) to `[local_server.config_stores.{name}.contents]` in {} (logical id `{logical}`); restart `fastly compute serve` to pick up changes", physical_entries.len(), entries.len(), fastly_path.display() - )]) + )]; + out.extend(warnings); + Ok(out) } fn read_config_entry( @@ -930,7 +948,8 @@ fn write_fastly_local_config_store( path: &Path, platform_name: &str, entries: &[(String, String)], -) -> Result<(), String> { + gc_roots: &[(String, HashSet)], +) -> Result, String> { use toml_edit::{table, DocumentMut, Item, Table, Value}; let raw = match fs::read_to_string(path) { @@ -995,13 +1014,44 @@ fn write_fastly_local_config_store( path.display() ) })?; + // Snapshot prior chunk keys per GC root BEFORE the upsert, using the + // exact keep-set the caller computed for each root (no prefix scan). + let mut plans: Vec = Vec::with_capacity(gc_roots.len()); + for (root_key, new_keys) in gc_roots { + let prior_keys = contents_tbl + .get(root_key) + .and_then(toml_edit::Item::as_str) + .map_or_else(|| Ok(Vec::new()), |raw| prior_chunk_keys(root_key, raw)); + plans.push(FastlyConfigGcPlan { + root_key: root_key.clone(), + prior_keys, + new_keys: new_keys.clone(), + new_root_value: String::new(), // unused locally (no remote concurrency) + }); + } + + // Upsert the new physical entries. for (key, value) in entries { contents_tbl.insert(key, Item::Value(Value::from(value.clone()))); } + // Prune orphans in the same in-memory rewrite; a suspicious prior + // pointer (Err) warns and deletes nothing. + let mut warnings = Vec::new(); + for plan in &plans { + match orphan_chunk_keys(plan) { + Ok(orphans) => { + for key in orphans { + contents_tbl.remove(&key); + } + } + Err(err) => warnings.push(format!("warning: {err}")), + } + } + fs::write(path, doc.to_string()) .map_err(|err| format!("failed to write {}: {err}", path.display()))?; - Ok(()) + Ok(warnings) } // ------------------------------------------------------------------- @@ -1068,6 +1118,66 @@ fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> Ok(()) } +/// Best-effort per-root orphan count for `config push --local --dry-run`. +/// Reads the current `fastly.toml` (offline) and, for each logical +/// `(root_key, body)`, counts `prior_chunk_keys(root, old) - new_keys` +/// where `new_keys` is the root's OWN expansion. Never fails the dry-run: +/// on a missing file / no prior pointer / direct prior value it reports +/// `Ok(0)`; on unreadable or malformed prior state it reports `Err(reason)` +/// which the caller renders as an "unknown" line. +fn local_orphan_counts_for_dry_run( + path: &Path, + platform_name: &str, + entries: &[(String, String)], +) -> Vec<(String, Result)> { + use toml_edit::{DocumentMut, Item}; + + // Parse the current file once (best-effort). Absent file => no prior. + let parsed: Result, String> = match fs::read_to_string(path) { + Ok(text) => text + .parse::() + .map(Some) + .map_err(|_| "could not read prior state".to_owned()), + Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), + Err(_) => Err("could not read prior state".to_owned()), + }; + + entries + .iter() + .map(|(root_key, body)| { + let new_keys = match expand_root(root_key, body) { + Ok((_, keys, _)) => keys, + Err(err) => return (root_key.clone(), Err(err)), + }; + let count = match &parsed { + Err(reason) => Err(reason.clone()), + Ok(None) => Ok(0), + Ok(Some(doc)) => { + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(platform_name)) + .and_then(|st| st.get("contents")) + .and_then(Item::as_table); + match contents.and_then(|tbl| tbl.get(root_key)) { + None => Ok(0), // no contents table, or no prior value for this root + Some(item) => match item.as_str() { + None => Err("could not read prior state".to_owned()), + Some(raw) => match prior_chunk_keys(root_key, raw) { + Ok(prior) => { + Ok(prior.iter().filter(|k| !new_keys.contains(*k)).count()) + } + Err(_) => Err("suspicious prior pointer".to_owned()), + }, + }, + } + } + }; + (root_key.clone(), count) + }) + .collect() +} + // ------------------------------------------------------------------- // `config push` helpers // ------------------------------------------------------------------- @@ -1975,7 +2085,7 @@ mod tests { ("greeting".to_owned(), "hello".to_owned()), ("service.timeout_ms".to_owned(), "1500".to_owned()), ]; - write_fastly_local_config_store(&path, TEST_CONFIG_ID, &entries).expect("write"); + write_fastly_local_config_store(&path, TEST_CONFIG_ID, &entries, &[]).expect("write"); let after = fs::read_to_string(&path).expect("read back"); assert!( after.contains(&format!("[local_server.config_stores.{TEST_CONFIG_ID}]")), @@ -2008,12 +2118,14 @@ mod tests { &path, TEST_CONFIG_ID, &[("greeting".to_owned(), "stale".to_owned())], + &[], ) .expect("first write"); write_fastly_local_config_store( &path, TEST_CONFIG_ID, &[("greeting".to_owned(), "fresh".to_owned())], + &[], ) .expect("second write"); let after = fs::read_to_string(&path).expect("read back"); @@ -2043,6 +2155,7 @@ build = \"cargo build --release\" &path, TEST_CONFIG_ID, &[("greeting".to_owned(), "hi".to_owned())], + &[], ) .expect("write"); let after = fs::read_to_string(&path).expect("read back"); @@ -2072,6 +2185,7 @@ build = \"cargo build --release\" &path, TEST_CONFIG_ID, &[("greeting".to_owned(), "hi".to_owned())], + &[], ) .expect("write"); let after = fs::read_to_string(&path).expect("read back"); @@ -2507,6 +2621,7 @@ build = \"cargo build --release\" &path, TEST_CONFIG_ID, &[("greeting".to_owned(), envelope_json.clone())], + &[], ) .expect("setup write"); @@ -3384,6 +3499,7 @@ build = \"cargo build --release\" &fastly_toml, TEST_CONFIG_ID, &[("cfg".to_owned(), json_str.clone())], + &[], ) .expect("write"); @@ -3412,7 +3528,8 @@ build = \"cargo build --release\" let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); // Write all physical entries (chunks + pointer) to the local store. - write_fastly_local_config_store(&fastly_toml, TEST_CONFIG_ID, &physical).expect("write"); + write_fastly_local_config_store(&fastly_toml, TEST_CONFIG_ID, &physical, &[]) + .expect("write"); let result = FastlyCliAdapter .read_config_entry_local( @@ -3440,14 +3557,11 @@ build = \"cargo build --release\" /// /// The local fastly.toml writer upserts per-key (so a sibling /// `--key app_config_staging` push leaves `app_config` intact per - /// spec 12.7). Within the SAME root key, old chunks for envelope - /// A remain in the contents table after envelope B's push — they're - /// unreferenced (the root pointer at `app_config` now names B's - /// chunks), matching the remote Fastly behaviour where the - /// per-entry `update --upsert` shell-out has no atomic-delete - /// pairing. The runtime-correctness property holds either way: a - /// read after push B follows the active pointer and reconstructs - /// envelope B, not A. + /// spec 12.7). Within the SAME root key, GC on re-push prunes the + /// prior generation: after envelope B's push, envelope A's chunks — + /// now unreferenced by the `app_config` pointer — are removed from + /// the contents table. A read after push B follows the active + /// pointer and reconstructs envelope B, not A. #[cfg(unix)] #[test] #[expect( @@ -3461,8 +3575,7 @@ build = \"cargo build --release\" fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); // First push: envelope A. Records the chunk-key set so we can - // confirm they survive the second push (no garbage collection - // in v1 — spec 9.3 + Q6). + // confirm they are pruned by the second push's GC. let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); FastlyCliAdapter .push_config_entries_local( @@ -3496,10 +3609,10 @@ build = \"cargo build --release\" ); // Second push: a DIFFERENT oversized envelope B. The - // content-addressed chunk keys must shift to B's sha; old - // A-chunks may remain in the table (v1 doesn't GC). Build - // envelope B with a distinct payload key so its SHA differs - // from A's even at the same total length. + // content-addressed chunk keys must shift to B's sha; GC then + // prunes the old A-chunks. Build envelope B with a distinct + // payload key so its SHA differs from A's even at the same + // total length. let envelope_b = { use edgezero_core::blob_envelope::BlobEnvelope; use serde_json::json; @@ -3541,8 +3654,7 @@ build = \"cargo build --release\" // Chunk keys are content-addressed by envelope SHA, so the B // push installs a fresh chunk-set whose keys are all distinct - // from A's. Under the upsert semantic the A-chunks remain in - // the contents table (no GC in v1); B's chunks are simply added. + // from A's. GC on re-push prunes the now-unreferenced A-chunks. let new_b_chunks: Vec<&String> = chunks_b .iter() .filter(|key| !chunks_a.contains(*key)) @@ -3551,12 +3663,12 @@ build = \"cargo build --release\" !new_b_chunks.is_empty(), "push B must have added at least one new content-addressed chunk: A-set={chunks_a:?} B-set={chunks_b:?}" ); - // Old A-chunks remain in the table (orphan-but-present — - // matches the remote Fastly write-only-upsert semantic). + // Old A-chunks are pruned: GC deletes the prior generation the + // old pointer referenced once B's pointer supersedes it. for chunk_key in &chunks_a { assert!( - chunks_b.contains(chunk_key), - "old A-chunk `{chunk_key}` must remain in the local table after push B (v1 has no GC); B-set={chunks_b:?}" + !chunks_b.contains(chunk_key), + "old A-chunk `{chunk_key}` must be pruned from the local table after push B; B-set={chunks_b:?}" ); } @@ -3584,4 +3696,288 @@ build = \"cargo build --release\" "old envelope A's chunks must be inert -- read must NOT return A" ); } + + // ---------- local chunk GC ---------- + + /// Config shrinks from chunked back under the 8 000-char limit: the + /// new value is a direct envelope, so GC prunes every prior chunk. + #[cfg(unix)] + #[test] + fn push_config_entries_local_prunes_prior_chunks_when_value_shrinks_to_direct() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let chunked = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), chunked)], + &AdapterPushContext::new(), + false, + ) + .expect("first push"); + + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), direct.clone())], + &AdapterPushContext::new(), + false, + ) + .expect("second push"); + + let after = fs::read_to_string(&fastly_toml).expect("read"); + let doc: toml_edit::DocumentMut = after.parse().expect("parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents"); + + assert_eq!( + contents + .get(TEST_CONFIG_ID) + .and_then(toml_edit::Item::as_str), + Some(direct.as_str()), + "root holds the direct envelope" + ); + assert!( + !contents + .iter() + .any(|(key, _)| key.contains(CHUNK_KEY_INFIX)), + "prior chunks must be pruned: {after}" + ); + } + + /// A logical key containing the reserved chunk infix is rejected + /// before any file I/O (it would collide with the chunk namespace). + #[cfg(unix)] + #[test] + fn push_config_entries_local_rejects_reserved_key() { + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + let bad_key = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); + + let err = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(bad_key.clone(), "{}".to_owned())], + &AdapterPushContext::new(), + false, + ) + .expect_err("reserved key must be rejected"); + assert!(err.contains(&bad_key), "error names the key: {err}"); + assert!( + !fastly_toml.exists(), + "rejection must happen before any write" + ); + } + + /// A suspicious prior pointer (pointer-kind but invalid) makes GC + /// warn and delete nothing — pre-seeded chunk keys must survive. + #[cfg(unix)] + #[test] + fn push_config_entries_local_warns_on_suspicious_prior_pointer_and_keeps_chunks() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + // Seed the root with a pointer-kind-but-invalid value AND a real + // chunk-like key so "no deletes" is non-vacuous. + let seed = concat!( + "name = \"demo\"\n\n", + "[local_server.config_stores.app_config]\n", + "format = \"inline-toml\"\n\n", + "[local_server.config_stores.app_config.contents]\n", + "app_config = \"{\\\"edgezero_kind\\\":\\\"fastly_config_chunks\\\",\\\"version\\\":2}\"\n", + "\"app_config.__edgezero_chunks.deadbeef.0\" = \"seeded-chunk-payload\"\n", + ); + fs::write(&fastly_toml, seed).expect("seed"); + + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), direct.clone())], + &AdapterPushContext::new(), + false, + ) + .expect("push must still succeed"); + + let combined = out.join("\n"); + assert!( + combined.contains("skipping chunk GC"), + "must warn about the suspicious prior pointer: {combined}" + ); + + let after = fs::read_to_string(&fastly_toml).expect("read"); + let doc: toml_edit::DocumentMut = after.parse().expect("parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents"); + assert!( + contents + .get("app_config.__edgezero_chunks.deadbeef.0") + .is_some(), + "pre-seeded chunk key must survive a suspicious-pointer skip: {after}" + ); + assert_eq!( + contents + .get(TEST_CONFIG_ID) + .and_then(toml_edit::Item::as_str), + Some(direct.as_str()), + "new value still written" + ); + } + + /// Dry-run reports the orphan count and writes nothing. + #[cfg(unix)] + #[test] + fn push_config_entries_local_dry_run_reports_orphan_count() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_a)], + &AdapterPushContext::new(), + false, + ) + .expect("seed push"); + let before = fs::read_to_string(&fastly_toml).expect("read"); + + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "y".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:02Z".to_owned())) + .expect("envelope B") + }; + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + true, // dry_run + ) + .expect("dry-run"); + + let combined = out.join("\n"); + assert!( + combined.contains("would delete") && combined.contains("orphan chunks"), + "dry-run must report orphan count: {combined}" + ); + assert_eq!( + fs::read_to_string(&fastly_toml).expect("read"), + before, + "dry-run must not edit fastly.toml" + ); + } + + /// Dry-run of an identical re-push reports zero orphans (new keys + /// equal prior keys — regression for expanding new_keys). + #[cfg(unix)] + #[test] + fn push_config_entries_local_dry_run_identical_repush_counts_zero() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope.clone())], + &AdapterPushContext::new(), + false, + ) + .expect("seed push"); + + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + true, // dry_run, same bytes + ) + .expect("dry-run"); + + assert!( + out.join("\n").contains("would delete 0 orphan chunks"), + "identical re-push must count 0 orphans: {out:?}" + ); + } + + /// Dry-run over a suspicious prior pointer reports an unknown count + /// and does not fail. + #[cfg(unix)] + #[test] + fn push_config_entries_local_dry_run_suspicious_prior_pointer_unknown() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + let seed = concat!( + "name = \"demo\"\n\n", + "[local_server.config_stores.app_config]\n", + "format = \"inline-toml\"\n\n", + "[local_server.config_stores.app_config.contents]\n", + "app_config = \"{\\\"edgezero_kind\\\":\\\"fastly_config_chunks\\\",\\\"version\\\":2}\"\n", + ); + fs::write(&fastly_toml, seed).expect("seed"); + + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), direct)], + &AdapterPushContext::new(), + true, // dry_run + ) + .expect("dry-run must not fail on suspicious pointer"); + + assert!( + out.join("\n").contains("unknown: suspicious prior pointer"), + "dry-run must degrade to unknown: {out:?}" + ); + } } From 77a30ec5a4fede99a3d77688d8a2580af2dbb6b8 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 10:30:31 -0700 Subject: [PATCH 13/25] feat(fastly): reclaim prior config chunks after cloud re-push (last-writer-wins) push_config_entries rejects reserved keys, reads each root's prior value before commit, and after the commit sweeps orphaned chunks guarded by a post-commit root read-back (deletes only while still the last writer; yields otherwise). Adds delete_config_store_entry (--key --auto-yes, never --all) and an offline dry-run GC-intent line. Failed deletes and suspicious/absent priors degrade to warnings; the push still succeeds. Adds a command-aware fake fastly harness and 7 cloud GC tests. --- crates/edgezero-adapter-fastly/src/cli.rs | 492 +++++++++++++++++++++- 1 file changed, 476 insertions(+), 16 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 699d6f9c..cf65dc97 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -374,21 +374,24 @@ impl Adapter for FastlyCliAdapter { "no config entries to push to fastly config-store `{name}` (logical id `{logical}`)" )]); } - // Expand each logical (key, envelope_json) into physical entries - // via the chunk-pointer helper. Entries ≤ 8 000 chars go through - // as a single direct entry; larger envelopes are split into - // content-addressed chunks with a root pointer written LAST. - // Collect all physical entries before any writes so pointer-too- - // large errors surface before touching the remote store. + // Reject reserved keys before any expansion or I/O. + reject_reserved_root_keys(entries)?; + // Expand each logical root once: flatten for the commit, and keep + // the exact per-root keep-set + the value written at the root key + // for GC (no prefix scan of the flattened set). Collecting all + // physical entries first also surfaces pointer-too-large errors + // before touching the remote store. let mut physical_entries: Vec<(String, String)> = Vec::new(); + let mut roots: Vec<(String, HashSet, String)> = Vec::with_capacity(entries.len()); for (key, body) in entries { - let expanded = prepare_fastly_config_entries(key, body)?; + let (expanded, new_keys, new_root_value) = expand_root(key, body)?; physical_entries.extend(expanded); + roots.push((key.clone(), new_keys, new_root_value)); } if dry_run { - // Report intent without shelling out. One line per logical key - // noting whether it would be direct or chunked, plus chunk count. - let mut out = Vec::with_capacity(entries.len().saturating_add(1)); + // Report intent without shelling out. Stays fully offline: no + // store-id resolution, no remote read (so no GC count). + let mut out = Vec::with_capacity(entries.len().saturating_mul(2).saturating_add(1)); out.push(format!( "would resolve fastly config-store `{name}` (logical id `{logical}`) via `fastly config-store list --json` and push entries:" )); @@ -407,18 +410,77 @@ impl Adapter for FastlyCliAdapter { body.len() )); } + out.push(format!( + " would delete orphaned prior-generation chunks of `{key}` (count determined at push time)" + )); } return Ok(out); } let resolved_id = resolve_remote_config_store_id(name)?; + // Build GC plans BEFORE the commit: read each root's prior value so + // we know which chunks the old pointer referenced. + let mut gc_plans: Vec = Vec::with_capacity(roots.len()); + for (root_key, new_keys, new_root_value) in roots { + let prior_keys = match fetch_remote_config_store_entry(&resolved_id, &root_key) { + Ok(Some(raw)) => prior_chunk_keys(&root_key, &raw), + Ok(None) => Ok(Vec::new()), + Err(err) => Err(format!( + "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" + )), + }; + gc_plans.push(FastlyConfigGcPlan { + root_key, + prior_keys, + new_keys, + new_root_value, + }); + } + // Commit all physical entries (each root's chunks first, its root + // pointer last). push_entries_with_committer(&physical_entries, |key, value| { create_config_store_entry(&resolved_id, key, value) })?; - Ok(vec![format!( + // Post-commit sweep, last-writer-wins: delete a root's orphans only + // while a read-back confirms the root still holds exactly what we + // wrote. If a newer push has superseded it, yield (delete nothing). + let mut warnings = Vec::new(); + for plan in &gc_plans { + let orphans = match orphan_chunk_keys(plan) { + Ok(keys) if !keys.is_empty() => keys, + Ok(_) => continue, + Err(err) => { + warnings.push(format!("warning: {err}")); + continue; + } + }; + match fetch_remote_config_store_entry(&resolved_id, &plan.root_key) { + Ok(Some(current)) if current == plan.new_root_value => { + for key in orphans { + if let Err(err) = delete_config_store_entry(&resolved_id, &key) { + warnings.push(format!( + "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be removed by a future `config gc`", + plan.root_key + )); + } + } + } + Ok(_) => warnings.push(format!( + "note: skipped chunk GC for `{}`: root changed since this push wrote it (concurrent push?); orphans left for a future `config gc`", + plan.root_key + )), + Err(err) => warnings.push(format!( + "note: skipped chunk GC for `{}`: could not re-read root before sweep ({err}); orphans left for a future `config gc`", + plan.root_key + )), + } + } + let mut out = vec![format!( "pushed {} physical entries ({} logical) to fastly config-store `{name}` (logical id `{logical}`, id={resolved_id})", physical_entries.len(), entries.len() - )]) + )]; + out.extend(warnings); + Ok(out) } fn push_config_entries_local( @@ -1294,6 +1356,46 @@ fn create_config_store_entry(store_id: &str, key: &str, value: &str) -> Result<( )) } +/// Shell `fastly config-store-entry delete --store-id= --key= +/// --auto-yes` for a single orphan chunk. Only ever passes `--key`: the +/// subcommand also accepts `-a/--all`, which wipes EVERY entry in the +/// store — never construct that flag here. `--auto-yes` suppresses the +/// interactive confirmation. A "not found" / "does not exist" / "404" +/// stderr is treated as success (the entry is already gone). +fn delete_config_store_entry(store_id: &str, key: &str) -> Result<(), String> { + let store_arg = format!("--store-id={store_id}"); + let key_arg = format!("--key={key}"); + let output = Command::new("fastly") + .args([ + "config-store-entry", + "delete", + store_arg.as_str(), + key_arg.as_str(), + "--auto-yes", + ]) + .output() + .map_err(|err| { + if err.kind() == ErrorKind::NotFound { + format!("`fastly` not found on PATH; {FASTLY_INSTALL_HINT}") + } else { + format!("failed to spawn `fastly`: {err}") + } + })?; + if output.status.success() { + return Ok(()); + } + let stderr = String::from_utf8_lossy(&output.stderr); + let lower = stderr.to_ascii_lowercase(); + if lower.contains("not found") || lower.contains("does not exist") || lower.contains("404") { + return Ok(()); + } + Err(format!( + "`fastly config-store-entry delete --store-id={store_id} --key={key} --auto-yes` exited with status {}\nstderr: {}", + output.status, + stderr.trim() + )) +} + /// Parse `fastly config-store list --json` output and return the /// platform `id` of the store whose `name` matches `name`. Accepts /// both a bare array (`[ {"id": "...", "name": "..."}, ... ]`) @@ -2483,10 +2585,14 @@ build = \"cargo build --release\" true, ) .expect("dry-run succeeds"); - // First line names the resolve+publish flow; subsequent lines preview - // each key the push would create (so callers can eyeball the keyset - // before running for real). - assert_eq!(out.len(), 1 + entries.len(), "header + per-entry preview"); + // First line names the resolve+publish flow; then per key a preview + // line plus an offline GC-intent line (so callers can eyeball the + // keyset before running for real). + assert_eq!( + out.len(), + 1 + entries.len() * 2, + "header + per-entry preview + per-entry GC-intent" + ); assert!( out[0].contains("would resolve fastly config-store `app_config`") && out[0].contains("push entries"), @@ -3010,6 +3116,360 @@ build = \"cargo build --release\" fake_dir } + /// Fake `fastly` for cloud chunk-GC tests. Logs each + /// `config-store-entry` op ("describe " / "update " / + /// "delete ") to `oplog`, one per line. `root_describe_seq` + /// gives the successive raw `item_value`s returned when the ROOT key + /// is described (call 1 = prior read, call 2 = read-back guard); + /// exhausting the sequence yields "not found". `fail_delete_key` + /// makes that one delete exit non-zero. + #[cfg(unix)] + fn fake_fastly_gc( + root_key: &str, + root_describe_seq: &[String], + fail_delete_key: Option<&str>, + oplog: &Path, + ) -> tempfile::TempDir { + use std::os::unix::fs::PermissionsExt as _; + let dir = tempdir().expect("tempdir"); + let list_file = dir.path().join("list.json"); + fs::write( + &list_file, + format!(r#"[{{"name":"{TEST_CONFIG_ID}","id":"store-abc123"}}]"#), + ) + .expect("list"); + for (i, value) in root_describe_seq.iter().enumerate() { + let wrapped = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(value).expect("escape") + ); + fs::write( + dir.path().join(format!("resp_{root_key}_{}.json", i + 1)), + wrapped, + ) + .expect("resp"); + } + let template = r#"#!/bin/sh +if [ "$1" = "config-store" ]; then cat '@LIST@'; exit 0; fi +sub="$2" +key="" +for arg in "$@"; do case "$arg" in --key=*) key="${arg#--key=}";; esac; done +if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '@OPLOG@'; exit 0; fi +if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '@OPLOG@'; if [ "$key" = "@FAIL@" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi +if [ "$sub" = "describe" ]; then + printf 'describe %s\n' "$key" >> '@OPLOG@' + cfile='@DIR@/count_'"$key" + n=0; [ -f "$cfile" ] && n=$(cat "$cfile"); n=$((n+1)); printf '%s' "$n" > "$cfile" + rf='@DIR@/resp_'"$key"'_'"$n"'.json' + if [ -f "$rf" ]; then cat "$rf"; exit 0; fi + echo 'Error: item not found' >&2; exit 1 +fi +echo 'unexpected' >&2; exit 1 +"#; + let script = template + .replace("@LIST@", &list_file.display().to_string()) + .replace("@OPLOG@", &oplog.display().to_string()) + .replace("@DIR@", &dir.path().display().to_string()) + .replace("@FAIL@", fail_delete_key.unwrap_or("")); + let sp = dir.path().join("fastly"); + fs::write(&sp, script).expect("script"); + let mut perms = fs::metadata(&sp).expect("meta").permissions(); + perms.set_mode(0o755); + fs::set_permissions(&sp, perms).expect("chmod"); + dir + } + + /// Split a chunked envelope into (chunk keys, root pointer value). + #[cfg(unix)] + fn chunked_parts(root_key: &str, envelope: &str) -> (Vec, String) { + let entries = prepare_fastly_config_entries(root_key, envelope).expect("expand"); + let (_, pointer) = entries.last().expect("pointer").clone(); + let chunk_keys = entries[..entries.len().saturating_sub(1)] + .iter() + .map(|(k, _)| k.clone()) + .collect(); + (chunk_keys, pointer) + } + + #[cfg(unix)] + fn oplog_has(oplog: &Path, line: &str) -> bool { + fs::read_to_string(oplog) + .unwrap_or_default() + .lines() + .any(|l| l == line) + } + + #[cfg(unix)] + #[test] + fn push_config_entries_deletes_prior_chunks_and_keeps_new() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "z".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:03Z".to_owned())) + .expect("B") + }; + let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + + // describe#1 = prior A pointer; describe#2 (read-back) = new B pointer. + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, &oplog); + let _path = PathPrepend::new(fake.path()); + + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push must succeed"); + + // Every prior A-chunk is deleted; the root and new B-chunks are not. + for key in &a_chunks { + assert!( + oplog_has(&oplog, &format!("delete {key}")), + "prior chunk `{key}` must be deleted; log:\n{}", + fs::read_to_string(&oplog).unwrap_or_default() + ); + } + assert!( + !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), + "root pointer must never be deleted" + ); + for key in &b_chunks { + assert!( + !oplog_has(&oplog, &format!("delete {key}")), + "new chunk `{key}` must not be deleted" + ); + } + // Deletes happen strictly after the root-pointer update (commit). + let log = fs::read_to_string(&oplog).unwrap_or_default(); + let root_update = log + .lines() + .position(|l| l == format!("update {TEST_CONFIG_ID}")) + .expect("root update logged"); + let first_delete = log + .lines() + .position(|l| l.starts_with("delete ")) + .expect("a delete logged"); + assert!( + first_delete > root_update, + "deletes must follow the root update; log:\n{log}" + ); + assert!(out[0].contains("pushed"), "summary present: {out:?}"); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_skips_gc_when_root_changed() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "q".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:04Z".to_owned())) + .expect("B") + }; + let (_a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + + // Read-back returns the OLD A pointer (a concurrent push reverted the + // root), which differs from what this push wrote -> GC yields. + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer.clone(), a_pointer], + None, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push must succeed"); + + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|l| l.starts_with("delete ")), + "no deletes when the root changed under us; log:\n{log}" + ); + assert!( + out.iter().any(|l| l.contains("root changed")), + "must warn that GC was skipped: {out:?}" + ); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_rejects_reserved_key() { + let dir = tempdir().expect("tempdir"); + let bad_key = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); + let err = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(bad_key.clone(), "{}".to_owned())], + &AdapterPushContext::new(), + false, + ) + .expect_err("reserved key must be rejected"); + assert!(err.contains(&bad_key), "names the key: {err}"); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_dry_run_reports_gc_intent_offline() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + // No fake on PATH: a dry-run must not shell out to fastly at all. + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + true, + ) + .expect("dry-run must not error"); + assert!( + out.iter() + .any(|l| l.contains("would delete orphaned prior-generation chunks")), + "dry-run reports GC intent: {out:?}" + ); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_no_prior_issues_no_deletes() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + // Empty describe sequence => root not found => no prior chunks. + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, &oplog); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|l| l.starts_with("delete ")), + "no prior => no deletes; log:\n{log}" + ); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_delete_failure_warns_but_succeeds() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "w".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:05Z".to_owned())) + .expect("B") + }; + let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let fail_key = a_chunks[0].clone(); + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer, b_pointer], + Some(&fail_key), + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push still succeeds despite a failed delete"); + assert!( + out.iter() + .any(|l| l.contains("could not reclaim orphan chunk") && l.contains(&fail_key)), + "failed delete surfaces an informational warning: {out:?}" + ); + } + + #[cfg(unix)] + #[test] + fn push_config_entries_suspicious_prior_warns_no_deletes() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope_b = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + // Prior is pointer-kind but invalid (version 2) => warn, no deletes. + let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad], None, &oplog); + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|l| l.starts_with("delete ")), + "suspicious prior => no deletes; log:\n{log}" + ); + assert!( + out.iter().any(|l| l.contains("skipping chunk GC")), + "warns about the suspicious pointer: {out:?}" + ); + } + #[cfg(unix)] #[test] fn push_config_entries_writes_direct_entry_at_exactly_8000_chars() { From 2c36b5d582c1a0018bfff3d3fc730f52d38281bd Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 13:56:36 -0700 Subject: [PATCH 14/25] chore(fastly): satisfy strict clippy for chunk-GC code; render fake fastly via handlebars Moves FastlyConfigGcPlan to the struct group with alphabetical fields; renames single-char closure idents; replaces bare arithmetic with saturating_add; fixes map_err/shadow/assert-on-result-state/absolute-path lints; relocates GC helper unit tests after the test-module structs. Adds handlebars dev-dependency and rewrites the cloud fake-fastly test shim to render its shell script from a handlebars template. --- Cargo.lock | 1 + crates/edgezero-adapter-fastly/Cargo.toml | 1 + crates/edgezero-adapter-fastly/src/cli.rs | 268 ++++++++++++---------- 3 files changed, 144 insertions(+), 126 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 2100fc07..fae5fac2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -719,6 +719,7 @@ dependencies = [ "flate2", "futures", "futures-util", + "handlebars", "log", "log-fastly", "serde", diff --git a/crates/edgezero-adapter-fastly/Cargo.toml b/crates/edgezero-adapter-fastly/Cargo.toml index 4fee1797..efd50c74 100644 --- a/crates/edgezero-adapter-fastly/Cargo.toml +++ b/crates/edgezero-adapter-fastly/Cargo.toml @@ -47,4 +47,5 @@ walkdir = { workspace = true, optional = true } [dev-dependencies] edgezero-core = { path = "../edgezero-core", features = ["test-utils"] } +handlebars = { workspace = true } tempfile = { workspace = true } diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index cf65dc97..397b5023 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -147,6 +147,22 @@ enum ConfigStoreLookup { SchemaDrift(String), } +/// Per-root plan for reclaiming a prior generation's chunk entries on +/// re-push (Stage 7). Built by the config-push paths; consumed by the +/// GC helpers `expand_root` / `orphan_chunk_keys`. +struct FastlyConfigGcPlan { + /// Exact keep-set this push writes for the root (chunk keys + root key). + new_keys: HashSet, + /// Exactly the value written at `root_key` (its new pointer, or the + /// direct envelope). Cloud uses it as a read-back concurrency guard + /// (last-writer-wins); local ignores it. + new_root_value: String, + /// Prior chunk keys to consider deleting, or a warning to surface + /// (suspicious prior pointer) that skips GC for this root. + prior_keys: Result, String>, + root_key: String, +} + // The three `validate_*` trait methods exist on `Adapter` because // spin requires them (variable-name regex, `[component.*]` // discovery, flat-namespace collision). The trait surface is typed @@ -429,10 +445,10 @@ impl Adapter for FastlyCliAdapter { )), }; gc_plans.push(FastlyConfigGcPlan { - root_key, - prior_keys, new_keys, new_root_value, + prior_keys, + root_key, }); } // Commit all physical entries (each root's chunks first, its root @@ -1083,12 +1099,12 @@ fn write_fastly_local_config_store( let prior_keys = contents_tbl .get(root_key) .and_then(toml_edit::Item::as_str) - .map_or_else(|| Ok(Vec::new()), |raw| prior_chunk_keys(root_key, raw)); + .map_or_else(|| Ok(Vec::new()), |value| prior_chunk_keys(root_key, value)); plans.push(FastlyConfigGcPlan { - root_key: root_key.clone(), - prior_keys, new_keys: new_keys.clone(), new_root_value: String::new(), // unused locally (no remote concurrency) + prior_keys, + root_key: root_key.clone(), }); } @@ -1120,34 +1136,24 @@ fn write_fastly_local_config_store( // chunk GC helpers (Stage 7 re-push reclamation) // ------------------------------------------------------------------- -/// Per-root plan for reclaiming a prior generation's chunk entries. -struct FastlyConfigGcPlan { - root_key: String, - /// Prior chunk keys to consider deleting, or a warning to surface - /// (suspicious prior pointer) that skips GC for this root. - prior_keys: Result, String>, - /// Exact keep-set this push writes for the root (chunk keys + root key). - new_keys: HashSet, - /// Exactly the value written at `root_key` (its new pointer, or the - /// direct envelope). Cloud uses it as a read-back concurrency guard - /// (last-writer-wins); local ignores it. - new_root_value: String, -} - /// Expand ONE logical `(root_key, body)` into its physical entries, the /// exact keep-set for that root, and the value written at the root key. /// No cross-root prefix scanning (a free-form `--key` can't mislead it). +#[expect( + clippy::type_complexity, + reason = "one-off internal return; a named type would not aid readability" +)] fn expand_root( root_key: &str, body: &str, ) -> Result<(Vec<(String, String)>, HashSet, String), String> { let expanded = prepare_fastly_config_entries(root_key, body)?; - let new_keys: HashSet = expanded.iter().map(|(k, _)| k.clone()).collect(); + let new_keys: HashSet = expanded.iter().map(|(key, _)| key.clone()).collect(); // prepare_* always emits the root entry LAST (root pointer or direct // value). Make the invariant explicit rather than silently defaulting. let new_root_value = expanded .last() - .map(|(_, v)| v.clone()) + .map(|(_, value)| value.clone()) .ok_or_else(|| format!("internal: no physical entries produced for root `{root_key}`"))?; Ok((expanded, new_keys, new_root_value)) } @@ -1199,7 +1205,7 @@ fn local_orphan_counts_for_dry_run( Ok(text) => text .parse::() .map(Some) - .map_err(|_| "could not read prior state".to_owned()), + .map_err(|_err| "could not read prior state".to_owned()), Err(err) if err.kind() == ErrorKind::NotFound => Ok(None), Err(_) => Err("could not read prior state".to_owned()), }; @@ -1227,7 +1233,7 @@ fn local_orphan_counts_for_dry_run( None => Err("could not read prior state".to_owned()), Some(raw) => match prior_chunk_keys(root_key, raw) { Ok(prior) => { - Ok(prior.iter().filter(|k| !new_keys.contains(*k)).count()) + Ok(prior.iter().filter(|key| !new_keys.contains(*key)).count()) } Err(_) => Err("suspicious prior pointer".to_owned()), }, @@ -1669,6 +1675,7 @@ pub fn serve(extra_args: &[String]) -> Result<(), String> { mod tests { use super::*; use edgezero_adapter::cli_support::read_package_name; + use std::collections::HashSet; #[cfg(unix)] use std::ffi::OsString; #[cfg(unix)] @@ -1685,79 +1692,6 @@ mod tests { const TEST_CONFIG_ID: &str = "app_config"; const TEST_SECRET_ID: &str = "default"; - // ---- chunk GC helpers ---- - - #[test] - fn reject_reserved_root_keys_accepts_clean_keys() { - let entries = vec![ - ("app_config".to_owned(), "{}".to_owned()), - ("app_config_staging".to_owned(), "{}".to_owned()), - ]; - assert!(reject_reserved_root_keys(&entries).is_ok()); - } - - #[test] - fn reject_reserved_root_keys_rejects_infix_key() { - let bad = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); - let entries = vec![(bad.clone(), "{}".to_owned())]; - let err = reject_reserved_root_keys(&entries).expect_err("reserved infix must reject"); - assert!(err.contains(&bad), "error names the key: {err}"); - assert!(err.contains("reserved"), "error explains why: {err}"); - } - - #[test] - fn orphan_chunk_keys_subtracts_new_keys() { - let mut new_keys = std::collections::HashSet::new(); - new_keys.insert("keep".to_owned()); - let plan = FastlyConfigGcPlan { - root_key: "app_config".to_owned(), - prior_keys: Ok(vec![ - "gone1".to_owned(), - "keep".to_owned(), - "gone2".to_owned(), - ]), - new_keys, - new_root_value: String::new(), - }; - let orphans = orphan_chunk_keys(&plan).expect("ok"); - assert_eq!(orphans, vec!["gone1".to_owned(), "gone2".to_owned()]); - } - - #[test] - fn orphan_chunk_keys_propagates_prior_err() { - let plan = FastlyConfigGcPlan { - root_key: "app_config".to_owned(), - prior_keys: Err("suspicious".to_owned()), - new_keys: std::collections::HashSet::new(), - new_root_value: String::new(), - }; - assert!(orphan_chunk_keys(&plan).is_err()); - } - - #[test] - fn expand_root_direct_value_has_single_entry() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); - let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); - assert_eq!(expanded.len(), 1); - assert_eq!(new_root_value, envelope); - assert!(new_keys.contains(TEST_CONFIG_ID)); - assert_eq!(new_keys.len(), 1); - } - - #[test] - fn expand_root_chunked_value_carries_pointer_as_root_value() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); - assert!(expanded.len() >= 2, "chunks + pointer"); - let (last_key, last_val) = expanded.last().unwrap(); - assert_eq!(last_key, TEST_CONFIG_ID); - assert_eq!(&new_root_value, last_val); - assert!(new_keys.contains(TEST_CONFIG_ID)); - assert_eq!(new_keys.len(), expanded.len()); - } - /// RAII guard: prepends a directory to `$PATH` and restores the original /// value on drop. #[cfg(unix)] @@ -3138,44 +3072,52 @@ build = \"cargo build --release\" format!(r#"[{{"name":"{TEST_CONFIG_ID}","id":"store-abc123"}}]"#), ) .expect("list"); - for (i, value) in root_describe_seq.iter().enumerate() { + for (index, value) in root_describe_seq.iter().enumerate() { let wrapped = format!( r#"{{"item_value":{}}}"#, serde_json::to_string(value).expect("escape") ); + let nth = index.saturating_add(1); fs::write( - dir.path().join(format!("resp_{root_key}_{}.json", i + 1)), + dir.path().join(format!("resp_{root_key}_{nth}.json")), wrapped, ) .expect("resp"); } - let template = r#"#!/bin/sh -if [ "$1" = "config-store" ]; then cat '@LIST@'; exit 0; fi + // Rendered with handlebars. Triple-stache `{{{ }}}` disables HTML + // escaping (paths are not markup); the shell's own `${var}` / + // `$(( ))` use single braces so they are literal text to handlebars. + const TEMPLATE: &str = r#"#!/bin/sh +if [ "$1" = "config-store" ]; then cat '{{{list}}}'; exit 0; fi sub="$2" key="" for arg in "$@"; do case "$arg" in --key=*) key="${arg#--key=}";; esac; done -if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '@OPLOG@'; exit 0; fi -if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '@OPLOG@'; if [ "$key" = "@FAIL@" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi +if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '{{{oplog}}}'; exit 0; fi +if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi if [ "$sub" = "describe" ]; then - printf 'describe %s\n' "$key" >> '@OPLOG@' - cfile='@DIR@/count_'"$key" + printf 'describe %s\n' "$key" >> '{{{oplog}}}' + cfile='{{{dir}}}/count_'"$key" n=0; [ -f "$cfile" ] && n=$(cat "$cfile"); n=$((n+1)); printf '%s' "$n" > "$cfile" - rf='@DIR@/resp_'"$key"'_'"$n"'.json' + rf='{{{dir}}}/resp_'"$key"'_'"$n"'.json' if [ -f "$rf" ]; then cat "$rf"; exit 0; fi echo 'Error: item not found' >&2; exit 1 fi echo 'unexpected' >&2; exit 1 "#; - let script = template - .replace("@LIST@", &list_file.display().to_string()) - .replace("@OPLOG@", &oplog.display().to_string()) - .replace("@DIR@", &dir.path().display().to_string()) - .replace("@FAIL@", fail_delete_key.unwrap_or("")); - let sp = dir.path().join("fastly"); - fs::write(&sp, script).expect("script"); - let mut perms = fs::metadata(&sp).expect("meta").permissions(); + let data = serde_json::json!({ + "list": list_file.display().to_string(), + "oplog": oplog.display().to_string(), + "dir": dir.path().display().to_string(), + "fail": fail_delete_key.unwrap_or(""), + }); + let script = handlebars::Handlebars::new() + .render_template(TEMPLATE, &data) + .expect("render fake fastly script"); + let script_path = dir.path().join("fastly"); + fs::write(&script_path, script).expect("script"); + let mut perms = fs::metadata(&script_path).expect("meta").permissions(); perms.set_mode(0o755); - fs::set_permissions(&sp, perms).expect("chmod"); + fs::set_permissions(&script_path, perms).expect("chmod"); dir } @@ -3186,7 +3128,7 @@ echo 'unexpected' >&2; exit 1 let (_, pointer) = entries.last().expect("pointer").clone(); let chunk_keys = entries[..entries.len().saturating_sub(1)] .iter() - .map(|(k, _)| k.clone()) + .map(|(key, _)| key.clone()) .collect(); (chunk_keys, pointer) } @@ -3196,7 +3138,7 @@ echo 'unexpected' >&2; exit 1 fs::read_to_string(oplog) .unwrap_or_default() .lines() - .any(|l| l == line) + .any(|entry| entry == line) } #[cfg(unix)] @@ -3256,11 +3198,11 @@ echo 'unexpected' >&2; exit 1 let log = fs::read_to_string(&oplog).unwrap_or_default(); let root_update = log .lines() - .position(|l| l == format!("update {TEST_CONFIG_ID}")) + .position(|line| line == format!("update {TEST_CONFIG_ID}")) .expect("root update logged"); let first_delete = log .lines() - .position(|l| l.starts_with("delete ")) + .position(|line| line.starts_with("delete ")) .expect("a delete logged"); assert!( first_delete > root_update, @@ -3311,11 +3253,11 @@ echo 'unexpected' >&2; exit 1 let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( - !log.lines().any(|l| l.starts_with("delete ")), + !log.lines().any(|line| line.starts_with("delete ")), "no deletes when the root changed under us; log:\n{log}" ); assert!( - out.iter().any(|l| l.contains("root changed")), + out.iter().any(|line| line.contains("root changed")), "must warn that GC was skipped: {out:?}" ); } @@ -3359,7 +3301,7 @@ echo 'unexpected' >&2; exit 1 .expect("dry-run must not error"); assert!( out.iter() - .any(|l| l.contains("would delete orphaned prior-generation chunks")), + .any(|line| line.contains("would delete orphaned prior-generation chunks")), "dry-run reports GC intent: {out:?}" ); } @@ -3388,7 +3330,7 @@ echo 'unexpected' >&2; exit 1 .expect("push succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( - !log.lines().any(|l| l.starts_with("delete ")), + !log.lines().any(|line| line.starts_with("delete ")), "no prior => no deletes; log:\n{log}" ); } @@ -3431,7 +3373,8 @@ echo 'unexpected' >&2; exit 1 .expect("push still succeeds despite a failed delete"); assert!( out.iter() - .any(|l| l.contains("could not reclaim orphan chunk") && l.contains(&fail_key)), + .any(|line| line.contains("could not reclaim orphan chunk") + && line.contains(&fail_key)), "failed delete surfaces an informational warning: {out:?}" ); } @@ -3461,11 +3404,11 @@ echo 'unexpected' >&2; exit 1 .expect("push succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( - !log.lines().any(|l| l.starts_with("delete ")), + !log.lines().any(|line| line.starts_with("delete ")), "suspicious prior => no deletes; log:\n{log}" ); assert!( - out.iter().any(|l| l.contains("skipping chunk GC")), + out.iter().any(|line| line.contains("skipping chunk GC")), "warns about the suspicious pointer: {out:?}" ); } @@ -4365,7 +4308,7 @@ echo 'unexpected' >&2; exit 1 } /// Dry-run of an identical re-push reports zero orphans (new keys - /// equal prior keys — regression for expanding new_keys). + /// equal prior keys — regression for expanding `new_keys`). #[cfg(unix)] #[test] fn push_config_entries_local_dry_run_identical_repush_counts_zero() { @@ -4440,4 +4383,77 @@ echo 'unexpected' >&2; exit 1 "dry-run must degrade to unknown: {out:?}" ); } + + // ---- chunk GC helpers ---- + + #[test] + fn reject_reserved_root_keys_accepts_clean_keys() { + let entries = vec![ + ("app_config".to_owned(), "{}".to_owned()), + ("app_config_staging".to_owned(), "{}".to_owned()), + ]; + reject_reserved_root_keys(&entries).expect("clean keys accepted"); + } + + #[test] + fn reject_reserved_root_keys_rejects_infix_key() { + let bad = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); + let entries = vec![(bad.clone(), "{}".to_owned())]; + let err = reject_reserved_root_keys(&entries).expect_err("reserved infix must reject"); + assert!(err.contains(&bad), "error names the key: {err}"); + assert!(err.contains("reserved"), "error explains why: {err}"); + } + + #[test] + fn orphan_chunk_keys_subtracts_new_keys() { + let mut new_keys = HashSet::new(); + new_keys.insert("keep".to_owned()); + let plan = FastlyConfigGcPlan { + new_keys, + new_root_value: String::new(), + prior_keys: Ok(vec![ + "gone1".to_owned(), + "keep".to_owned(), + "gone2".to_owned(), + ]), + root_key: "app_config".to_owned(), + }; + let orphans = orphan_chunk_keys(&plan).expect("ok"); + assert_eq!(orphans, vec!["gone1".to_owned(), "gone2".to_owned()]); + } + + #[test] + fn orphan_chunk_keys_propagates_prior_err() { + let plan = FastlyConfigGcPlan { + new_keys: HashSet::new(), + new_root_value: String::new(), + prior_keys: Err("suspicious".to_owned()), + root_key: "app_config".to_owned(), + }; + orphan_chunk_keys(&plan).unwrap_err(); + } + + #[test] + fn expand_root_direct_value_has_single_entry() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); + assert_eq!(expanded.len(), 1); + assert_eq!(new_root_value, envelope); + assert!(new_keys.contains(TEST_CONFIG_ID)); + assert_eq!(new_keys.len(), 1); + } + + #[test] + fn expand_root_chunked_value_carries_pointer_as_root_value() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let (expanded, new_keys, new_root_value) = expand_root(TEST_CONFIG_ID, &envelope).unwrap(); + assert!(expanded.len() >= 2, "chunks + pointer"); + let (last_key, last_value) = expanded.last().unwrap(); + assert_eq!(last_key, TEST_CONFIG_ID); + assert_eq!(&new_root_value, last_value); + assert!(new_keys.contains(TEST_CONFIG_ID)); + assert_eq!(new_keys.len(), expanded.len()); + } } From 0f516dd4d72f38902256fb5e2d734344ca8473de Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 14:01:01 -0700 Subject: [PATCH 15/25] chore(fastly): hoist fake-fastly TEMPLATE const above statements (items_after_statements) --- crates/edgezero-adapter-fastly/src/cli.rs | 38 +++++++++++------------ 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 397b5023..dfb48f70 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -3065,25 +3065,6 @@ build = \"cargo build --release\" oplog: &Path, ) -> tempfile::TempDir { use std::os::unix::fs::PermissionsExt as _; - let dir = tempdir().expect("tempdir"); - let list_file = dir.path().join("list.json"); - fs::write( - &list_file, - format!(r#"[{{"name":"{TEST_CONFIG_ID}","id":"store-abc123"}}]"#), - ) - .expect("list"); - for (index, value) in root_describe_seq.iter().enumerate() { - let wrapped = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(value).expect("escape") - ); - let nth = index.saturating_add(1); - fs::write( - dir.path().join(format!("resp_{root_key}_{nth}.json")), - wrapped, - ) - .expect("resp"); - } // Rendered with handlebars. Triple-stache `{{{ }}}` disables HTML // escaping (paths are not markup); the shell's own `${var}` / // `$(( ))` use single braces so they are literal text to handlebars. @@ -3104,6 +3085,25 @@ if [ "$sub" = "describe" ]; then fi echo 'unexpected' >&2; exit 1 "#; + let dir = tempdir().expect("tempdir"); + let list_file = dir.path().join("list.json"); + fs::write( + &list_file, + format!(r#"[{{"name":"{TEST_CONFIG_ID}","id":"store-abc123"}}]"#), + ) + .expect("list"); + for (index, value) in root_describe_seq.iter().enumerate() { + let wrapped = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(value).expect("escape") + ); + let nth = index.saturating_add(1); + fs::write( + dir.path().join(format!("resp_{root_key}_{nth}.json")), + wrapped, + ) + .expect("resp"); + } let data = serde_json::json!({ "list": list_file.display().to_string(), "oplog": oplog.display().to_string(), From 205048dd6841cf4cc20de1fbf7b40161d10d53cd Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 14:27:00 -0700 Subject: [PATCH 16/25] test(fastly): close self-review coverage gaps - local: GC of a chunked root leaves a chunked sibling's chunks intact (prefix-scoping vs shared string prefix app_config / app_config_staging) - cloud: identical-bytes re-push deletes nothing (read-back returns our own value, so the assertion is non-vacuous) - cloud: prior-read failure warns and deletes nothing (extends the fake fastly with a describe_hard_error mode) --- crates/edgezero-adapter-fastly/src/cli.rs | 148 +++++++++++++++++++++- 1 file changed, 144 insertions(+), 4 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index dfb48f70..9ff21a61 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -3056,12 +3056,16 @@ build = \"cargo build --release\" /// gives the successive raw `item_value`s returned when the ROOT key /// is described (call 1 = prior read, call 2 = read-back guard); /// exhausting the sequence yields "not found". `fail_delete_key` - /// makes that one delete exit non-zero. + /// makes that one delete exit non-zero. `describe_hard_error` makes + /// every describe fail with a non-not-found error (so + /// `fetch_remote_config_store_entry` returns `Err`, exercising the + /// prior-read-failure GC path). #[cfg(unix)] fn fake_fastly_gc( root_key: &str, root_describe_seq: &[String], fail_delete_key: Option<&str>, + describe_hard_error: bool, oplog: &Path, ) -> tempfile::TempDir { use std::os::unix::fs::PermissionsExt as _; @@ -3077,6 +3081,7 @@ if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '{ if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi if [ "$sub" = "describe" ]; then printf 'describe %s\n' "$key" >> '{{{oplog}}}' + {{#if hard_error}}echo 'Error: internal server error' >&2; exit 1{{/if}} cfile='{{{dir}}}/count_'"$key" n=0; [ -f "$cfile" ] && n=$(cat "$cfile"); n=$((n+1)); printf '%s' "$n" > "$cfile" rf='{{{dir}}}/resp_'"$key"'_'"$n"'.json' @@ -3109,6 +3114,7 @@ echo 'unexpected' >&2; exit 1 "oplog": oplog.display().to_string(), "dir": dir.path().display().to_string(), "fail": fail_delete_key.unwrap_or(""), + "hard_error": describe_hard_error, }); let script = handlebars::Handlebars::new() .render_template(TEMPLATE, &data) @@ -3161,7 +3167,7 @@ echo 'unexpected' >&2; exit 1 let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); // describe#1 = prior A pointer; describe#2 (read-back) = new B pointer. - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, &oplog); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, false, &oplog); let _path = PathPrepend::new(fake.path()); let out = FastlyCliAdapter @@ -3235,6 +3241,7 @@ echo 'unexpected' >&2; exit 1 TEST_CONFIG_ID, &[a_pointer.clone(), a_pointer], None, + false, &oplog, ); let _path = PathPrepend::new(fake.path()); @@ -3315,7 +3322,7 @@ echo 'unexpected' >&2; exit 1 let oplog = dir.path().join("ops.log"); let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); // Empty describe sequence => root not found => no prior chunks. - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, &oplog); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, false, &oplog); let _path = PathPrepend::new(fake.path()); FastlyCliAdapter .push_config_entries( @@ -3357,6 +3364,7 @@ echo 'unexpected' >&2; exit 1 TEST_CONFIG_ID, &[a_pointer, b_pointer], Some(&fail_key), + false, &oplog, ); let _path = PathPrepend::new(fake.path()); @@ -3389,7 +3397,7 @@ echo 'unexpected' >&2; exit 1 let envelope_b = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); // Prior is pointer-kind but invalid (version 2) => warn, no deletes. let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad], None, &oplog); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad], None, false, &oplog); let _path = PathPrepend::new(fake.path()); let out = FastlyCliAdapter .push_config_entries( @@ -3413,6 +3421,41 @@ echo 'unexpected' >&2; exit 1 ); } + #[cfg(unix)] + #[test] + fn push_config_entries_prior_read_failure_warns_no_deletes() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + // Describe fails hard (not a not-found) => prior read errors => GC + // is skipped for this root with a warning; the push still succeeds. + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, true, &oplog); + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds despite prior-read failure"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "prior-read failure => no deletes; log:\n{log}" + ); + assert!( + out.iter() + .any(|line| line.contains("failed to read prior root")), + "must warn about the failed prior read: {out:?}" + ); + } + #[cfg(unix)] #[test] fn push_config_entries_writes_direct_entry_at_exactly_8000_chars() { @@ -4384,6 +4427,103 @@ echo 'unexpected' >&2; exit 1 ); } + /// GC of a chunked root must not touch a chunked SIBLING's chunks — + /// the prefix `app_config.__edgezero_chunks.` must not match + /// `app_config_staging.__edgezero_chunks.` (shared string prefix). + #[cfg(unix)] + #[test] + fn push_config_entries_local_gc_preserves_sibling_chunks() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let make = |tag: &str| { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:00Z".to_owned())) + .expect("envelope") + }; + let push = |key: &str, body: String| { + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(key.to_owned(), body)], + &AdapterPushContext::new(), + false, + ) + .expect("push"); + }; + + // app_config gen X, then a chunked sibling, then app_config gen Z. + push("app_config", make("x1")); + push("app_config_staging", make("staging")); + let staging_chunks = { + let (chunks, _) = chunked_parts("app_config_staging", &make("staging")); + chunks + }; + push("app_config", make("z2")); // GCs app_config's gen-X chunks + + let after = fs::read_to_string(&fastly_toml).expect("read"); + let doc: toml_edit::DocumentMut = after.parse().expect("parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents"); + for key in &staging_chunks { + assert!( + contents.get(key).is_some(), + "sibling chunk `{key}` must survive app_config GC: {after}" + ); + } + } + + /// Cloud identical-bytes re-push reclaims nothing (new keys equal prior + /// keys). Read-back returns our own value so the guard would PASS if + /// orphans were miscomputed — making "no deletes" non-vacuous. + #[cfg(unix)] + #[test] + fn push_config_entries_identical_repush_issues_no_deletes() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); + // Prior == what this push writes; read-back also returns it. + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[pointer.clone(), pointer], + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "identical re-push must delete nothing; log:\n{log}" + ); + } + // ---- chunk GC helpers ---- #[test] From 891ecc6c3377ad8674bc1f64325775c5104cada4 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Wed, 8 Jul 2026 17:34:26 -0700 Subject: [PATCH 17/25] fix(fastly): dry-run classifies malformed local state as unknown; test delete argv + cloud shrink-to-direct - local dry-run: distinguish absent (0) from present-but-non-table ("unknown: could not read prior state") via local_contents_table, so --local --dry-run no longer reports 0 orphans for state the real writer would reject; + non-table-contents test - cloud: assert every delete argv passes --key + --auto-yes and NEVER --all (blast radius); fake now logs the full delete argv - cloud: add the shrink-to-direct test (prior chunked -> new direct deletes all prior chunks, root upserted not deleted) --- crates/edgezero-adapter-fastly/src/cli.rs | 204 ++++++++++++++++++++-- 1 file changed, 190 insertions(+), 14 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 9ff21a61..4c0cf531 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -1187,6 +1187,41 @@ fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> } /// Best-effort per-root orphan count for `config push --local --dry-run`. +/// Navigate to `[local_server.config_stores..contents]` for the +/// dry-run counter. `Ok(None)` when any level is absent (no prior state); +/// `Err` when a level is present but the wrong type — prior state the real +/// writer would reject, so the count must degrade to "unknown", not 0. +fn local_contents_table<'doc>( + doc: &'doc toml_edit::DocumentMut, + platform_name: &str, +) -> Result, String> { + let malformed = || "could not read prior state".to_owned(); + let Some(server_item) = doc.get("local_server") else { + return Ok(None); + }; + let Some(server) = server_item.as_table() else { + return Err(malformed()); + }; + let Some(stores_item) = server.get("config_stores") else { + return Ok(None); + }; + let Some(stores) = stores_item.as_table() else { + return Err(malformed()); + }; + let Some(store_item) = stores.get(platform_name) else { + return Ok(None); + }; + let Some(store) = store_item.as_table() else { + return Err(malformed()); + }; + let Some(contents_item) = store.get("contents") else { + return Ok(None); + }; + contents_item + .as_table() + .map_or_else(|| Err(malformed()), |table| Ok(Some(table))) +} + /// Reads the current `fastly.toml` (offline) and, for each logical /// `(root_key, body)`, counts `prior_chunk_keys(root, old) - new_keys` /// where `new_keys` is the root's OWN expansion. Never fails the dry-run: @@ -1198,7 +1233,7 @@ fn local_orphan_counts_for_dry_run( platform_name: &str, entries: &[(String, String)], ) -> Vec<(String, Result)> { - use toml_edit::{DocumentMut, Item}; + use toml_edit::DocumentMut; // Parse the current file once (best-effort). Absent file => no prior. let parsed: Result, String> = match fs::read_to_string(path) { @@ -1220,15 +1255,11 @@ fn local_orphan_counts_for_dry_run( let count = match &parsed { Err(reason) => Err(reason.clone()), Ok(None) => Ok(0), - Ok(Some(doc)) => { - let contents = doc - .get("local_server") - .and_then(|ls| ls.get("config_stores")) - .and_then(|cs| cs.get(platform_name)) - .and_then(|st| st.get("contents")) - .and_then(Item::as_table); - match contents.and_then(|tbl| tbl.get(root_key)) { - None => Ok(0), // no contents table, or no prior value for this root + Ok(Some(doc)) => match local_contents_table(doc, platform_name) { + Err(reason) => Err(reason), + Ok(None) => Ok(0), + Ok(Some(contents)) => match contents.get(root_key) { + None => Ok(0), // no prior value for this root Some(item) => match item.as_str() { None => Err("could not read prior state".to_owned()), Some(raw) => match prior_chunk_keys(root_key, raw) { @@ -1238,8 +1269,8 @@ fn local_orphan_counts_for_dry_run( Err(_) => Err("suspicious prior pointer".to_owned()), }, }, - } - } + }, + }, }; (root_key.clone(), count) }) @@ -3052,7 +3083,8 @@ build = \"cargo build --release\" /// Fake `fastly` for cloud chunk-GC tests. Logs each /// `config-store-entry` op ("describe " / "update " / - /// "delete ") to `oplog`, one per line. `root_describe_seq` + /// "delete ", plus "delete-argv ") to `oplog`, one per + /// line. `root_describe_seq` /// gives the successive raw `item_value`s returned when the ROOT key /// is described (call 1 = prior read, call 2 = read-back guard); /// exhausting the sequence yields "not found". `fail_delete_key` @@ -3078,7 +3110,7 @@ sub="$2" key="" for arg in "$@"; do case "$arg" in --key=*) key="${arg#--key=}";; esac; done if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '{{{oplog}}}'; exit 0; fi -if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi +if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; printf 'delete-argv %s\n' "$*" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi if [ "$sub" = "describe" ]; then printf 'describe %s\n' "$key" >> '{{{oplog}}}' {{#if hard_error}}echo 'Error: internal server error' >&2; exit 1{{/if}} @@ -3456,6 +3488,113 @@ echo 'unexpected' >&2; exit 1 ); } + /// Every delete must pass `--key` + `--auto-yes` and NEVER `--all` + /// (which would wipe the whole store). Asserts the actual argv. + #[cfg(unix)] + #[test] + fn push_config_entries_delete_uses_key_and_auto_yes_never_all() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "k".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:06Z".to_owned())) + .expect("B") + }; + let (_a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, false, &oplog); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + let argv_lines: Vec<&str> = log + .lines() + .filter(|line| line.starts_with("delete-argv ")) + .collect(); + assert!( + !argv_lines.is_empty(), + "at least one delete happened: {log}" + ); + for line in argv_lines { + assert!( + line.contains("--auto-yes"), + "delete must pass --auto-yes: {line}" + ); + assert!( + line.contains("--key="), + "delete must target a --key: {line}" + ); + assert!( + !line.contains("--all"), + "delete must NEVER pass --all (store-wide wipe): {line}" + ); + } + } + + /// Cloud config shrinking back under the limit: the new value is a + /// direct envelope, so every prior chunk is reclaimed and the root is + /// upserted (never deleted). + #[cfg(unix)] + #[test] + fn push_config_entries_shrink_to_direct_deletes_all_prior_chunks() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let chunked = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let (prior_chunks, prior_pointer) = chunked_parts(TEST_CONFIG_ID, &chunked); + // New value is direct, so new_root_value == the direct envelope; + // read-back returns it so the guard passes. + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[prior_pointer, direct.clone()], + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), direct)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + for key in &prior_chunks { + assert!( + oplog_has(&oplog, &format!("delete {key}")), + "prior chunk `{key}` must be deleted on shrink-to-direct" + ); + } + assert!( + !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), + "root pointer must never be deleted" + ); + assert!( + oplog_has(&oplog, &format!("update {TEST_CONFIG_ID}")), + "root must be upserted with the direct value" + ); + } + #[cfg(unix)] #[test] fn push_config_entries_writes_direct_entry_at_exactly_8000_chars() { @@ -4427,6 +4566,43 @@ echo 'unexpected' >&2; exit 1 ); } + /// A present-but-malformed `contents` (non-table) is prior state the + /// real writer would reject — the dry-run count must degrade to + /// `unknown: could not read prior state`, not silently report 0. + #[cfg(unix)] + #[test] + fn push_config_entries_local_dry_run_non_table_contents_unknown() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + let seed = concat!( + "name = \"demo\"\n\n", + "[local_server.config_stores.app_config]\n", + "format = \"inline-toml\"\n", + "contents = \"bad\"\n", + ); + fs::write(&fastly_toml, seed).expect("seed"); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + true, // dry_run + ) + .expect("dry-run must not fail on malformed contents"); + + assert!( + out.join("\n") + .contains("unknown: could not read prior state"), + "non-table contents must degrade to unknown: {out:?}" + ); + } + /// GC of a chunked root must not touch a chunked SIBLING's chunks — /// the prefix `app_config.__edgezero_chunks.` must not match /// `app_config_staging.__edgezero_chunks.` (shared string prefix). From 6f65e514560e6cff64ed592e6108855fd83d4c90 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Sun, 12 Jul 2026 14:23:52 -0700 Subject: [PATCH 18/25] fix(fastly): redact describe payloads from diagnostics (P1); reject duplicate root keys (P2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P1 (security): fetch/read error paths interpolated the raw `describe` stdout — i.e. the stored config envelope, which may hold credentials — into errors the CLI logs verbatim. The GC prior-read runs on EVERY cloud push, so this exposed the previous config on any schema drift. Now report only size + top-level field-name shape via redact_describe_response; adds sentinel-secret tests for both the read and push paths. P2 (correctness): a duplicate logical root in one batch built two GC plans against the same prior; the earlier plan reclaimed the chunks the LAST tuple had just installed, leaving the final pointer dangling. Reject duplicate root keys in both push paths, before any expansion or I/O. --- crates/edgezero-adapter-fastly/src/cli.rs | 229 ++++++++++++++++++++-- 1 file changed, 218 insertions(+), 11 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 4c0cf531..11012a32 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -392,6 +392,7 @@ impl Adapter for FastlyCliAdapter { } // Reject reserved keys before any expansion or I/O. reject_reserved_root_keys(entries)?; + reject_duplicate_root_keys(entries)?; // Expand each logical root once: flatten for the commit, and keep // the exact per-root keep-set + the value written at the root key // for GC (no prefix scan of the flattened set). Collecting all @@ -533,6 +534,7 @@ impl Adapter for FastlyCliAdapter { } // Reject reserved keys before any expansion or I/O. reject_reserved_root_keys(entries)?; + reject_duplicate_root_keys(entries)?; // Expand each logical root once: flatten for the write, keep the // exact per-root keep-set for GC (no prefix scan of the flattened set). let mut physical_entries: Vec<(String, String)> = Vec::new(); @@ -637,19 +639,20 @@ impl Adapter for FastlyCliAdapter { if output.status.success() { let stdout = String::from_utf8_lossy(&output.stdout); // Parse the JSON and extract the `item_value` field. - let parsed: serde_json::Value = - serde_json::from_str(&stdout).map_err(|err| { - format!( - "failed to parse `fastly config-store-entry describe` JSON: {err}\nraw stdout: {stdout}" - ) - })?; + let parsed: serde_json::Value = serde_json::from_str(&stdout).map_err(|err| { + format!( + "failed to parse `fastly config-store-entry describe` JSON: {err} (response: {})", + redact_describe_response(&stdout) + ) + })?; let value = parsed .get("item_value") .and_then(serde_json::Value::as_str) .ok_or_else(|| { format!( "`fastly config-store-entry describe` JSON has no string `item_value` field; \ - fastly CLI may have changed its output schema. Raw stdout: {stdout}" + fastly CLI may have changed its output schema. (response: {})", + redact_describe_response(&stdout) ) })?; // Resolve chunk pointers: if `value` is a direct BlobEnvelope it @@ -790,8 +793,9 @@ fn fetch_remote_config_store_entry(store_id: &str, key: &str) -> Result Result Result<(), String> Ok(()) } +/// Reject a batch that names the same logical root key more than once. +/// +/// The adapter trait takes an entry slice and does not enforce uniqueness, +/// but GC builds one plan per entry and snapshots every plan against the +/// SAME prior generation. With `[(root, A), (root, B)]` the last tuple wins +/// the upsert (root = B), yet A's plan would still reclaim `prior - A_keys` +/// — which includes B's freshly-written chunks — leaving the final pointer +/// referencing missing chunks. Rejecting is safer than silently coalescing: +/// a duplicated key is a caller bug, and picking a winner would hide it. +fn reject_duplicate_root_keys(entries: &[(String, String)]) -> Result<(), String> { + let mut seen: HashSet<&str> = HashSet::with_capacity(entries.len()); + for (key, _) in entries { + if !seen.insert(key.as_str()) { + return Err(format!( + "config key `{key}` appears more than once in a single push; each logical key must be pushed exactly once" + )); + } + } + Ok(()) +} + /// Best-effort per-root orphan count for `config push --local --dry-run`. /// Navigate to `[local_server.config_stores..contents]` for the /// dry-run counter. `Ok(None)` when any level is absent (no prior state); @@ -1482,6 +1508,40 @@ fn find_config_store_id(stdout: &str, name: &str) -> ConfigStoreLookup { } } +/// Summarise a `fastly ... describe` response for diagnostics WITHOUT +/// leaking its contents. +/// +/// The response body is the stored config value. App config may hold +/// credentials, internal endpoints, or security policy, and this adapter +/// performs no secret stripping — while CLI status lines are logged +/// verbatim and CI logs are commonly retained and shared. So a schema-drift +/// diagnostic must never echo the payload: report only its size and its +/// top-level *shape* (field names for an object, type otherwise), never a +/// value. +fn redact_describe_response(stdout: &str) -> String { + let len = stdout.len(); + serde_json::from_str::(stdout).map_or_else( + |_err| format!("{len} bytes, not valid JSON"), + |value| match value { + serde_json::Value::Object(map) => { + let mut names: Vec<&str> = map.keys().map(String::as_str).collect(); + names.sort_unstable(); + format!( + "{len} bytes, JSON object with fields [{}]", + names.join(", ") + ) + } + other @ (serde_json::Value::Null + | serde_json::Value::Bool(_) + | serde_json::Value::Number(_) + | serde_json::Value::String(_) + | serde_json::Value::Array(_)) => { + format!("{len} bytes, JSON {}", shape_summary(&other)) + } + }, + ) +} + /// One-line type label for a `serde_json::Value` (for diagnostic /// error messages — not a canonical JSON-schema description). fn shape_summary(value: &serde_json::Value) -> &'static str { @@ -3488,6 +3548,74 @@ echo 'unexpected' >&2; exit 1 ); } + /// Schema drift must never echo the config payload. App config can hold + /// credentials; CLI status lines are logged verbatim and CI logs are + /// retained/shared. Only a size + field-name shape may be reported. + #[cfg(unix)] + #[test] + fn read_config_entry_schema_drift_does_not_leak_payload() { + const SENTINEL: &str = "SUPER_SECRET_TOKEN_abc123"; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + // Valid JSON, but the value moved out of `item_value` (schema drift). + let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); + let fake = fake_fastly_returning(&drift, "", 0); + let _path = PathPrepend::new(fake.path()); + + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + "cfg", + &AdapterPushContext::new(), + ); + let Err(err) = result else { + panic!("schema drift must error") + }; + assert!( + !err.contains(SENTINEL), + "error must not leak the config payload: {err}" + ); + assert!( + err.contains("bytes"), + "error should carry a redacted size/shape summary: {err}" + ); + } + + /// The GC prior-read runs on EVERY cloud push, so a drifted response must + /// not leak the previous config envelope into the push's status lines. + #[cfg(unix)] + #[test] + fn push_config_entries_prior_read_drift_does_not_leak_payload() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + const SENTINEL: &str = "SUPER_SECRET_TOKEN_xyz789"; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); + let fake = fake_fastly_returning(&drift, "", 0); + let _path = PathPrepend::new(fake.path()); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + false, + ) + .expect("push still succeeds; the prior read only warns"); + for line in &out { + assert!( + !line.contains(SENTINEL), + "push status must not leak the prior config payload: {line}" + ); + } + } + /// Every delete must pass `--key` + `--auto-yes` and NEVER `--all` /// (which would wipe the whole store). Asserts the actual argv. #[cfg(unix)] @@ -4603,6 +4731,85 @@ echo 'unexpected' >&2; exit 1 ); } + /// A duplicate root key in one batch is rejected before any I/O. + /// Otherwise the earlier tuple's GC plan would reclaim the chunks the + /// LAST tuple just installed, leaving the final pointer dangling. + /// Regression: prior B, batch `[(root, A), (root, B)]` — the root must + /// still resolve afterwards. + #[cfg(unix)] + #[test] + fn push_config_entries_local_rejects_duplicate_root_keys() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + + let make = |tag: &str| { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:00Z".to_owned())) + .expect("envelope") + }; + let envelope_a = make("aaa"); + let envelope_b = make("bbb"); + + // Prior generation B is live. + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b.clone())], + &AdapterPushContext::new(), + false, + ) + .expect("seed push"); + let before = fs::read_to_string(&fastly_toml).expect("read"); + + // Duplicate-root batch must be rejected outright. + let err = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[ + (TEST_CONFIG_ID.to_owned(), envelope_a), + (TEST_CONFIG_ID.to_owned(), envelope_b.clone()), + ], + &AdapterPushContext::new(), + false, + ) + .expect_err("duplicate root keys must be rejected"); + assert!( + err.contains("more than once"), + "error explains the duplicate: {err}" + ); + assert_eq!( + fs::read_to_string(&fastly_toml).expect("read"), + before, + "rejection must happen before any write" + ); + + // The live root still resolves to B (nothing was reclaimed). + let read = FastlyCliAdapter + .read_config_entry_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + TEST_CONFIG_ID, + &AdapterPushContext::new(), + ) + .expect("root must still resolve"); + let ReadConfigEntry::Present(value) = read else { + panic!("expected Present"); + }; + assert_eq!(value, envelope_b, "root still reconstructs envelope B"); + } + /// GC of a chunked root must not touch a chunked SIBLING's chunks — /// the prefix `app_config.__edgezero_chunks.` must not match /// `app_config_staging.__edgezero_chunks.` (shared string prefix). From 33932def25ba88547123c22f38eab1f1f7d6285e Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:42:06 -0700 Subject: [PATCH 19/25] fix(fastly)!: defer cloud chunk reclamation (eager delete is unsafe under eventual consistency) Fastly Config Store is eventually consistent: after the control plane accepts pointer N, POPs may still serve pointer N-1 and need N-1's chunks. The post-commit read-back only observes the CONTROL PLANE, so eager deletion could strip chunks from a POP mid-propagation -- breaking readers on every re-push, not merely under concurrent writers. (Reported on PR #314.) Cloud GC is now DEFERRED: - a push never deletes what it just superseded; it RECORDS that generation in a per-root pending record at .__edgezero_gc.pending - a later push reclaims a recorded generation only once it has aged past a grace window (EDGEZERO_FASTLY_GC_GRACE_SECS, default 24h) AND nothing live references it (neither the new generation nor the just-superseded one) - the record stores (sha, count, superseded_at) per generation, not raw keys: keys are derivable, and the raw list would blow the 8 000-char entry limit - it is a LIST, so pushing faster than the grace window cannot drop an un-reclaimed generation and leak it untracked - failed deletes keep the generation pending so a later push retries - the LWW read-back guard still gates all GC state changes The LOCAL path keeps eager pruning: fastly.toml is a single file Viceroy reads at startup -- no propagation window, no POPs. Extracts reclaim_and_record; adds pending-record round-trip and derive_chunk_keys tests, plus reclaim-after-grace / retain-within-grace / defer-instead-of-delete cloud tests. --- .../src/chunked_config.rs | 215 +++++++ crates/edgezero-adapter-fastly/src/cli.rs | 602 +++++++++++++++--- 2 files changed, 715 insertions(+), 102 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/chunked_config.rs b/crates/edgezero-adapter-fastly/src/chunked_config.rs index fbe610fc..3af156ca 100644 --- a/crates/edgezero-adapter-fastly/src/chunked_config.rs +++ b/crates/edgezero-adapter-fastly/src/chunked_config.rs @@ -39,6 +39,14 @@ pub(crate) const CHUNK_KEY_INFIX: &str = ".__edgezero_chunks."; /// BOTH the writer (when serialising the pointer) AND the resolver /// (when validating the parsed pointer) -- stays unconditional. pub(crate) const POINTER_KIND: &str = "fastly_config_chunks"; +/// Infix of the per-root deferred-reclamation record: +/// `.__edgezero_gc.pending`. Reserved like `CHUNK_KEY_INFIX` -- +/// a logical config key must never contain it. CLI writer only. +#[cfg(any(feature = "cli", test))] +pub(crate) const GC_PENDING_INFIX: &str = ".__edgezero_gc."; +/// `edgezero_kind` discriminant stored in the pending-reclamation record. +#[cfg(any(feature = "cli", test))] +pub(crate) const GC_PENDING_KIND: &str = "fastly_config_gc_pending"; // --------------------------------------------------------------------------- // Private pointer schema @@ -61,6 +69,43 @@ struct FastlyChunkRef { sha256: String, } +/// One chunk generation, identified by the envelope SHA its chunk keys are +/// addressed by. The keys themselves are DERIVABLE +/// (`.__edgezero_chunks..` for `idx` in `0..count`), so the +/// deferred-reclamation record stores this compact form rather than the raw +/// key list -- which would blow the 8 000-char entry limit after a couple of +/// generations. +#[cfg(any(feature = "cli", test))] +#[derive(Clone, PartialEq, Eq, Debug)] +pub(crate) struct ChunkGeneration { + pub count: usize, + pub sha: String, +} + +/// A generation awaiting reclamation, plus when it was superseded. +#[cfg(any(feature = "cli", test))] +#[derive(Clone, Debug, serde::Deserialize, serde::Serialize)] +pub(crate) struct GcPendingGeneration { + pub count: usize, + pub sha: String, + /// Unix epoch seconds at which this generation stopped being the live + /// pointer. Reclamation waits until it has aged past the grace window, + /// so lagging POPs still serving the old pointer keep their chunks. + pub superseded_at: u64, +} + +/// The per-root deferred-reclamation record stored at +/// `.__edgezero_gc.pending`. A LIST, not a single slot: pushing faster +/// than the grace window must not drop an un-reclaimed generation on the +/// floor (that would leak it untracked, forever). +#[cfg(any(feature = "cli", test))] +#[derive(serde::Deserialize, serde::Serialize)] +pub(crate) struct GcPendingRecord { + edgezero_kind: String, + generations: Vec, + version: u8, +} + // --------------------------------------------------------------------------- // Public helpers // --------------------------------------------------------------------------- @@ -362,6 +407,100 @@ pub(crate) fn prior_chunk_keys(root_key: &str, raw: &str) -> Result, Ok(keys) } +/// The prior pointer's generation (envelope SHA + chunk count), from which +/// its chunk keys are derivable. Same validation and prefix-scoping as +/// [`prior_chunk_keys`]: `Ok(None)` for a direct/absent/non-pointer value, +/// `Err` for a pointer-kind value that fails validation. +#[cfg(any(feature = "cli", test))] +pub(crate) fn prior_chunk_generation( + root_key: &str, + raw: &str, +) -> Result, String> { + let keys = prior_chunk_keys(root_key, raw)?; + if keys.is_empty() { + return Ok(None); + } + // The pointer validated above; re-read its sha for the compact form. + let value: serde_json::Value = serde_json::from_str(raw) + .map_err(|err| format!("prior chunk pointer at `{root_key}` is malformed: {err}"))?; + let sha = value + .get("envelope_sha256") + .and_then(serde_json::Value::as_str) + .ok_or_else(|| { + format!( + "prior chunk pointer at `{root_key}` has no `envelope_sha256`; skipping chunk GC" + ) + })?; + Ok(Some(ChunkGeneration { + count: keys.len(), + sha: sha.to_owned(), + })) +} + +/// Rebuild a generation's chunk keys from its content address. Mirrors the +/// key construction in [`prepare_fastly_config_entries`]. +#[cfg(any(feature = "cli", test))] +pub(crate) fn derive_chunk_keys(root_key: &str, generation: &ChunkGeneration) -> Vec { + (0..generation.count) + .map(|idx| { + format!( + "{root_key}{CHUNK_KEY_INFIX}{sha}.{idx}", + sha = generation.sha + ) + }) + .collect() +} + +/// Key of the per-root deferred-reclamation record. +#[cfg(any(feature = "cli", test))] +pub(crate) fn gc_pending_key(root_key: &str) -> String { + format!("{root_key}{GC_PENDING_INFIX}pending") +} + +/// Parse a pending-reclamation record. `Ok(vec![])` when absent or not our +/// record; `Err` when it IS our kind but fails validation (caller warns and +/// skips reclamation rather than clobbering state it cannot understand). +#[cfg(any(feature = "cli", test))] +pub(crate) fn parse_gc_pending( + root_key: &str, + raw: &str, +) -> Result, String> { + let value: serde_json::Value = match serde_json::from_str(raw) { + Ok(value) => value, + Err(_) => return Ok(Vec::new()), + }; + if value + .get("edgezero_kind") + .and_then(serde_json::Value::as_str) + != Some(GC_PENDING_KIND) + { + return Ok(Vec::new()); + } + let record: GcPendingRecord = serde_json::from_value(value).map_err(|err| { + format!("GC pending record at `{root_key}` is malformed: {err}; skipping chunk reclamation") + })?; + if record.version != 1 { + return Err(format!( + "GC pending record at `{root_key}` has unsupported version {}; skipping chunk reclamation", + record.version + )); + } + Ok(record.generations) +} + +/// Serialise a pending-reclamation record. +#[cfg(any(feature = "cli", test))] +pub(crate) fn serialize_gc_pending( + generations: Vec, +) -> Result { + serde_json::to_string(&GcPendingRecord { + edgezero_kind: GC_PENDING_KIND.to_owned(), + generations, + version: 1, + }) + .map_err(|err| format!("failed to serialise GC pending record: {err}")) +} + // --------------------------------------------------------------------------- // Unit tests // --------------------------------------------------------------------------- @@ -790,6 +929,82 @@ mod tests { assert!(err.contains("outside"), "{err}"); } + // ---- deferred-reclamation tests ---- + + /// The compact generation form must regenerate EXACTLY the pointer's keys + /// — the pending record stores `(sha, count)` instead of the raw key list + /// (which would blow the 8 000-char entry limit), so reclamation depends + /// on this round-trip being lossless. + #[test] + fn derive_chunk_keys_round_trips_the_pointers_keys() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); + let (_, pointer_json) = entries.last().unwrap(); + let expected: Vec = entries[..entries.len().saturating_sub(1)] + .iter() + .map(|(key, _)| key.clone()) + .collect(); + + let generation = prior_chunk_generation("app_config", pointer_json) + .expect("valid pointer") + .expect("chunked"); + assert_eq!(generation.count, expected.len()); + assert_eq!(derive_chunk_keys("app_config", &generation), expected); + } + + #[test] + fn prior_chunk_generation_is_none_for_direct_envelope() { + let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT); + assert!(prior_chunk_generation("app_config", &envelope) + .unwrap() + .is_none()); + } + + #[test] + fn gc_pending_round_trips() { + let generations = vec![ + GcPendingGeneration { + count: 3, + sha: "aa".repeat(32), + superseded_at: 1_720_000_000, + }, + GcPendingGeneration { + count: 1, + sha: "bb".repeat(32), + superseded_at: 1_720_000_500, + }, + ]; + let raw = serialize_gc_pending(generations).expect("serialise"); + let parsed = parse_gc_pending("app_config", &raw).expect("parse"); + assert_eq!(parsed.len(), 2); + assert_eq!(parsed[0].count, 3); + assert_eq!(parsed[1].superseded_at, 1_720_000_500); + } + + #[test] + fn parse_gc_pending_is_silent_for_unrelated_values() { + assert!(parse_gc_pending("app_config", r#"{"hello":"world"}"#) + .unwrap() + .is_empty()); + assert!(parse_gc_pending("app_config", "not json") + .unwrap() + .is_empty()); + } + + #[test] + fn parse_gc_pending_warns_on_unsupported_version() { + let raw = r#"{"edgezero_kind":"fastly_config_gc_pending","version":2,"generations":[]}"#; + let err = parse_gc_pending("app_config", raw).expect_err("version 2 must warn"); + assert!(err.contains("unsupported version"), "{err}"); + } + + #[test] + fn gc_pending_key_is_in_the_reserved_namespace() { + let key = gc_pending_key("app_config"); + assert!(key.contains(GC_PENDING_INFIX), "{key}"); + assert!(key.starts_with("app_config"), "{key}"); + } + // Regression for the Value-first rule: a value that IS pointer-kind but // is missing required fields (`chunks`, `data_sha256`, …) must WARN, not // be silently dropped by a failed struct deserialize. diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 11012a32..cfeda033 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -5,9 +5,12 @@ use std::io::{ErrorKind, Write as _}; use std::path::{Path, PathBuf}; use std::process::Command; use std::process::Stdio; +use std::time::{SystemTime, UNIX_EPOCH}; use crate::chunked_config::{ - prepare_fastly_config_entries, prior_chunk_keys, resolve_fastly_config_value, CHUNK_KEY_INFIX, + derive_chunk_keys, gc_pending_key, parse_gc_pending, prepare_fastly_config_entries, + prior_chunk_generation, prior_chunk_keys, resolve_fastly_config_value, serialize_gc_pending, + ChunkGeneration, GcPendingGeneration, CHUNK_KEY_INFIX, GC_PENDING_INFIX, }; use ctor::ctor; use edgezero_adapter::cli_support::{ @@ -147,20 +150,39 @@ enum ConfigStoreLookup { SchemaDrift(String), } -/// Per-root plan for reclaiming a prior generation's chunk entries on -/// re-push (Stage 7). Built by the config-push paths; consumed by the -/// GC helpers `expand_root` / `orphan_chunk_keys`. -struct FastlyConfigGcPlan { +/// Per-root plan for the CLOUD path's deferred reclamation. +/// +/// Fastly's config store is eventually consistent: after we write pointer N, +/// some POPs still serve pointer N-1 and need N-1's chunks. So a cloud push +/// never deletes what it just superseded — it *records* that generation and +/// reclaims it on a later push, once it has aged past the grace window. +struct FastlyCloudGcPlan { /// Exact keep-set this push writes for the root (chunk keys + root key). new_keys: HashSet, - /// Exactly the value written at `root_key` (its new pointer, or the - /// direct envelope). Cloud uses it as a read-back concurrency guard - /// (last-writer-wins); local ignores it. + /// Exactly the value written at `root_key`; the read-back guard compares + /// against it to confirm we are still the last writer (LWW). new_root_value: String, + /// Generations already awaiting reclamation, read before the commit. + /// `Err` => record we cannot understand; warn and touch nothing. + pending: Result, String>, + /// The generation this push supersedes (N-1) — recorded as pending, never + /// deleted now. `Err` => suspicious prior pointer; warn and skip. + prior: Result, String>, + root_key: String, +} + +/// Per-root plan for the LOCAL path's eager prune. +/// +/// Local reclamation is safe to do immediately: `fastly.toml` is a single +/// file that Viceroy reads at startup — there is no propagation window and no +/// POP that could still be serving the previous pointer. (The cloud path +/// cannot do this; see [`FastlyCloudGcPlan`].) +struct FastlyConfigGcPlan { + /// Exact keep-set this push writes for the root (chunk keys + root key). + new_keys: HashSet, /// Prior chunk keys to consider deleting, or a warning to surface /// (suspicious prior pointer) that skips GC for this root. prior_keys: Result, String>, - root_key: String, } // The three `validate_*` trait methods exist on `Adapter` because @@ -428,27 +450,39 @@ impl Adapter for FastlyCliAdapter { )); } out.push(format!( - " would delete orphaned prior-generation chunks of `{key}` (count determined at push time)" + " would defer reclamation of `{key}`'s prior generation (reclaimed on a later push, once aged past the grace window)" )); } return Ok(out); } let resolved_id = resolve_remote_config_store_id(name)?; - // Build GC plans BEFORE the commit: read each root's prior value so - // we know which chunks the old pointer referenced. - let mut gc_plans: Vec = Vec::with_capacity(roots.len()); + let now = unix_now_secs(); + let grace = gc_grace_secs(); + // Build GC plans BEFORE the commit: read each root's prior pointer (the + // generation we are about to supersede) and its pending-reclamation + // record (generations superseded by EARLIER pushes). + let mut gc_plans: Vec = Vec::with_capacity(roots.len()); for (root_key, new_keys, new_root_value) in roots { - let prior_keys = match fetch_remote_config_store_entry(&resolved_id, &root_key) { - Ok(Some(raw)) => prior_chunk_keys(&root_key, &raw), - Ok(None) => Ok(Vec::new()), + let prior = match fetch_remote_config_store_entry(&resolved_id, &root_key) { + Ok(Some(raw)) => prior_chunk_generation(&root_key, &raw), + Ok(None) => Ok(None), Err(err) => Err(format!( "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" )), }; - gc_plans.push(FastlyConfigGcPlan { + let pending = + match fetch_remote_config_store_entry(&resolved_id, &gc_pending_key(&root_key)) { + Ok(Some(raw)) => parse_gc_pending(&root_key, &raw), + Ok(None) => Ok(Vec::new()), + Err(err) => Err(format!( + "failed to read the GC pending record for `{root_key}`: {err}; skipping chunk reclamation" + )), + }; + gc_plans.push(FastlyCloudGcPlan { new_keys, new_root_value, - prior_keys, + pending, + prior, root_key, }); } @@ -457,39 +491,13 @@ impl Adapter for FastlyCliAdapter { push_entries_with_committer(&physical_entries, |key, value| { create_config_store_entry(&resolved_id, key, value) })?; - // Post-commit sweep, last-writer-wins: delete a root's orphans only - // while a read-back confirms the root still holds exactly what we - // wrote. If a newer push has superseded it, yield (delete nothing). + // Post-commit: DEFERRED reclamation. We never delete the generation we + // just superseded — the config store is eventually consistent, so POPs + // may still be serving it. Instead we record it, and reclaim only older + // generations that have aged past the grace window. let mut warnings = Vec::new(); - for plan in &gc_plans { - let orphans = match orphan_chunk_keys(plan) { - Ok(keys) if !keys.is_empty() => keys, - Ok(_) => continue, - Err(err) => { - warnings.push(format!("warning: {err}")); - continue; - } - }; - match fetch_remote_config_store_entry(&resolved_id, &plan.root_key) { - Ok(Some(current)) if current == plan.new_root_value => { - for key in orphans { - if let Err(err) = delete_config_store_entry(&resolved_id, &key) { - warnings.push(format!( - "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be removed by a future `config gc`", - plan.root_key - )); - } - } - } - Ok(_) => warnings.push(format!( - "note: skipped chunk GC for `{}`: root changed since this push wrote it (concurrent push?); orphans left for a future `config gc`", - plan.root_key - )), - Err(err) => warnings.push(format!( - "note: skipped chunk GC for `{}`: could not re-read root before sweep ({err}); orphans left for a future `config gc`", - plan.root_key - )), - } + for plan in gc_plans { + warnings.extend(reclaim_and_record(&resolved_id, plan, now, grace)); } let mut out = vec![format!( "pushed {} physical entries ({} logical) to fastly config-store `{name}` (logical id `{logical}`, id={resolved_id})", @@ -1107,9 +1115,7 @@ fn write_fastly_local_config_store( .map_or_else(|| Ok(Vec::new()), |value| prior_chunk_keys(root_key, value)); plans.push(FastlyConfigGcPlan { new_keys: new_keys.clone(), - new_root_value: String::new(), // unused locally (no remote concurrency) prior_keys, - root_key: root_key.clone(), }); } @@ -1182,15 +1188,41 @@ fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { /// space, and could not be reclaimed correctly. fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> { for (key, _) in entries { - if key.contains(CHUNK_KEY_INFIX) { - return Err(format!( - "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" - )); + for infix in [CHUNK_KEY_INFIX, GC_PENDING_INFIX] { + if key.contains(infix) { + return Err(format!( + "config key `{key}` contains the reserved infix `{infix}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" + )); + } } } Ok(()) } +/// Unix epoch seconds. Push-time only (the `cli` feature is native). +fn unix_now_secs() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_secs()) +} + +/// How long a superseded generation must age before it may be reclaimed. +/// +/// Fastly's config store is eventually consistent and documents no +/// propagation bound, so eager deletion can strip chunks from a POP that is +/// still serving the previous pointer. We therefore retain generation N-1 +/// unconditionally and only reclaim older generations once they have aged +/// past this window. Override with `EDGEZERO_FASTLY_GC_GRACE_SECS` +/// (e.g. `0` in tests, or a smaller value for a fast deploy cadence). +fn gc_grace_secs() -> u64 { + /// 24h — generous against any realistic propagation lag. + const DEFAULT_GRACE_SECS: u64 = 86_400; + env::var("EDGEZERO_FASTLY_GC_GRACE_SECS") + .ok() + .and_then(|raw| raw.parse::().ok()) + .unwrap_or(DEFAULT_GRACE_SECS) +} + /// Reject a batch that names the same logical root key more than once. /// /// The adapter trait takes an entry slice and does not enforce uniqueness, @@ -1313,6 +1345,117 @@ fn local_orphan_counts_for_dry_run( /// exists" error, which is the operator's signal to delete the /// entry (or use `config-store-entry update` manually) before /// re-running push. +/// Post-commit deferred reclamation for ONE root. +/// +/// Never deletes the generation this push just superseded: Fastly's config +/// store is eventually consistent, so POPs may still be serving the previous +/// pointer and need its chunks. Instead the superseded generation is RECORDED, +/// and only generations that have aged past the grace window (and that nothing +/// live references) are reclaimed. Returns status/warning lines; a failure here +/// never fails the push. +fn reclaim_and_record( + store_id: &str, + plan: FastlyCloudGcPlan, + now: u64, + grace: u64, +) -> Vec { + let mut warnings = Vec::new(); + // Last-writer-wins: only touch GC state while the root still holds exactly + // what this push wrote. + match fetch_remote_config_store_entry(store_id, &plan.root_key) { + Ok(Some(current)) if current == plan.new_root_value => {} + Ok(_) => { + warnings.push(format!( + "note: skipped chunk GC for `{}`: root changed since this push wrote it (concurrent push?); reclamation deferred to a later push", + plan.root_key + )); + return warnings; + } + Err(err) => { + warnings.push(format!( + "note: skipped chunk GC for `{}`: could not re-read the root ({err}); reclamation deferred to a later push", + plan.root_key + )); + return warnings; + } + } + let prior = match plan.prior { + Ok(prior) => prior, + Err(err) => { + warnings.push(format!("warning: {err}")); + return warnings; + } + }; + let pending = match plan.pending { + Ok(pending) => pending, + Err(err) => { + warnings.push(format!("warning: {err}")); + return warnings; + } + }; + // Keys that must NOT be reclaimed: the generation we just wrote, and the one + // we just superseded (still live at lagging POPs). + let mut protected: HashSet = plan.new_keys.clone(); + if let Some(generation) = &prior { + protected.extend(derive_chunk_keys(&plan.root_key, generation)); + } + + let mut retained: Vec = Vec::with_capacity(pending.len()); + for generation in pending { + let candidate = ChunkGeneration { + count: generation.count, + sha: generation.sha.clone(), + }; + let keys = derive_chunk_keys(&plan.root_key, &candidate); + let aged = now.saturating_sub(generation.superseded_at) >= grace; + let live = keys.iter().any(|key| protected.contains(key)); + if !aged || live { + retained.push(generation); // not yet safe to reclaim + continue; + } + // Aged out and unreferenced: reclaim. A failed delete keeps the + // generation pending so a later push retries (deletes are idempotent -- + // a not-found is treated as success). + let mut all_deleted = true; + for key in keys { + if let Err(err) = delete_config_store_entry(store_id, &key) { + warnings.push(format!( + "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be retried on a later push", + plan.root_key + )); + all_deleted = false; + } + } + if !all_deleted { + retained.push(generation); + } + } + // Record the generation this push superseded, so a LATER push can reclaim it + // once it has aged out. + if let Some(generation) = prior { + retained.retain(|held| held.sha != generation.sha); + retained.push(GcPendingGeneration { + count: generation.count, + sha: generation.sha, + superseded_at: now, + }); + } + match serialize_gc_pending(retained) { + Ok(raw) => { + if let Err(err) = + create_config_store_entry(store_id, &gc_pending_key(&plan.root_key), &raw) + { + warnings.push(format!( + "note: could not update the GC pending record for `{}` ({err}); reclamation will retry on a later push", + plan.root_key + )); + } + } + Err(err) => warnings.push(format!("warning: {err}")), + } + warnings +} + /// Drive a sequential per-entry commit loop and produce the /// partial-failure diagnostic when the committer fails mid-way. /// Pure (no I/O) so the diagnostic shape is unit-testable without @@ -1818,6 +1961,32 @@ mod tests { } } + /// RAII guard for the GC grace-window override. Callers hold + /// `path_mutation_guard()` too, which serialises env mutation. + #[cfg(unix)] + struct GraceGuard { + original: Option, + } + + #[cfg(unix)] + impl GraceGuard { + fn set(secs: &str) -> Self { + let original = env::var("EDGEZERO_FASTLY_GC_GRACE_SECS").ok(); + env::set_var("EDGEZERO_FASTLY_GC_GRACE_SECS", secs); + Self { original } + } + } + + #[cfg(unix)] + impl Drop for GraceGuard { + fn drop(&mut self) { + match self.original.take() { + Some(prev) => env::set_var("EDGEZERO_FASTLY_GC_GRACE_SECS", prev), + None => env::remove_var("EDGEZERO_FASTLY_GC_GRACE_SECS"), + } + } + } + #[test] fn finds_closest_manifest_when_multiple_exist() { let dir = tempdir().unwrap(); @@ -3148,10 +3317,10 @@ build = \"cargo build --release\" /// gives the successive raw `item_value`s returned when the ROOT key /// is described (call 1 = prior read, call 2 = read-back guard); /// exhausting the sequence yields "not found". `fail_delete_key` - /// makes that one delete exit non-zero. `describe_hard_error` makes - /// every describe fail with a non-not-found error (so - /// `fetch_remote_config_store_entry` returns `Err`, exercising the - /// prior-read-failure GC path). + /// makes that one delete exit non-zero. `describe_hard_error` makes the + /// FIRST describe of each key fail with a non-not-found error (so the + /// pre-commit prior read returns `Err` while the post-commit read-back + /// still succeeds — exercising the prior-read-failure GC path). #[cfg(unix)] fn fake_fastly_gc( root_key: &str, @@ -3159,6 +3328,27 @@ build = \"cargo build --release\" fail_delete_key: Option<&str>, describe_hard_error: bool, oplog: &Path, + ) -> tempfile::TempDir { + fake_fastly_gc_with_pending( + root_key, + root_describe_seq, + None, + fail_delete_key, + describe_hard_error, + oplog, + ) + } + + /// As [`fake_fastly_gc`], plus an optional seeded pending-reclamation + /// record served when the `.__edgezero_gc.pending` key is described. + #[cfg(unix)] + fn fake_fastly_gc_with_pending( + root_key: &str, + root_describe_seq: &[String], + pending_seed: Option<&str>, + fail_delete_key: Option<&str>, + describe_hard_error: bool, + oplog: &Path, ) -> tempfile::TempDir { use std::os::unix::fs::PermissionsExt as _; // Rendered with handlebars. Triple-stache `{{{ }}}` disables HTML @@ -3173,9 +3363,9 @@ if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '{ if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; printf 'delete-argv %s\n' "$*" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi if [ "$sub" = "describe" ]; then printf 'describe %s\n' "$key" >> '{{{oplog}}}' - {{#if hard_error}}echo 'Error: internal server error' >&2; exit 1{{/if}} cfile='{{{dir}}}/count_'"$key" n=0; [ -f "$cfile" ] && n=$(cat "$cfile"); n=$((n+1)); printf '%s' "$n" > "$cfile" + {{#if hard_error}}if [ "$n" = "1" ]; then echo 'Error: internal server error' >&2; exit 1; fi{{/if}} rf='{{{dir}}}/resp_'"$key"'_'"$n"'.json' if [ -f "$rf" ]; then cat "$rf"; exit 0; fi echo 'Error: item not found' >&2; exit 1 @@ -3201,6 +3391,14 @@ echo 'unexpected' >&2; exit 1 ) .expect("resp"); } + if let Some(record) = pending_seed { + let key = gc_pending_key(root_key); + let wrapped = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(record).expect("escape") + ); + fs::write(dir.path().join(format!("resp_{key}_1.json")), wrapped).expect("pending"); + } let data = serde_json::json!({ "list": list_file.display().to_string(), "oplog": oplog.display().to_string(), @@ -3219,6 +3417,26 @@ echo 'unexpected' >&2; exit 1 dir } + /// The chunk generation (sha + count) a chunked envelope produces. + #[cfg(unix)] + fn generation_of(root_key: &str, envelope: &str) -> ChunkGeneration { + let (_, pointer) = chunked_parts(root_key, envelope); + prior_chunk_generation(root_key, &pointer) + .expect("valid pointer") + .expect("chunked envelope has a generation") + } + + /// A seeded pending-reclamation record holding one generation. + #[cfg(unix)] + fn pending_record_json(generation: &ChunkGeneration, superseded_at: u64) -> String { + serialize_gc_pending(vec![GcPendingGeneration { + count: generation.count, + sha: generation.sha.clone(), + superseded_at, + }]) + .expect("serialize pending record") + } + /// Split a chunked envelope into (chunk keys, root pointer value). #[cfg(unix)] fn chunked_parts(root_key: &str, envelope: &str) -> (Vec, String) { @@ -3241,7 +3459,7 @@ echo 'unexpected' >&2; exit 1 #[cfg(unix)] #[test] - fn push_config_entries_deletes_prior_chunks_and_keeps_new() { + fn push_config_entries_defers_prior_generation_instead_of_deleting() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); @@ -3277,14 +3495,16 @@ echo 'unexpected' >&2; exit 1 // Every prior A-chunk is deleted; the root and new B-chunks are not. for key in &a_chunks { assert!( - oplog_has(&oplog, &format!("delete {key}")), - "prior chunk `{key}` must be deleted; log:\n{}", + !oplog_has(&oplog, &format!("delete {key}")), + "prior chunk `{key}` must NOT be deleted eagerly — POPs may still \ + be serving the previous pointer; log:\n{}", fs::read_to_string(&oplog).unwrap_or_default() ); } + let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( - !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), - "root pointer must never be deleted" + !log.lines().any(|line| line.starts_with("delete ")), + "a push must never delete the generation it just superseded; log:\n{log}" ); for key in &b_chunks { assert!( @@ -3292,8 +3512,86 @@ echo 'unexpected' >&2; exit 1 "new chunk `{key}` must not be deleted" ); } - // Deletes happen strictly after the root-pointer update (commit). + // Instead, the superseded generation is RECORDED for later reclamation. + assert!( + oplog_has( + &oplog, + &format!("update {}", gc_pending_key(TEST_CONFIG_ID)) + ), + "the superseded generation must be recorded as pending; log:\n{log}" + ); + assert!(out[0].contains("pushed"), "summary present: {out:?}"); + } + + /// Once a recorded generation has aged past the grace window and nothing + /// live references it, a later push reclaims it. + #[cfg(unix)] + #[test] + fn push_config_entries_reclaims_pending_generation_after_grace() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); // everything is immediately "aged" + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + + let make = |tag: &str, stamp: &str| { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") + }; + let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); + let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); + let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); + + let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); + let a_chunks = derive_chunk_keys(TEST_CONFIG_ID, &gen_a); + let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); + // A was superseded long ago and is already pending; B is the live prior. + let pending = pending_record_json(&gen_a, 0); + + let fake = fake_fastly_gc_with_pending( + TEST_CONFIG_ID, + &[b_pointer, c_pointer], + Some(&pending), + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_c)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + // The aged generation A is reclaimed... + for key in &a_chunks { + assert!( + oplog_has(&oplog, &format!("delete {key}")), + "aged pending chunk `{key}` must be reclaimed; log:\n{log}" + ); + } + // ...but B (just superseded, still live at lagging POPs) is retained. + for key in &b_chunks { + assert!( + !oplog_has(&oplog, &format!("delete {key}")), + "the just-superseded generation must be retained, not deleted: `{key}`" + ); + } + assert!( + !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), + "root pointer must never be deleted" + ); + // Reclamation happens after the commit. let root_update = log .lines() .position(|line| line == format!("update {TEST_CONFIG_ID}")) @@ -3304,9 +3602,63 @@ echo 'unexpected' >&2; exit 1 .expect("a delete logged"); assert!( first_delete > root_update, - "deletes must follow the root update; log:\n{log}" + "reclamation must follow the commit; log:\n{log}" + ); + } + + /// A pending generation inside the grace window is retained, not deleted — + /// this is what protects POPs that have not yet seen the new pointer. + #[cfg(unix)] + #[test] + fn push_config_entries_retains_pending_generation_within_grace() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("86400"); // 24h: nothing recent is aged + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + + let make = |tag: &str, stamp: &str| { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") + }; + let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); + let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); + let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); + + let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); + let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); + // A was superseded just now -> still inside the grace window. + let pending = pending_record_json(&gen_a, unix_now_secs()); + + let fake = fake_fastly_gc_with_pending( + TEST_CONFIG_ID, + &[b_pointer, c_pointer], + Some(&pending), + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_c)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "nothing may be reclaimed inside the grace window; log:\n{log}" ); - assert!(out[0].contains("pushed"), "summary present: {out:?}"); } #[cfg(unix)] @@ -3400,8 +3752,8 @@ echo 'unexpected' >&2; exit 1 .expect("dry-run must not error"); assert!( out.iter() - .any(|line| line.contains("would delete orphaned prior-generation chunks")), - "dry-run reports GC intent: {out:?}" + .any(|line| line.contains("would defer reclamation")), + "dry-run reports deferred-reclamation intent: {out:?}" ); } @@ -3413,8 +3765,15 @@ echo 'unexpected' >&2; exit 1 let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // Empty describe sequence => root not found => no prior chunks. - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, false, &oplog); + // No prior pointer (slot 1 empty); slot 2 is the post-commit read-back. + let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[String::new(), pointer], + None, + false, + &oplog, + ); let _path = PathPrepend::new(fake.path()); FastlyCliAdapter .push_config_entries( @@ -3439,22 +3798,30 @@ echo 'unexpected' >&2; exit 1 fn push_config_entries_delete_failure_warns_but_succeeds() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let envelope_b = { + let make = |tag: &str, stamp: &str| { use edgezero_core::blob_envelope::BlobEnvelope; use serde_json::json; - let data = json!({ "alt": "w".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:05Z".to_owned())) - .expect("B") + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") }; - let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); + let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); + let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); + + let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); + let a_chunks = derive_chunk_keys(TEST_CONFIG_ID, &gen_a); let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let fail_key = a_chunks[0].clone(); - let fake = fake_fastly_gc( + let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); + let pending = pending_record_json(&gen_a, 0); + let fail_key = a_chunks.first().expect("a chunk").clone(); + + let fake = fake_fastly_gc_with_pending( TEST_CONFIG_ID, - &[a_pointer, b_pointer], + &[b_pointer, c_pointer], + Some(&pending), Some(&fail_key), false, &oplog, @@ -3466,17 +3833,22 @@ echo 'unexpected' >&2; exit 1 None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &[(TEST_CONFIG_ID.to_owned(), envelope_c)], &AdapterPushContext::new(), false, ) - .expect("push still succeeds despite a failed delete"); + .expect("push still succeeds despite a failed reclamation delete"); assert!( out.iter() .any(|line| line.contains("could not reclaim orphan chunk") && line.contains(&fail_key)), "failed delete surfaces an informational warning: {out:?}" ); + assert!( + out.iter() + .any(|line| line.contains("retried on a later push")), + "the generation stays pending so a later push retries: {out:?}" + ); } #[cfg(unix)] @@ -3488,8 +3860,10 @@ echo 'unexpected' >&2; exit 1 let oplog = dir.path().join("ops.log"); let envelope_b = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); // Prior is pointer-kind but invalid (version 2) => warn, no deletes. + // Slot 2 is the post-commit read-back, which must show our own write. let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad], None, false, &oplog); + let (_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad, b_pointer], None, false, &oplog); let _path = PathPrepend::new(fake.path()); let out = FastlyCliAdapter .push_config_entries( @@ -3521,9 +3895,17 @@ echo 'unexpected' >&2; exit 1 let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // Describe fails hard (not a not-found) => prior read errors => GC - // is skipped for this root with a warning; the push still succeeds. - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], None, true, &oplog); + // The FIRST describe (the pre-commit prior read) fails hard; the + // post-commit read-back (slot 2) still returns what we wrote, so the + // LWW guard passes and we reach the prior-read warning. + let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[String::new(), pointer], + None, + true, + &oplog, + ); let _path = PathPrepend::new(fake.path()); let out = FastlyCliAdapter .push_config_entries( @@ -3623,19 +4005,30 @@ echo 'unexpected' >&2; exit 1 fn push_config_entries_delete_uses_key_and_auto_yes_never_all() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); // force a reclamation so deletes occur let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let envelope_b = { + let make = |tag: &str, stamp: &str| { use edgezero_core::blob_envelope::BlobEnvelope; use serde_json::json; - let data = json!({ "alt": "k".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:06Z".to_owned())) - .expect("B") + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") }; - let (_a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); + let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); + let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); + let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, false, &oplog); + let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); + let pending = pending_record_json(&gen_a, 0); + let fake = fake_fastly_gc_with_pending( + TEST_CONFIG_ID, + &[b_pointer, c_pointer], + Some(&pending), + None, + false, + &oplog, + ); let _path = PathPrepend::new(fake.path()); FastlyCliAdapter .push_config_entries( @@ -3643,7 +4036,7 @@ echo 'unexpected' >&2; exit 1 None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &[(TEST_CONFIG_ID.to_owned(), envelope_c)], &AdapterPushContext::new(), false, ) @@ -3707,12 +4100,21 @@ echo 'unexpected' >&2; exit 1 false, ) .expect("push succeeds"); + // Shrinking to a direct value orphans the whole prior chunk set — but + // it is DEFERRED, not deleted now: POPs may still serve the old pointer. for key in &prior_chunks { assert!( - oplog_has(&oplog, &format!("delete {key}")), - "prior chunk `{key}` must be deleted on shrink-to-direct" + !oplog_has(&oplog, &format!("delete {key}")), + "prior chunk `{key}` must be deferred, not deleted eagerly" ); } + assert!( + oplog_has( + &oplog, + &format!("update {}", gc_pending_key(TEST_CONFIG_ID)) + ), + "the orphaned generation must be recorded as pending" + ); assert!( !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), "root pointer must never be deleted" @@ -4933,13 +5335,11 @@ echo 'unexpected' >&2; exit 1 new_keys.insert("keep".to_owned()); let plan = FastlyConfigGcPlan { new_keys, - new_root_value: String::new(), prior_keys: Ok(vec![ "gone1".to_owned(), "keep".to_owned(), "gone2".to_owned(), ]), - root_key: "app_config".to_owned(), }; let orphans = orphan_chunk_keys(&plan).expect("ok"); assert_eq!(orphans, vec!["gone1".to_owned(), "gone2".to_owned()]); @@ -4949,9 +5349,7 @@ echo 'unexpected' >&2; exit 1 fn orphan_chunk_keys_propagates_prior_err() { let plan = FastlyConfigGcPlan { new_keys: HashSet::new(), - new_root_value: String::new(), prior_keys: Err("suspicious".to_owned()), - root_key: "app_config".to_owned(), }; orphan_chunk_keys(&plan).unwrap_err(); } From 20470956fc2bc89d76c1d4b8007393e0d87f2155 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:43:10 -0700 Subject: [PATCH 20/25] docs: spec the deferred cloud reclamation design + payload-redaction and duplicate-root invariants Records the design correction from the PR #314 review: eager cloud deletion is unsafe under Fastly's eventual consistency (the read-back only sees the control plane, not POP propagation), so the superseded generation is recorded and reclaimed by a later push after a grace window. Documents the pending-record shape (compact sha+count, list-not-slot), the 24h configurable grace, and the two further invariants (never echo config payloads; reject duplicate roots). --- .../specs/2026-07-07-fastly-chunk-gc.md | 104 ++++++++++++++++-- 1 file changed, 92 insertions(+), 12 deletions(-) diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 4638527c..7a2c4dd3 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -147,6 +147,84 @@ pointer no longer does. whole file in one in-memory rewrite and has no remote-concurrency window. +## Two further invariants (PR #314 review) + +**Diagnostics must never echo config payloads.** The `describe` shell-outs +interpolated their raw stdout — i.e. the stored config envelope, which may hold +credentials — into errors that the CLI logs verbatim. Because the GC prior-read +runs on *every* cloud push, this exposed the previous config on any schema +drift. Errors now report only the response's **size and top-level field-name +shape** (`redact_describe_response`), never a value. Enforced by sentinel-secret +tests on both the read and push paths. + +**A batch must not name the same logical root twice.** GC builds one plan per +entry and snapshots every plan against the *same* prior generation. With +`[(root, A), (root, B)]` the last tuple wins the upsert (root = B), yet A's plan +would still reclaim `prior − A_keys` — which includes B's freshly-written chunks +— leaving the final pointer dangling. Duplicate root keys are therefore a **hard +error** in both push paths, before any expansion or I/O. (Rejecting beats +silently coalescing: a duplicated key is a caller bug, and picking a winner would +hide it.) + +## Cloud reclamation is DEFERRED (eventual consistency) + +> **Design correction (PR #314 review).** An earlier revision of this spec had +> the cloud push delete the superseded generation immediately, guarded only by a +> post-commit read-back. That is **unsafe**: Fastly Config Store is eventually +> consistent, and the read-back observes only the **control plane**. After we +> write pointer N, POPs may still be serving pointer N-1 — which references +> N-1's chunks. Deleting them immediately strips chunks out from under those +> POPs, breaking reads on *every* re-push, not merely under concurrent writers. +> Knowing precisely *what* to delete was never the problem; knowing *when* it is +> safe is, and no control-plane read can answer that. + +So a cloud push **never deletes the generation it just superseded**. It +*records* it, and a **later** push reclaims it once it has aged out: + +``` +push N (root R): + 1. read prior pointer -> prior generation (N-1) + 2. read pending record -> generations superseded by EARLIER pushes + 3. commit N's chunks + N's pointer <- commit point + 4. LWW read-back: is the root still exactly what we wrote? (else yield) + 5. RECLAIM each pending generation g where + age(g.superseded_at) >= GRACE + and keys(g) disjoint from N's keep-set + and keys(g) disjoint from keys(N-1) <- may still be live at POPs + 6. RECORD pending += { N-1, superseded_at: now } +``` + +**Pending record** at `.__edgezero_gc.pending` (a reserved namespace — +`GC_PENDING_INFIX`, rejected in logical keys exactly like the chunk infix): + +```json +{ "edgezero_kind": "fastly_config_gc_pending", "version": 1, + "generations": [ { "sha": "", "count": 12, "superseded_at": 1720000000 } ] } +``` + +Two properties that are load-bearing: + +- **Compact, not raw keys.** Chunk keys are derivable + (`.__edgezero_chunks..`), so the record stores `(sha, count)`. + Storing the raw key list would exceed the record's own 8 000-char entry limit + after ~2 generations. +- **A list, not a slot.** If a push outpaces the grace window, the un-reclaimed + generation must not be overwritten — that would leak it *untracked*, forever. + +**Grace window:** `EDGEZERO_FASTLY_GC_GRACE_SECS`, default **86 400 (24h)**. +Fastly documents no propagation bound, so the default is deliberately generous; +operators on a faster cadence can lower it. Generation N-1 is retained +unconditionally regardless of the window. + +**Failure handling:** a failed delete leaves the generation pending, so a later +push retries (deletes are idempotent — not-found is success). A malformed +pending record warns and touches nothing rather than clobbering state it cannot +understand. + +**The LOCAL path still prunes eagerly** and that is correct: `fastly.toml` is a +single file Viceroy reads at startup — there is no propagation window and no POP +that could still be serving the previous pointer. + ## Concurrency model: last-writer-wins The config value follows **last-writer-wins (LWW)**. The root key is a @@ -165,18 +243,20 @@ last writer own reclamation. This is what keeps LWW honest — a superseded (losing) push never deletes the *winner's* live chunks, so the winner's config stays readable. -Best-effort boundary (stated honestly): the read-back narrows but does -not fully close the window, because Fastly exposes no compare-and-delete. -A newer push can still supersede us *between* our read-back and an -individual delete, so in a rare interleaving a losing push can delete a -chunk the new winner references. The blast radius is one store's chunk -data — never another store, never the root pointer. The next read of -that config then surfaces an integrity error (missing chunk) rather than -serving wrong data, and re-pushing the winning config restores it -(chunks are written before the pointer). GC is therefore a best-effort -reclaimer under LWW, not a transactional one. If that residual ever -matters, the deferred `config gc` non-goal is the path to fully -lease-guarded reclamation. +Best-effort boundary (stated honestly): Fastly exposes no +compare-and-delete, so a newer push can still supersede us *between* the +read-back and a delete. Deferral makes that residual far narrower than it +was under eager deletion, because the only thing a sweep can ever delete +is a generation that (a) was superseded at least one push ago, (b) has +aged past the grace window, and (c) is referenced by neither the new +generation nor the just-superseded one. A losing push therefore cannot +touch the winner's live chunks in the ordinary case; it would take a +concurrent revert to a *stale, aged-out* generation inside the sweep +window. The blast radius stays bounded to one store's chunk data — never +another store, never the root pointer — and a missing chunk surfaces as +an integrity error on read rather than wrong data, recoverable by +re-pushing (chunks are written before the pointer). GC remains a +best-effort reclaimer under LWW, not a transactional one. ## `prior_chunk_keys` helper (`chunked_config.rs`) From 545440207379cd2091db966a58554ec93c60f2ae Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 13 Jul 2026 08:37:26 -0700 Subject: [PATCH 21/25] =?UTF-8?q?fix(fastly):=20complete=20the=20payload-r?= =?UTF-8?q?edaction=20invariant=20=E2=80=94=20redact=20stderr=20too?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stdout fix was half a fix. The describe and update --stdin paths carry the stored config value, so a Fastly failure that quotes the payload back in stderr put credentials into logs via the FAILURE branch — the same exposure, different door. Redact stderr on those three value-bearing sites (not-found classification still inspects it internally; only the user-facing string is suppressed). delete/create/list stderr carry no value and are left verbatim. Adds sentinel-secret tests for the stderr path on both the read and push paths, complementing the existing successful-but-drifted stdout tests. --- crates/edgezero-adapter-fastly/src/cli.rs | 87 ++++++++++++++++++++++- 1 file changed, 84 insertions(+), 3 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index cfeda033..3f2ee7da 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -680,7 +680,7 @@ impl Adapter for FastlyCliAdapter { Err(format!( "`fastly config-store-entry describe --store-id={store_id} --key={key} --json` exited with status {}\nstderr: {}", output.status, - stderr.trim() + redact_stderr(&stderr) )) } @@ -828,7 +828,7 @@ fn fetch_remote_config_store_entry(store_id: &str, key: &str) -> Result Result<( Err(format!( "`fastly config-store-entry update --store-id={store_id} --key={key} --upsert --stdin` exited with status {}\nstderr: {}", output.status, - String::from_utf8_lossy(&output.stderr).trim() + redact_stderr(&String::from_utf8_lossy(&output.stderr)) )) } @@ -1685,6 +1685,20 @@ fn redact_describe_response(stdout: &str) -> String { ) } +/// Summarise a failing `fastly` invocation's stderr WITHOUT echoing it. +/// +/// The `describe` and `update --stdin` paths carry the stored config value, so +/// a Fastly error that quotes the payload back would put credentials straight +/// into CI logs — the same exposure as the stdout leak, via the failure branch. +/// Not-found *classification* still inspects stderr internally; only the +/// user-facing string is redacted. +fn redact_stderr(stderr: &str) -> String { + let len = stderr.trim().len(); + format!( + "{len} bytes suppressed (may echo the stored config value); re-run the `fastly` command directly to inspect it" + ) +} + /// One-line type label for a `serde_json::Value` (for diagnostic /// error messages — not a canonical JSON-schema description). fn shape_summary(value: &serde_json::Value) -> &'static str { @@ -3965,6 +3979,73 @@ echo 'unexpected' >&2; exit 1 ); } + /// The FAILURE branch leaks too: a Fastly error that quotes the stored + /// value back in stderr must not reach the user-facing error. + #[cfg(unix)] + #[test] + fn read_config_entry_stderr_failure_does_not_leak_payload() { + const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr1"; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + // Not a "not found" — a hard failure that echoes the value. + let stderr = format!("Error: internal failure processing value {SENTINEL}"); + let fake = fake_fastly_returning("", &stderr, 1); + let _path = PathPrepend::new(fake.path()); + + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + "cfg", + &AdapterPushContext::new(), + ); + let Err(err) = result else { + panic!("hard stderr failure must error") + }; + assert!( + !err.contains(SENTINEL), + "stderr must be redacted, not echoed: {err}" + ); + assert!( + err.contains("suppressed"), + "error should say the stderr was suppressed: {err}" + ); + } + + /// Same for the push path's GC prior-read, which runs on every cloud push. + #[cfg(unix)] + #[test] + fn push_config_entries_stderr_failure_does_not_leak_payload() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr2"; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + let stderr = format!("Error: internal failure processing value {SENTINEL}"); + let fake = fake_fastly_returning("", &stderr, 1); + let _path = PathPrepend::new(fake.path()); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let result = FastlyCliAdapter.push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), + false, + ); + // Whether it errors or warns, the payload must never appear. + let rendered = match result { + Ok(lines) => lines.join("\n"), + Err(err) => err, + }; + assert!( + !rendered.contains(SENTINEL), + "push output must not echo the stored value from stderr: {rendered}" + ); + } + /// The GC prior-read runs on EVERY cloud push, so a drifted response must /// not leak the previous config envelope into the push's status lines. #[cfg(unix)] From abda2261b23e975ab3c5c6a358e15eb9fd8a2d47 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 13 Jul 2026 12:16:49 -0700 Subject: [PATCH 22/25] refactor(fastly)!: derive cloud reclamation from the store, deleting the metadata sidecar The pending-record protocol was unsound and unfixable in place (PR #314 review): Fastly has no CAS, so a failed sidecar write / failed read-back / concurrent push PERMANENTLY lost a generation (and the 'will retry' warnings were false); and the record itself overflowed the 8000-char entry limit at ~71 generations, after which every later generation was silently lost. Verified against the live Fastly API and rebuilt with NO metadata at all -- the store IS the state: - `config-store-entry list --json` returns item_key + created_at per entry - a generation is superseded exactly when the NEXT one is written, so superseded_at(G) = created_at(successor(G)), read straight off the listing - reclaim G iff it is unreferenced by the live pointer AND now - superseded_at(G) >= grace (EDGEZERO_FASTLY_GC_GRACE_SECS, default 24h) - the generation this push just superseded is ALWAYS protected, so POPs that have not yet seen the new pointer keep their chunks IMPORTANT: the root entry's own updated_at is NOT usable -- Fastly does not bump it on `update --upsert`. Verified live: a root last 'updated' 2026-07-07 pointed at chunks created 2026-07-13. Using it as the supersession clock would have reclaimed the previous generation immediately. This dissolves four review findings structurally rather than patching them: no record to be malformed or to overflow, no lost updates, and deletes now target the store's ACTUAL keys (never keys re-derived from a content address, which could retarget onto keys the pointer never referenced). chunk_key_generation validates the key shape (hex sha + numeric index) so a hand-edited or foreign key can never become a delete target. --- .../src/chunked_config.rs | 240 +- crates/edgezero-adapter-fastly/src/cli.rs | 2423 +++++++++-------- 2 files changed, 1253 insertions(+), 1410 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/chunked_config.rs b/crates/edgezero-adapter-fastly/src/chunked_config.rs index 3af156ca..3bfae50c 100644 --- a/crates/edgezero-adapter-fastly/src/chunked_config.rs +++ b/crates/edgezero-adapter-fastly/src/chunked_config.rs @@ -39,14 +39,6 @@ pub(crate) const CHUNK_KEY_INFIX: &str = ".__edgezero_chunks."; /// BOTH the writer (when serialising the pointer) AND the resolver /// (when validating the parsed pointer) -- stays unconditional. pub(crate) const POINTER_KIND: &str = "fastly_config_chunks"; -/// Infix of the per-root deferred-reclamation record: -/// `.__edgezero_gc.pending`. Reserved like `CHUNK_KEY_INFIX` -- -/// a logical config key must never contain it. CLI writer only. -#[cfg(any(feature = "cli", test))] -pub(crate) const GC_PENDING_INFIX: &str = ".__edgezero_gc."; -/// `edgezero_kind` discriminant stored in the pending-reclamation record. -#[cfg(any(feature = "cli", test))] -pub(crate) const GC_PENDING_KIND: &str = "fastly_config_gc_pending"; // --------------------------------------------------------------------------- // Private pointer schema @@ -69,43 +61,6 @@ struct FastlyChunkRef { sha256: String, } -/// One chunk generation, identified by the envelope SHA its chunk keys are -/// addressed by. The keys themselves are DERIVABLE -/// (`.__edgezero_chunks..` for `idx` in `0..count`), so the -/// deferred-reclamation record stores this compact form rather than the raw -/// key list -- which would blow the 8 000-char entry limit after a couple of -/// generations. -#[cfg(any(feature = "cli", test))] -#[derive(Clone, PartialEq, Eq, Debug)] -pub(crate) struct ChunkGeneration { - pub count: usize, - pub sha: String, -} - -/// A generation awaiting reclamation, plus when it was superseded. -#[cfg(any(feature = "cli", test))] -#[derive(Clone, Debug, serde::Deserialize, serde::Serialize)] -pub(crate) struct GcPendingGeneration { - pub count: usize, - pub sha: String, - /// Unix epoch seconds at which this generation stopped being the live - /// pointer. Reclamation waits until it has aged past the grace window, - /// so lagging POPs still serving the old pointer keep their chunks. - pub superseded_at: u64, -} - -/// The per-root deferred-reclamation record stored at -/// `.__edgezero_gc.pending`. A LIST, not a single slot: pushing faster -/// than the grace window must not drop an un-reclaimed generation on the -/// floor (that would leak it untracked, forever). -#[cfg(any(feature = "cli", test))] -#[derive(serde::Deserialize, serde::Serialize)] -pub(crate) struct GcPendingRecord { - edgezero_kind: String, - generations: Vec, - version: u8, -} - // --------------------------------------------------------------------------- // Public helpers // --------------------------------------------------------------------------- @@ -407,98 +362,25 @@ pub(crate) fn prior_chunk_keys(root_key: &str, raw: &str) -> Result, Ok(keys) } -/// The prior pointer's generation (envelope SHA + chunk count), from which -/// its chunk keys are derivable. Same validation and prefix-scoping as -/// [`prior_chunk_keys`]: `Ok(None)` for a direct/absent/non-pointer value, -/// `Err` for a pointer-kind value that fails validation. -#[cfg(any(feature = "cli", test))] -pub(crate) fn prior_chunk_generation( - root_key: &str, - raw: &str, -) -> Result, String> { - let keys = prior_chunk_keys(root_key, raw)?; - if keys.is_empty() { - return Ok(None); - } - // The pointer validated above; re-read its sha for the compact form. - let value: serde_json::Value = serde_json::from_str(raw) - .map_err(|err| format!("prior chunk pointer at `{root_key}` is malformed: {err}"))?; - let sha = value - .get("envelope_sha256") - .and_then(serde_json::Value::as_str) - .ok_or_else(|| { - format!( - "prior chunk pointer at `{root_key}` has no `envelope_sha256`; skipping chunk GC" - ) - })?; - Ok(Some(ChunkGeneration { - count: keys.len(), - sha: sha.to_owned(), - })) -} - -/// Rebuild a generation's chunk keys from its content address. Mirrors the -/// key construction in [`prepare_fastly_config_entries`]. -#[cfg(any(feature = "cli", test))] -pub(crate) fn derive_chunk_keys(root_key: &str, generation: &ChunkGeneration) -> Vec { - (0..generation.count) - .map(|idx| { - format!( - "{root_key}{CHUNK_KEY_INFIX}{sha}.{idx}", - sha = generation.sha - ) - }) - .collect() -} - -/// Key of the per-root deferred-reclamation record. -#[cfg(any(feature = "cli", test))] -pub(crate) fn gc_pending_key(root_key: &str) -> String { - format!("{root_key}{GC_PENDING_INFIX}pending") -} - -/// Parse a pending-reclamation record. `Ok(vec![])` when absent or not our -/// record; `Err` when it IS our kind but fails validation (caller warns and -/// skips reclamation rather than clobbering state it cannot understand). +/// Extract the content-address (envelope SHA) of the generation a chunk key +/// belongs to, for `root_key`. Returns `None` when `key` is not a well-formed +/// chunk key of that root. +/// +/// This is how reclamation groups the store's ACTUAL keys into generations. It +/// validates the shape (`.__edgezero_chunks..`) rather +/// than trusting it: a hand-edited or foreign key never becomes a delete target. #[cfg(any(feature = "cli", test))] -pub(crate) fn parse_gc_pending( - root_key: &str, - raw: &str, -) -> Result, String> { - let value: serde_json::Value = match serde_json::from_str(raw) { - Ok(value) => value, - Err(_) => return Ok(Vec::new()), - }; - if value - .get("edgezero_kind") - .and_then(serde_json::Value::as_str) - != Some(GC_PENDING_KIND) - { - return Ok(Vec::new()); +pub(crate) fn chunk_key_generation(root_key: &str, key: &str) -> Option { + let prefix = format!("{root_key}{CHUNK_KEY_INFIX}"); + let rest = key.strip_prefix(&prefix)?; + let (sha, index) = rest.rsplit_once('.')?; + if sha.is_empty() || !sha.chars().all(|ch| ch.is_ascii_hexdigit()) { + return None; } - let record: GcPendingRecord = serde_json::from_value(value).map_err(|err| { - format!("GC pending record at `{root_key}` is malformed: {err}; skipping chunk reclamation") - })?; - if record.version != 1 { - return Err(format!( - "GC pending record at `{root_key}` has unsupported version {}; skipping chunk reclamation", - record.version - )); + if index.is_empty() || !index.chars().all(|ch| ch.is_ascii_digit()) { + return None; } - Ok(record.generations) -} - -/// Serialise a pending-reclamation record. -#[cfg(any(feature = "cli", test))] -pub(crate) fn serialize_gc_pending( - generations: Vec, -) -> Result { - serde_json::to_string(&GcPendingRecord { - edgezero_kind: GC_PENDING_KIND.to_owned(), - generations, - version: 1, - }) - .map_err(|err| format!("failed to serialise GC pending record: {err}")) + Some(sha.to_owned()) } // --------------------------------------------------------------------------- @@ -929,80 +811,36 @@ mod tests { assert!(err.contains("outside"), "{err}"); } - // ---- deferred-reclamation tests ---- + // ---- chunk_key_generation ---- - /// The compact generation form must regenerate EXACTLY the pointer's keys - /// — the pending record stores `(sha, count)` instead of the raw key list - /// (which would blow the 8 000-char entry limit), so reclamation depends - /// on this round-trip being lossless. #[test] - fn derive_chunk_keys_round_trips_the_pointers_keys() { + fn chunk_key_generation_extracts_the_sha() { let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); let entries = prepare_fastly_config_entries("app_config", &envelope).unwrap(); - let (_, pointer_json) = entries.last().unwrap(); - let expected: Vec = entries[..entries.len().saturating_sub(1)] - .iter() - .map(|(key, _)| key.clone()) - .collect(); - - let generation = prior_chunk_generation("app_config", pointer_json) - .expect("valid pointer") - .expect("chunked"); - assert_eq!(generation.count, expected.len()); - assert_eq!(derive_chunk_keys("app_config", &generation), expected); - } - - #[test] - fn prior_chunk_generation_is_none_for_direct_envelope() { - let envelope = make_envelope_json(FASTLY_CONFIG_ENTRY_LIMIT); - assert!(prior_chunk_generation("app_config", &envelope) - .unwrap() - .is_none()); - } - - #[test] - fn gc_pending_round_trips() { - let generations = vec![ - GcPendingGeneration { - count: 3, - sha: "aa".repeat(32), - superseded_at: 1_720_000_000, - }, - GcPendingGeneration { - count: 1, - sha: "bb".repeat(32), - superseded_at: 1_720_000_500, - }, - ]; - let raw = serialize_gc_pending(generations).expect("serialise"); - let parsed = parse_gc_pending("app_config", &raw).expect("parse"); - assert_eq!(parsed.len(), 2); - assert_eq!(parsed[0].count, 3); - assert_eq!(parsed[1].superseded_at, 1_720_000_500); + let (chunk_key, _) = &entries[0]; + let sha = chunk_key_generation("app_config", chunk_key).expect("a chunk key"); + assert_eq!(sha.len(), 64, "sha256 hex"); + assert!(chunk_key.contains(&sha)); } #[test] - fn parse_gc_pending_is_silent_for_unrelated_values() { - assert!(parse_gc_pending("app_config", r#"{"hello":"world"}"#) - .unwrap() - .is_empty()); - assert!(parse_gc_pending("app_config", "not json") - .unwrap() - .is_empty()); - } - - #[test] - fn parse_gc_pending_warns_on_unsupported_version() { - let raw = r#"{"edgezero_kind":"fastly_config_gc_pending","version":2,"generations":[]}"#; - let err = parse_gc_pending("app_config", raw).expect_err("version 2 must warn"); - assert!(err.contains("unsupported version"), "{err}"); - } - - #[test] - fn gc_pending_key_is_in_the_reserved_namespace() { - let key = gc_pending_key("app_config"); - assert!(key.contains(GC_PENDING_INFIX), "{key}"); - assert!(key.starts_with("app_config"), "{key}"); + fn chunk_key_generation_rejects_foreign_and_malformed_keys() { + // Different root. + assert!(chunk_key_generation("app_config", "other.__edgezero_chunks.abc123.0").is_none()); + // The root key itself. + assert!(chunk_key_generation("app_config", "app_config").is_none()); + // Non-hex content address. + assert!( + chunk_key_generation("app_config", "app_config.__edgezero_chunks.zzzz.0").is_none() + ); + // Non-numeric index. + assert!( + chunk_key_generation("app_config", "app_config.__edgezero_chunks.abc123.x").is_none() + ); + // Missing index. + assert!( + chunk_key_generation("app_config", "app_config.__edgezero_chunks.abc123").is_none() + ); } // Regression for the Value-first rule: a value that IS pointer-kind but diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 3f2ee7da..179b8750 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -1,4 +1,4 @@ -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::env; use std::fs; use std::io::{ErrorKind, Write as _}; @@ -8,9 +8,8 @@ use std::process::Stdio; use std::time::{SystemTime, UNIX_EPOCH}; use crate::chunked_config::{ - derive_chunk_keys, gc_pending_key, parse_gc_pending, prepare_fastly_config_entries, - prior_chunk_generation, prior_chunk_keys, resolve_fastly_config_value, serialize_gc_pending, - ChunkGeneration, GcPendingGeneration, CHUNK_KEY_INFIX, GC_PENDING_INFIX, + chunk_key_generation, prepare_fastly_config_entries, prior_chunk_keys, + resolve_fastly_config_value, CHUNK_KEY_INFIX, }; use ctor::ctor; use edgezero_adapter::cli_support::{ @@ -150,25 +149,11 @@ enum ConfigStoreLookup { SchemaDrift(String), } -/// Per-root plan for the CLOUD path's deferred reclamation. -/// -/// Fastly's config store is eventually consistent: after we write pointer N, -/// some POPs still serve pointer N-1 and need N-1's chunks. So a cloud push -/// never deletes what it just superseded — it *records* that generation and -/// reclaims it on a later push, once it has aged past the grace window. -struct FastlyCloudGcPlan { - /// Exact keep-set this push writes for the root (chunk keys + root key). - new_keys: HashSet, - /// Exactly the value written at `root_key`; the read-back guard compares - /// against it to confirm we are still the last writer (LWW). - new_root_value: String, - /// Generations already awaiting reclamation, read before the commit. - /// `Err` => record we cannot understand; warn and touch nothing. - pending: Result, String>, - /// The generation this push supersedes (N-1) — recorded as pending, never - /// deleted now. `Err` => suspicious prior pointer; warn and skip. - prior: Result, String>, - root_key: String, +/// One `config-store-entry list` item. The stored VALUE is deliberately not +/// captured: it is the config payload and must never be held or logged. +struct ConfigStoreItem { + created_at: String, + item_key: String, } /// Per-root plan for the LOCAL path's eager prune. @@ -176,7 +161,7 @@ struct FastlyCloudGcPlan { /// Local reclamation is safe to do immediately: `fastly.toml` is a single /// file that Viceroy reads at startup — there is no propagation window and no /// POP that could still be serving the previous pointer. (The cloud path -/// cannot do this; see [`FastlyCloudGcPlan`].) +/// cannot do this; see `reclaim_orphan_generations`.) struct FastlyConfigGcPlan { /// Exact keep-set this push writes for the root (chunk keys + root key). new_keys: HashSet, @@ -458,46 +443,37 @@ impl Adapter for FastlyCliAdapter { let resolved_id = resolve_remote_config_store_id(name)?; let now = unix_now_secs(); let grace = gc_grace_secs(); - // Build GC plans BEFORE the commit: read each root's prior pointer (the - // generation we are about to supersede) and its pending-reclamation - // record (generations superseded by EARLIER pushes). - let mut gc_plans: Vec = Vec::with_capacity(roots.len()); - for (root_key, new_keys, new_root_value) in roots { - let prior = match fetch_remote_config_store_entry(&resolved_id, &root_key) { - Ok(Some(raw)) => prior_chunk_generation(&root_key, &raw), - Ok(None) => Ok(None), - Err(err) => Err(format!( - "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" - )), - }; - let pending = - match fetch_remote_config_store_entry(&resolved_id, &gc_pending_key(&root_key)) { - Ok(Some(raw)) => parse_gc_pending(&root_key, &raw), + // BEFORE the commit, capture the generation each root is about to + // supersede. It stays protected: POPs may still be serving it. + let mut priors: Vec, String>> = Vec::with_capacity(roots.len()); + for (root_key, _, _) in &roots { + priors.push( + match fetch_remote_config_store_entry(&resolved_id, root_key) { + Ok(Some(raw)) => prior_chunk_keys(root_key, &raw), Ok(None) => Ok(Vec::new()), Err(err) => Err(format!( - "failed to read the GC pending record for `{root_key}`: {err}; skipping chunk reclamation" + "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" )), - }; - gc_plans.push(FastlyCloudGcPlan { - new_keys, - new_root_value, - pending, - prior, - root_key, - }); + }, + ); } // Commit all physical entries (each root's chunks first, its root // pointer last). push_entries_with_committer(&physical_entries, |key, value| { create_config_store_entry(&resolved_id, key, value) })?; - // Post-commit: DEFERRED reclamation. We never delete the generation we - // just superseded — the config store is eventually consistent, so POPs - // may still be serving it. Instead we record it, and reclaim only older - // generations that have aged past the grace window. + // Post-commit reclamation, derived entirely from the store itself. let mut warnings = Vec::new(); - for plan in gc_plans { - warnings.extend(reclaim_and_record(&resolved_id, plan, now, grace)); + for ((root_key, new_keys, new_root_value), prior) in roots.into_iter().zip(priors) { + warnings.extend(reclaim_orphan_generations( + &resolved_id, + &root_key, + &new_keys, + &new_root_value, + prior, + now, + grace, + )); } let mut out = vec![format!( "pushed {} physical entries ({} logical) to fastly config-store `{name}` (logical id `{logical}`, id={resolved_id})", @@ -1188,12 +1164,10 @@ fn orphan_chunk_keys(plan: &FastlyConfigGcPlan) -> Result, String> { /// space, and could not be reclaimed correctly. fn reject_reserved_root_keys(entries: &[(String, String)]) -> Result<(), String> { for (key, _) in entries { - for infix in [CHUNK_KEY_INFIX, GC_PENDING_INFIX] { - if key.contains(infix) { - return Err(format!( - "config key `{key}` contains the reserved infix `{infix}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" - )); - } + if key.contains(CHUNK_KEY_INFIX) { + return Err(format!( + "config key `{key}` contains the reserved infix `{CHUNK_KEY_INFIX}`, which collides with Fastly chunk storage; choose a different config key (or --key override)" + )); } } Ok(()) @@ -1345,113 +1319,173 @@ fn local_orphan_counts_for_dry_run( /// exists" error, which is the operator's signal to delete the /// entry (or use `config-store-entry update` manually) before /// re-running push. -/// Post-commit deferred reclamation for ONE root. +/// `fastly config-store-entry list --store-id= --json`, keeping only each +/// item's key and `created_at`. The item VALUE is discarded on purpose (it is +/// the stored config; see `redact_describe_response`). +fn list_config_store_entries(store_id: &str) -> Result, String> { + let store_arg = format!("--store-id={store_id}"); + let output = Command::new("fastly") + .args(["config-store-entry", "list", store_arg.as_str(), "--json"]) + .output() + .map_err(|err| { + if err.kind() == ErrorKind::NotFound { + format!("`fastly` not found on PATH; {FASTLY_INSTALL_HINT}") + } else { + format!("failed to spawn `fastly`: {err}") + } + })?; + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + return Err(format!( + "`fastly config-store-entry list --store-id={store_id} --json` exited with status {}\nstderr: {}", + output.status, + redact_stderr(&stderr) + )); + } + let stdout = String::from_utf8_lossy(&output.stdout); + let parsed: serde_json::Value = serde_json::from_str(&stdout).map_err(|err| { + format!( + "failed to parse `fastly config-store-entry list` JSON: {err} (response: {})", + redact_describe_response(&stdout) + ) + })?; + let array = parsed + .as_array() + .or_else(|| parsed.get("items").and_then(serde_json::Value::as_array)) + .ok_or_else(|| { + format!( + "`fastly config-store-entry list` returned unexpected JSON (response: {})", + redact_describe_response(&stdout) + ) + })?; + let mut items = Vec::with_capacity(array.len()); + for entry in array { + let Some(item_key) = entry.get("item_key").and_then(serde_json::Value::as_str) else { + continue; + }; + let created_at = entry + .get("created_at") + .and_then(serde_json::Value::as_str) + .unwrap_or_default(); + items.push(ConfigStoreItem { + created_at: created_at.to_owned(), + item_key: item_key.to_owned(), + }); + } + Ok(items) +} + +/// RFC 3339 (`2026-07-13T03:27:42Z`) -> unix seconds. +fn parse_rfc3339_secs(raw: &str) -> Option { + let stamp = chrono::DateTime::parse_from_rfc3339(raw).ok()?; + u64::try_from(stamp.timestamp()).ok() +} + +/// Post-commit reclamation for ONE root, derived entirely FROM THE STORE. +/// +/// Fastly's config store is eventually consistent: after the control plane +/// accepts pointer N, POPs may still serve pointer N-1 and need N-1's chunks. +/// So a generation may only be deleted once it has been superseded for longer +/// than the grace window. The question is how to know WHEN it was superseded. +/// +/// A generation is superseded exactly when the NEXT one is written, so +/// `superseded_at(G) = created_at(successor(G))` — read straight off +/// `config-store-entry list`. There is no metadata sidecar: the store IS the +/// state, so there is nothing to lose on a failed write, nothing to overflow +/// the 8 000-char entry limit, and no lost update under concurrent pushes. /// -/// Never deletes the generation this push just superseded: Fastly's config -/// store is eventually consistent, so POPs may still be serving the previous -/// pointer and need its chunks. Instead the superseded generation is RECORDED, -/// and only generations that have aged past the grace window (and that nothing -/// live references) are reclaimed. Returns status/warning lines; a failure here -/// never fails the push. -fn reclaim_and_record( +/// NOTE: the root entry's own `updated_at` is NOT usable — Fastly does not bump +/// it on `update --upsert` (verified against the live API: a root last +/// "updated" 2026-07-07 pointed at chunks created 2026-07-13). +fn reclaim_orphan_generations( store_id: &str, - plan: FastlyCloudGcPlan, + root_key: &str, + new_keys: &HashSet, + new_root_value: &str, + prior: Result, String>, now: u64, grace: u64, ) -> Vec { let mut warnings = Vec::new(); - // Last-writer-wins: only touch GC state while the root still holds exactly - // what this push wrote. - match fetch_remote_config_store_entry(store_id, &plan.root_key) { - Ok(Some(current)) if current == plan.new_root_value => {} + // Last-writer-wins: only reclaim while the root still holds our write. + match fetch_remote_config_store_entry(store_id, root_key) { + Ok(Some(current)) if current == new_root_value => {} Ok(_) => { warnings.push(format!( - "note: skipped chunk GC for `{}`: root changed since this push wrote it (concurrent push?); reclamation deferred to a later push", - plan.root_key + "note: skipped chunk reclamation for `{root_key}`: root changed since this push wrote it (concurrent push?); a later push will reclaim" )); return warnings; } Err(err) => { warnings.push(format!( - "note: skipped chunk GC for `{}`: could not re-read the root ({err}); reclamation deferred to a later push", - plan.root_key + "note: skipped chunk reclamation for `{root_key}`: could not re-read the root ({err}); a later push will reclaim" )); return warnings; } } - let prior = match plan.prior { - Ok(prior) => prior, + let prior_keys = match prior { + Ok(keys) => keys, Err(err) => { warnings.push(format!("warning: {err}")); return warnings; } }; - let pending = match plan.pending { - Ok(pending) => pending, + let items = match list_config_store_entries(store_id) { + Ok(items) => items, Err(err) => { - warnings.push(format!("warning: {err}")); + warnings.push(format!( + "note: skipped chunk reclamation for `{root_key}`: {err}" + )); return warnings; } }; - // Keys that must NOT be reclaimed: the generation we just wrote, and the one - // we just superseded (still live at lagging POPs). - let mut protected: HashSet = plan.new_keys.clone(); - if let Some(generation) = &prior { - protected.extend(derive_chunk_keys(&plan.root_key, generation)); - } - - let mut retained: Vec = Vec::with_capacity(pending.len()); - for generation in pending { - let candidate = ChunkGeneration { - count: generation.count, - sha: generation.sha.clone(), + + // Group the store's ACTUAL chunk keys into generations. We delete only keys + // the store really has — never keys re-derived from a content address. + let mut grouped: HashMap)> = HashMap::new(); + for item in items { + let Some(sha) = chunk_key_generation(root_key, &item.item_key) else { + continue; }; - let keys = derive_chunk_keys(&plan.root_key, &candidate); - let aged = now.saturating_sub(generation.superseded_at) >= grace; - let live = keys.iter().any(|key| protected.contains(key)); - if !aged || live { - retained.push(generation); // not yet safe to reclaim + let created = parse_rfc3339_secs(&item.created_at).unwrap_or(0); + let slot = grouped.entry(sha).or_insert((0, Vec::new())); + slot.0 = slot.0.max(created); // the generation is complete at its newest chunk + slot.1.push(item.item_key); + } + if grouped.is_empty() { + return warnings; + } + let mut ordered: Vec<(String, u64, Vec)> = grouped + .into_iter() + .map(|(sha, (created, keys))| (sha, created, keys)) + .collect(); + ordered.sort_by(|left, right| left.1.cmp(&right.1).then_with(|| left.0.cmp(&right.0))); + + // Never touch the generation we just wrote, nor the one we just superseded + // (still live at POPs that have not caught up). + let mut protected: HashSet<&str> = new_keys.iter().map(String::as_str).collect(); + protected.extend(prior_keys.iter().map(String::as_str)); + + for (idx, (_, created, keys)) in ordered.iter().enumerate() { + if keys.iter().any(|key| protected.contains(key.as_str())) { continue; } - // Aged out and unreferenced: reclaim. A failed delete keeps the - // generation pending so a later push retries (deletes are idempotent -- - // a not-found is treated as success). - let mut all_deleted = true; - for key in keys { - if let Err(err) = delete_config_store_entry(store_id, &key) { - warnings.push(format!( - "note: could not reclaim orphan chunk `{key}` for `{}` ({err}); it is inert and will be retried on a later push", - plan.root_key - )); - all_deleted = false; - } - } - if !all_deleted { - retained.push(generation); + // Superseded when the NEXT generation was written. With no successor it + // was never referenced by any pointer (e.g. a partially-committed push), + // so age it from its own creation. + let superseded_at = ordered + .get(idx.saturating_add(1)) + .map_or(*created, |next| next.1); + if now.saturating_sub(superseded_at) < grace { + continue; } - } - // Record the generation this push superseded, so a LATER push can reclaim it - // once it has aged out. - if let Some(generation) = prior { - retained.retain(|held| held.sha != generation.sha); - retained.push(GcPendingGeneration { - count: generation.count, - sha: generation.sha, - superseded_at: now, - }); - } - match serialize_gc_pending(retained) { - Ok(raw) => { - if let Err(err) = - create_config_store_entry(store_id, &gc_pending_key(&plan.root_key), &raw) - { + for key in keys { + if let Err(err) = delete_config_store_entry(store_id, key) { warnings.push(format!( - "note: could not update the GC pending record for `{}` ({err}); reclamation will retry on a later push", - plan.root_key + "note: could not reclaim orphan chunk `{key}` for `{root_key}` ({err}); it is inert and a later push will retry" )); } } - Err(err) => warnings.push(format!("warning: {err}")), } warnings } @@ -3326,40 +3360,20 @@ build = \"cargo build --release\" /// Fake `fastly` for cloud chunk-GC tests. Logs each /// `config-store-entry` op ("describe " / "update " / - /// "delete ", plus "delete-argv ") to `oplog`, one per - /// line. `root_describe_seq` - /// gives the successive raw `item_value`s returned when the ROOT key - /// is described (call 1 = prior read, call 2 = read-back guard); - /// exhausting the sequence yields "not found". `fail_delete_key` - /// makes that one delete exit non-zero. `describe_hard_error` makes the - /// FIRST describe of each key fail with a non-not-found error (so the - /// pre-commit prior read returns `Err` while the post-commit read-back - /// still succeeds — exercising the prior-read-failure GC path). + /// "delete ", plus "delete-argv ") to `oplog`. + /// + /// `root_describe_seq` gives the successive raw `item_value`s returned when + /// the ROOT key is described (call 1 = the pre-commit prior read, call 2 = + /// the post-commit read-back). `entry_list` is served for + /// `config-store-entry list` and is what reclamation derives generations + /// and supersession times from. `fail_delete_key` makes that one delete + /// exit non-zero. `describe_hard_error` makes the FIRST describe of each key + /// fail hard (so the prior read errors while the read-back still works). #[cfg(unix)] fn fake_fastly_gc( root_key: &str, root_describe_seq: &[String], - fail_delete_key: Option<&str>, - describe_hard_error: bool, - oplog: &Path, - ) -> tempfile::TempDir { - fake_fastly_gc_with_pending( - root_key, - root_describe_seq, - None, - fail_delete_key, - describe_hard_error, - oplog, - ) - } - - /// As [`fake_fastly_gc`], plus an optional seeded pending-reclamation - /// record served when the `.__edgezero_gc.pending` key is described. - #[cfg(unix)] - fn fake_fastly_gc_with_pending( - root_key: &str, - root_describe_seq: &[String], - pending_seed: Option<&str>, + entry_list: &[(String, String)], fail_delete_key: Option<&str>, describe_hard_error: bool, oplog: &Path, @@ -3373,6 +3387,7 @@ if [ "$1" = "config-store" ]; then cat '{{{list}}}'; exit 0; fi sub="$2" key="" for arg in "$@"; do case "$arg" in --key=*) key="${arg#--key=}";; esac; done +if [ "$sub" = "list" ]; then printf 'list\n' >> '{{{oplog}}}'; cat '{{{entries}}}'; exit 0; fi if [ "$sub" = "update" ]; then cat >/dev/null; printf 'update %s\n' "$key" >> '{{{oplog}}}'; exit 0; fi if [ "$sub" = "delete" ]; then printf 'delete %s\n' "$key" >> '{{{oplog}}}'; printf 'delete-argv %s\n' "$*" >> '{{{oplog}}}'; if [ "$key" = "{{{fail}}}" ]; then echo 'Error: boom' >&2; exit 1; fi; exit 0; fi if [ "$sub" = "describe" ]; then @@ -3393,6 +3408,8 @@ echo 'unexpected' >&2; exit 1 format!(r#"[{{"name":"{TEST_CONFIG_ID}","id":"store-abc123"}}]"#), ) .expect("list"); + let entries_file = dir.path().join("entries.json"); + fs::write(&entries_file, entry_list_json(entry_list)).expect("entries"); for (index, value) in root_describe_seq.iter().enumerate() { let wrapped = format!( r#"{{"item_value":{}}}"#, @@ -3405,16 +3422,9 @@ echo 'unexpected' >&2; exit 1 ) .expect("resp"); } - if let Some(record) = pending_seed { - let key = gc_pending_key(root_key); - let wrapped = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(record).expect("escape") - ); - fs::write(dir.path().join(format!("resp_{key}_1.json")), wrapped).expect("pending"); - } let data = serde_json::json!({ "list": list_file.display().to_string(), + "entries": entries_file.display().to_string(), "oplog": oplog.display().to_string(), "dir": dir.path().display().to_string(), "fail": fail_delete_key.unwrap_or(""), @@ -3431,24 +3441,39 @@ echo 'unexpected' >&2; exit 1 dir } - /// The chunk generation (sha + count) a chunked envelope produces. + /// A `config-store-entry list --json` payload. The item VALUE is a + /// placeholder: reclamation must only ever use keys and timestamps. + #[cfg(unix)] + fn entry_list_json(items: &[(String, String)]) -> String { + let entries: Vec = items + .iter() + .map(|(key, created)| { + serde_json::json!({ + "item_key": key, + "created_at": created, + "item_value": "PLACEHOLDER", + }) + }) + .collect(); + serde_json::to_string(&entries).expect("entry list json") + } + + /// An RFC-3339 stamp `secs` in the past (the shape Fastly returns). #[cfg(unix)] - fn generation_of(root_key: &str, envelope: &str) -> ChunkGeneration { - let (_, pointer) = chunked_parts(root_key, envelope); - prior_chunk_generation(root_key, &pointer) - .expect("valid pointer") - .expect("chunked envelope has a generation") + fn stamp_secs_ago(secs: u64) -> String { + let delta = chrono::Duration::seconds(i64::try_from(secs).unwrap_or(0)); + let now = chrono::Utc::now(); + now.checked_sub_signed(delta) + .unwrap_or(now) + .to_rfc3339_opts(chrono::SecondsFormat::Secs, true) } - /// A seeded pending-reclamation record holding one generation. + /// Every chunk key of `envelope` paired with a creation stamp. #[cfg(unix)] - fn pending_record_json(generation: &ChunkGeneration, superseded_at: u64) -> String { - serialize_gc_pending(vec![GcPendingGeneration { - count: generation.count, - sha: generation.sha.clone(), - superseded_at, - }]) - .expect("serialize pending record") + fn listed_generation(root_key: &str, envelope: &str, secs_ago: u64) -> Vec<(String, String)> { + let (chunks, _) = chunked_parts(root_key, envelope); + let stamp = stamp_secs_ago(secs_ago); + chunks.into_iter().map(|key| (key, stamp.clone())).collect() } /// Split a chunked envelope into (chunk keys, root pointer value). @@ -3473,286 +3498,139 @@ echo 'unexpected' >&2; exit 1 #[cfg(unix)] #[test] - fn push_config_entries_defers_prior_generation_instead_of_deleting() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let _lock = path_mutation_guard().lock().expect("guard"); + fn push_config_entries_rejects_reserved_key() { let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let envelope_b = { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ "alt": "z".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:03Z".to_owned())) - .expect("B") - }; - let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - - // describe#1 = prior A pointer; describe#2 (read-back) = new B pointer. - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[a_pointer, b_pointer], None, false, &oplog); - let _path = PathPrepend::new(fake.path()); - - let out = FastlyCliAdapter + let bad_key = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); + let err = FastlyCliAdapter .push_config_entries( dir.path(), None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &[(bad_key.clone(), "{}".to_owned())], &AdapterPushContext::new(), false, ) - .expect("push must succeed"); - - // Every prior A-chunk is deleted; the root and new B-chunks are not. - for key in &a_chunks { - assert!( - !oplog_has(&oplog, &format!("delete {key}")), - "prior chunk `{key}` must NOT be deleted eagerly — POPs may still \ - be serving the previous pointer; log:\n{}", - fs::read_to_string(&oplog).unwrap_or_default() - ); - } - let log = fs::read_to_string(&oplog).unwrap_or_default(); - assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "a push must never delete the generation it just superseded; log:\n{log}" - ); - for key in &b_chunks { - assert!( - !oplog_has(&oplog, &format!("delete {key}")), - "new chunk `{key}` must not be deleted" - ); - } - // Instead, the superseded generation is RECORDED for later reclamation. - assert!( - oplog_has( - &oplog, - &format!("update {}", gc_pending_key(TEST_CONFIG_ID)) - ), - "the superseded generation must be recorded as pending; log:\n{log}" - ); - assert!(out[0].contains("pushed"), "summary present: {out:?}"); + .expect_err("reserved key must be rejected"); + assert!(err.contains(&bad_key), "names the key: {err}"); } - /// Once a recorded generation has aged past the grace window and nothing - /// live references it, a later push reclaims it. + /// Schema drift must never echo the config payload. App config can hold + /// credentials; CLI status lines are logged verbatim and CI logs are + /// retained/shared. Only a size + field-name shape may be reported. #[cfg(unix)] #[test] - fn push_config_entries_reclaims_pending_generation_after_grace() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn read_config_entry_schema_drift_does_not_leak_payload() { + const SENTINEL: &str = "SUPER_SECRET_TOKEN_abc123"; let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); // everything is immediately "aged" let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let make = |tag: &str, stamp: &str| { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") - }; - let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); - let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); - let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); - - let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); - let a_chunks = derive_chunk_keys(TEST_CONFIG_ID, &gen_a); - let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); - // A was superseded long ago and is already pending; B is the live prior. - let pending = pending_record_json(&gen_a, 0); + // Valid JSON, but the value moved out of `item_value` (schema drift). + let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); + let fake = fake_fastly_returning(&drift, "", 0); + let _path = PathPrepend::new(fake.path()); - let fake = fake_fastly_gc_with_pending( - TEST_CONFIG_ID, - &[b_pointer, c_pointer], - Some(&pending), + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), None, - false, - &oplog, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + "cfg", + &AdapterPushContext::new(), ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_c)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - - let log = fs::read_to_string(&oplog).unwrap_or_default(); - // The aged generation A is reclaimed... - for key in &a_chunks { - assert!( - oplog_has(&oplog, &format!("delete {key}")), - "aged pending chunk `{key}` must be reclaimed; log:\n{log}" - ); - } - // ...but B (just superseded, still live at lagging POPs) is retained. - for key in &b_chunks { - assert!( - !oplog_has(&oplog, &format!("delete {key}")), - "the just-superseded generation must be retained, not deleted: `{key}`" - ); - } + let Err(err) = result else { + panic!("schema drift must error") + }; assert!( - !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), - "root pointer must never be deleted" + !err.contains(SENTINEL), + "error must not leak the config payload: {err}" ); - // Reclamation happens after the commit. - let root_update = log - .lines() - .position(|line| line == format!("update {TEST_CONFIG_ID}")) - .expect("root update logged"); - let first_delete = log - .lines() - .position(|line| line.starts_with("delete ")) - .expect("a delete logged"); assert!( - first_delete > root_update, - "reclamation must follow the commit; log:\n{log}" + err.contains("bytes"), + "error should carry a redacted size/shape summary: {err}" ); } - /// A pending generation inside the grace window is retained, not deleted — - /// this is what protects POPs that have not yet seen the new pointer. + /// The FAILURE branch leaks too: a Fastly error that quotes the stored + /// value back in stderr must not reach the user-facing error. #[cfg(unix)] #[test] - fn push_config_entries_retains_pending_generation_within_grace() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn read_config_entry_stderr_failure_does_not_leak_payload() { + const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr1"; let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("86400"); // 24h: nothing recent is aged let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let make = |tag: &str, stamp: &str| { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") - }; - let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); - let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); - let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); - - let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); - let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); - // A was superseded just now -> still inside the grace window. - let pending = pending_record_json(&gen_a, unix_now_secs()); + // Not a "not found" — a hard failure that echoes the value. + let stderr = format!("Error: internal failure processing value {SENTINEL}"); + let fake = fake_fastly_returning("", &stderr, 1); + let _path = PathPrepend::new(fake.path()); - let fake = fake_fastly_gc_with_pending( - TEST_CONFIG_ID, - &[b_pointer, c_pointer], - Some(&pending), + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), None, - false, - &oplog, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + "cfg", + &AdapterPushContext::new(), ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_c)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - - let log = fs::read_to_string(&oplog).unwrap_or_default(); + let Err(err) = result else { + panic!("hard stderr failure must error") + }; assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "nothing may be reclaimed inside the grace window; log:\n{log}" + !err.contains(SENTINEL), + "stderr must be redacted, not echoed: {err}" + ); + assert!( + err.contains("suppressed"), + "error should say the stderr was suppressed: {err}" ); } + /// Same for the push path's GC prior-read, which runs on every cloud push. #[cfg(unix)] #[test] - fn push_config_entries_skips_gc_when_root_changed() { + fn push_config_entries_stderr_failure_does_not_leak_payload() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr2"; let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); + let stderr = format!("Error: internal failure processing value {SENTINEL}"); + let fake = fake_fastly_returning("", &stderr, 1); + let _path = PathPrepend::new(fake.path()); - let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let envelope_b = { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ "alt": "q".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:04Z".to_owned())) - .expect("B") - }; - let (_a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - - // Read-back returns the OLD A pointer (a concurrent push reverted the - // root), which differs from what this push wrote -> GC yields. - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer.clone(), a_pointer], + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let result = FastlyCliAdapter.push_config_entries( + dir.path(), + None, None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope)], + &AdapterPushContext::new(), false, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push must succeed"); - - let log = fs::read_to_string(&oplog).unwrap_or_default(); - assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "no deletes when the root changed under us; log:\n{log}" ); + // Whether it errors or warns, the payload must never appear. + let rendered = match result { + Ok(lines) => lines.join("\n"), + Err(err) => err, + }; assert!( - out.iter().any(|line| line.contains("root changed")), - "must warn that GC was skipped: {out:?}" + !rendered.contains(SENTINEL), + "push output must not echo the stored value from stderr: {rendered}" ); } + /// The GC prior-read runs on EVERY cloud push, so a drifted response must + /// not leak the previous config envelope into the push's status lines. #[cfg(unix)] #[test] - fn push_config_entries_rejects_reserved_key() { - let dir = tempdir().expect("tempdir"); - let bad_key = format!("app_config{CHUNK_KEY_INFIX}deadbeef.0"); - let err = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(bad_key.clone(), "{}".to_owned())], - &AdapterPushContext::new(), - false, - ) - .expect_err("reserved key must be rejected"); - assert!(err.contains(&bad_key), "names the key: {err}"); - } - - #[cfg(unix)] - #[test] - fn push_config_entries_dry_run_reports_gc_intent_offline() { + fn push_config_entries_prior_read_drift_does_not_leak_payload() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + const SENTINEL: &str = "SUPER_SECRET_TOKEN_xyz789"; + let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); + let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); + let fake = fake_fastly_returning(&drift, "", 0); + let _path = PathPrepend::new(fake.path()); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // No fake on PATH: a dry-run must not shell out to fastly at all. let out = FastlyCliAdapter .push_config_entries( dir.path(), @@ -3761,1135 +3639,1258 @@ echo 'unexpected' >&2; exit 1 &ResolvedStoreId::from_logical(TEST_CONFIG_ID), &[(TEST_CONFIG_ID.to_owned(), envelope)], &AdapterPushContext::new(), - true, + false, ) - .expect("dry-run must not error"); - assert!( - out.iter() - .any(|line| line.contains("would defer reclamation")), - "dry-run reports deferred-reclamation intent: {out:?}" - ); + .expect("push still succeeds; the prior read only warns"); + for line in &out { + assert!( + !line.contains(SENTINEL), + "push status must not leak the prior config payload: {line}" + ); + } } #[cfg(unix)] #[test] - fn push_config_entries_no_prior_issues_no_deletes() { + fn push_config_entries_writes_direct_entry_at_exactly_8000_chars() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // No prior pointer (slot 1 empty); slot 2 is the post-commit read-back. - let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[String::new(), pointer], - None, - false, - &oplog, - ); + let argv_log = dir.path().join("argv.txt"); + let fake = fake_fastly_argv_log(&argv_log); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + assert_eq!(envelope.len(), FASTLY_CONFIG_ENTRY_LIMIT); + + let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; + let out = FastlyCliAdapter .push_config_entries( dir.path(), None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], + &entries, &AdapterPushContext::new(), false, ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); + .expect("push must succeed"); + // One physical entry written (direct). + let captured = fs::read_to_string(&argv_log).expect("argv log"); assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "no prior => no deletes; log:\n{log}" + captured.contains(&format!("--key={TEST_CONFIG_ID}")), + "must write root key directly: {captured}" + ); + assert!( + out[0].contains("1 physical entries (1 logical)"), + "summary reports 1 physical entry: {out:?}" ); } #[cfg(unix)] #[test] - fn push_config_entries_delete_failure_warns_but_succeeds() { + fn push_config_entries_writes_chunks_and_root_pointer_for_8001_chars() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let make = |tag: &str, stamp: &str| { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") - }; - let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); - let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); - let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); - - let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); - let a_chunks = derive_chunk_keys(TEST_CONFIG_ID, &gen_a); - let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); - let pending = pending_record_json(&gen_a, 0); - let fail_key = a_chunks.first().expect("a chunk").clone(); - - let fake = fake_fastly_gc_with_pending( - TEST_CONFIG_ID, - &[b_pointer, c_pointer], - Some(&pending), - Some(&fail_key), - false, - &oplog, - ); + let argv_log = dir.path().join("argv.txt"); + let fake = fake_fastly_argv_log(&argv_log); let _path = PathPrepend::new(fake.path()); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + assert!(envelope.len() > FASTLY_CONFIG_ENTRY_LIMIT); + + let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; let out = FastlyCliAdapter .push_config_entries( dir.path(), None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_c)], + &entries, &AdapterPushContext::new(), false, ) - .expect("push still succeeds despite a failed reclamation delete"); + .expect("push must succeed"); + let captured = fs::read_to_string(&argv_log).expect("argv log"); + // At least one chunk key must appear before the root key. assert!( - out.iter() - .any(|line| line.contains("could not reclaim orphan chunk") - && line.contains(&fail_key)), - "failed delete surfaces an informational warning: {out:?}" + captured.contains(".__edgezero_chunks."), + "chunk keys must be written: {captured}" ); + // Root pointer must also be written. assert!( - out.iter() - .any(|line| line.contains("retried on a later push")), - "the generation stays pending so a later push retries: {out:?}" + captured.contains(&format!("--key={TEST_CONFIG_ID}")), + "root pointer must be written: {captured}" + ); + // Root key must be LAST in the log (chunk lines come before it). + let root_pos = captured.rfind(&format!("--key={TEST_CONFIG_ID}")).unwrap(); + let chunk_pos = captured.find(".__edgezero_chunks.").unwrap(); + assert!( + chunk_pos < root_pos, + "chunk writes must precede root pointer write: chunk_pos={chunk_pos} root_pos={root_pos}" ); + assert!(out[0].contains("logical"), "summary line present: {out:?}"); } #[cfg(unix)] #[test] - fn push_config_entries_suspicious_prior_warns_no_deletes() { + fn push_config_entries_dry_run_reports_direct_vs_chunked() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let envelope_b = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // Prior is pointer-kind but invalid (version 2) => warn, no deletes. - // Slot 2 is the post-commit read-back, which must show our own write. - let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); - let (_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let fake = fake_fastly_gc(TEST_CONFIG_ID, &[bad, b_pointer], None, false, &oplog); - let _path = PathPrepend::new(fake.path()); + + let direct_envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let chunked_envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + + let entries = vec![ + ("cfg_direct".to_owned(), direct_envelope), + ("cfg_chunked".to_owned(), chunked_envelope), + ]; let out = FastlyCliAdapter .push_config_entries( dir.path(), None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &entries, &AdapterPushContext::new(), - false, + true, // dry_run ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); + .expect("dry-run must not error"); + + // No shellout happens; output must describe intent. + let combined = out.join("\n"); assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "suspicious prior => no deletes; log:\n{log}" + combined.contains("would push `cfg_direct` as direct entry"), + "must report direct: {combined}" ); assert!( - out.iter().any(|line| line.contains("skipping chunk GC")), - "warns about the suspicious pointer: {out:?}" + combined.contains("would push `cfg_chunked` as chunked"), + "must report chunked: {combined}" ); } + /// Spec 12.7: pushing two blobs under different root keys + /// (e.g. `app_config` + `app_config_staging`) must leave both + /// keys readable from the local fastly.toml so the runtime + /// `EDGEZERO__STORES__CONFIG__APP_CONFIG__KEY` override can + /// switch between them. Prior to the upsert fix the second + /// push wholesale-replaced the per-store contents table. #[cfg(unix)] #[test] - fn push_config_entries_prior_read_failure_warns_no_deletes() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let _lock = path_mutation_guard().lock().expect("guard"); + fn push_config_entries_local_preserves_sibling_keys() { let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - // The FIRST describe (the pre-commit prior read) fails hard; the - // post-commit read-back (slot 2) still returns what we wrote, so the - // LWW guard passes and we reach the prior-read warning. - let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[String::new(), pointer], - None, - true, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - let out = FastlyCliAdapter - .push_config_entries( + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + let store = ResolvedStoreId::from_logical(TEST_CONFIG_ID); + let ctx = AdapterPushContext::new(); + + FastlyCliAdapter + .push_config_entries_local( dir.path(), + Some("fastly.toml"), None, + &store, + &[("app_config".to_owned(), "{\"envelope\":\"A\"}".to_owned())], + &ctx, + false, + ) + .expect("first push"); + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], - &AdapterPushContext::new(), + &store, + &[( + "app_config_staging".to_owned(), + "{\"envelope\":\"B\"}".to_owned(), + )], + &ctx, false, ) - .expect("push succeeds despite prior-read failure"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "prior-read failure => no deletes; log:\n{log}" - ); - assert!( - out.iter() - .any(|line| line.contains("failed to read prior root")), - "must warn about the failed prior read: {out:?}" - ); - } + .expect("second push (sibling key)"); - /// Schema drift must never echo the config payload. App config can hold - /// credentials; CLI status lines are logged verbatim and CI logs are - /// retained/shared. Only a size + field-name shape may be reported. - #[cfg(unix)] - #[test] - fn read_config_entry_schema_drift_does_not_leak_payload() { - const SENTINEL: &str = "SUPER_SECRET_TOKEN_abc123"; - let _lock = path_mutation_guard().lock().expect("guard"); - let dir = tempdir().expect("tempdir"); - // Valid JSON, but the value moved out of `item_value` (schema drift). - let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); - let fake = fake_fastly_returning(&drift, "", 0); - let _path = PathPrepend::new(fake.path()); - - let result = FastlyCliAdapter.read_config_entry( - dir.path(), - Some("fastly.toml"), - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - "cfg", - &AdapterPushContext::new(), - ); - let Err(err) = result else { - panic!("schema drift must error") - }; - assert!( - !err.contains(SENTINEL), - "error must not leak the config payload: {err}" - ); - assert!( - err.contains("bytes"), - "error should carry a redacted size/shape summary: {err}" + let raw = fs::read_to_string(&fastly_toml).expect("read"); + let doc: toml_edit::DocumentMut = raw.parse().expect("parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents after sibling push"); + let app_config = contents + .get("app_config") + .and_then(toml_edit::Item::as_str) + .expect("default key must survive sibling push"); + assert_eq!( + app_config, "{\"envelope\":\"A\"}", + "default key value: {raw}" ); + let staging = contents + .get("app_config_staging") + .and_then(toml_edit::Item::as_str) + .expect("staging key must be present"); + assert_eq!(staging, "{\"envelope\":\"B\"}", "staging key value: {raw}"); } - /// The FAILURE branch leaks too: a Fastly error that quotes the stored - /// value back in stderr must not reach the user-facing error. #[cfg(unix)] #[test] - fn read_config_entry_stderr_failure_does_not_leak_payload() { - const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr1"; - let _lock = path_mutation_guard().lock().expect("guard"); + fn push_config_entries_local_writes_literal_dotted_chunk_keys() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let dir = tempdir().expect("tempdir"); - // Not a "not found" — a hard failure that echoes the value. - let stderr = format!("Error: internal failure processing value {SENTINEL}"); - let fake = fake_fastly_returning("", &stderr, 1); - let _path = PathPrepend::new(fake.path()); + let fastly_toml = dir.path().join("fastly.toml"); + fs::write(&fastly_toml, "name = \"demo\"\n").expect("write"); - let result = FastlyCliAdapter.read_config_entry( - dir.path(), - Some("fastly.toml"), - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - "cfg", - &AdapterPushContext::new(), - ); - let Err(err) = result else { - panic!("hard stderr failure must error") - }; + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; + FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &entries, + &AdapterPushContext::new(), + false, + ) + .expect("local push must succeed"); + + let after = fs::read_to_string(&fastly_toml).expect("read back"); + // Chunk keys contain '.' and must appear as quoted string keys, + // not as TOML nested tables (which would look like [table.sub]). assert!( - !err.contains(SENTINEL), - "stderr must be redacted, not echoed: {err}" + after.contains(".__edgezero_chunks."), + "chunk keys written to fastly.toml: {after}" ); + // Parse with toml_edit and confirm chunk keys are string-keyed entries. + let doc: toml_edit::DocumentMut = after.parse().expect("must parse"); + let contents = doc + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .expect("contents table must exist"); + // At least one chunk key must be present as a string value (not a table). + let has_chunk_string = contents.as_table().is_some_and(|tbl| { + tbl.iter() + .any(|(key, val)| key.contains(".__edgezero_chunks.") && val.as_value().is_some()) + }); assert!( - err.contains("suppressed"), - "error should say the stderr was suppressed: {err}" + has_chunk_string, + "chunk keys must be literal string-valued entries, not nested tables: {after}" ); } - /// Same for the push path's GC prior-read, which runs on every cloud push. #[cfg(unix)] #[test] - fn push_config_entries_stderr_failure_does_not_leak_payload() { + fn push_config_entries_local_dry_run_reports_chunking_and_does_not_edit_fastly_toml() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - const SENTINEL: &str = "SUPER_SECRET_TOKEN_stderr2"; - let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let stderr = format!("Error: internal failure processing value {SENTINEL}"); - let fake = fake_fastly_returning("", &stderr, 1); - let _path = PathPrepend::new(fake.path()); + let fastly_toml = dir.path().join("fastly.toml"); + let original = "name = \"demo\"\n"; + fs::write(&fastly_toml, original).expect("write"); let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let result = FastlyCliAdapter.push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], - &AdapterPushContext::new(), - false, - ); - // Whether it errors or warns, the payload must never appear. - let rendered = match result { - Ok(lines) => lines.join("\n"), - Err(err) => err, - }; + let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; + let out = FastlyCliAdapter + .push_config_entries_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &entries, + &AdapterPushContext::new(), + true, // dry_run + ) + .expect("local dry-run must not error"); + + // File must be untouched. + let after = fs::read_to_string(&fastly_toml).expect("read back"); + assert_eq!(after, original, "dry-run must not edit fastly.toml"); + + // Output must describe chunking intent. + let combined = out.join("\n"); assert!( - !rendered.contains(SENTINEL), - "push output must not echo the stored value from stderr: {rendered}" + combined.contains("would set") && combined.contains("chunked"), + "must report chunked intent: {combined}" ); } - /// The GC prior-read runs on EVERY cloud push, so a drifted response must - /// not leak the previous config envelope into the push's status lines. + // ---------- chunked read integration tests ---------- + #[cfg(unix)] #[test] - fn push_config_entries_prior_read_drift_does_not_leak_payload() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - const SENTINEL: &str = "SUPER_SECRET_TOKEN_xyz789"; + fn read_config_entry_resolves_direct_value_unchanged() { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let drift = format!(r#"{{"value_moved_here":"{SENTINEL}"}}"#); - let fake = fake_fastly_returning(&drift, "", 0); + + let envelope = BlobEnvelope::new(json!({"hello": "world"}), "2026-06-22T00:00:00Z".into()); + let json_str = serde_json::to_string(&envelope).unwrap(); + let item_json = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(&json_str).unwrap() + ); + let fake = fake_fastly_returning(&item_json, "", 0); let _path = PathPrepend::new(fake.path()); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let out = FastlyCliAdapter - .push_config_entries( + let result = FastlyCliAdapter + .read_config_entry( dir.path(), - None, + Some("fastly.toml"), None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], + "cfg", &AdapterPushContext::new(), - false, ) - .expect("push still succeeds; the prior read only warns"); - for line in &out { - assert!( - !line.contains(SENTINEL), - "push status must not leak the prior config payload: {line}" - ); - } + .expect("read must succeed"); + let ReadConfigEntry::Present(value) = result else { + panic!("expected Present"); + }; + assert_eq!(value, json_str, "direct envelope passes through unchanged"); } - /// Every delete must pass `--key` + `--auto-yes` and NEVER `--all` - /// (which would wipe the whole store). Asserts the actual argv. #[cfg(unix)] #[test] - fn push_config_entries_delete_uses_key_and_auto_yes_never_all() { + fn read_config_entry_reconstructs_chunked_envelope() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); // force a reclamation so deletes occur let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let make = |tag: &str, stamp: &str| { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, stamp.to_owned())).expect("envelope") - }; - let envelope_a = make("gen_a", "2026-06-22T00:00:01Z"); - let envelope_b = make("gen_b", "2026-06-22T00:00:02Z"); - let envelope_c = make("gen_c", "2026-06-22T00:00:03Z"); - let gen_a = generation_of(TEST_CONFIG_ID, &envelope_a); - let (_b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let (_c_chunks, c_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_c); - let pending = pending_record_json(&gen_a, 0); - let fake = fake_fastly_gc_with_pending( - TEST_CONFIG_ID, - &[b_pointer, c_pointer], - Some(&pending), - None, - false, - &oplog, + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); + let (_, pointer_json) = physical.last().unwrap(); + // Build a key→response map for every physical entry. + let mut key_responses: Vec<(String, String)> = Vec::new(); + for (pk, pv) in &physical { + let resp = format!(r#"{{"item_value":{}}}"#, serde_json::to_string(pv).unwrap()); + key_responses.push((pk.clone(), resp)); + } + // The root key should return the pointer. + let ptr_resp = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(pointer_json).unwrap() ); + key_responses.push((TEST_CONFIG_ID.to_owned(), ptr_resp)); + + let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( + + let result = FastlyCliAdapter + .read_config_entry( dir.path(), - None, + Some("fastly.toml"), None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_c)], + TEST_CONFIG_ID, &AdapterPushContext::new(), - false, ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - let argv_lines: Vec<&str> = log - .lines() - .filter(|line| line.starts_with("delete-argv ")) - .collect(); - assert!( - !argv_lines.is_empty(), - "at least one delete happened: {log}" + .expect("chunked read must succeed"); + let ReadConfigEntry::Present(value) = result else { + panic!("expected Present"); + }; + assert_eq!( + value, envelope, + "reconstructed envelope must equal original" + ); + } + + #[cfg(unix)] + #[test] + fn read_config_entry_errors_on_missing_chunk() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let dir = tempdir().expect("tempdir"); + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); + let (_, pointer_json) = physical.last().unwrap(); + // Only provide the root pointer; omit chunk responses so chunk fetch returns not-found. + let ptr_resp = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(pointer_json).unwrap() + ); + let key_responses = vec![(TEST_CONFIG_ID.to_owned(), ptr_resp)]; + let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); + let _path = PathPrepend::new(fake.path()); + + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + TEST_CONFIG_ID, + &AdapterPushContext::new(), + ); + let Err(err) = result else { + panic!("missing chunk must error") + }; + assert!( + err.contains("missing chunk"), + "error must mention missing chunk: {err}" ); - for line in argv_lines { - assert!( - line.contains("--auto-yes"), - "delete must pass --auto-yes: {line}" - ); - assert!( - line.contains("--key="), - "delete must target a --key: {line}" - ); - assert!( - !line.contains("--all"), - "delete must NEVER pass --all (store-wide wipe): {line}" - ); - } } - /// Cloud config shrinking back under the limit: the new value is a - /// direct envelope, so every prior chunk is reclaimed and the root is - /// upserted (never deleted). #[cfg(unix)] #[test] - fn push_config_entries_shrink_to_direct_deletes_all_prior_chunks() { + fn read_config_entry_errors_on_corrupt_chunk_hash() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let chunked = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); - let (prior_chunks, prior_pointer) = chunked_parts(TEST_CONFIG_ID, &chunked); - // New value is direct, so new_root_value == the direct envelope; - // read-back returns it so the guard passes. - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[prior_pointer, direct.clone()], - None, - false, - &oplog, + + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); + let (_, pointer_json) = physical.last().unwrap(); + let mut key_responses: Vec<(String, String)> = Vec::new(); + // Corrupt first chunk's content. + let (first_chunk_key, first_chunk_val) = &physical[0]; + let corrupted: String = first_chunk_val.chars().map(|_| 'Z').collect(); + let corrupt_resp = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(&corrupted).unwrap() ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), direct)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - // Shrinking to a direct value orphans the whole prior chunk set — but - // it is DEFERRED, not deleted now: POPs may still serve the old pointer. - for key in &prior_chunks { - assert!( - !oplog_has(&oplog, &format!("delete {key}")), - "prior chunk `{key}` must be deferred, not deleted eagerly" - ); + key_responses.push((first_chunk_key.clone(), corrupt_resp)); + // Remaining chunks as normal. + for (pk, pv) in physical + .iter() + .take(physical.len().saturating_sub(1)) + .skip(1) + { + key_responses.push(( + pk.clone(), + format!(r#"{{"item_value":{}}}"#, serde_json::to_string(pv).unwrap()), + )); } - assert!( - oplog_has( - &oplog, - &format!("update {}", gc_pending_key(TEST_CONFIG_ID)) + key_responses.push(( + TEST_CONFIG_ID.to_owned(), + format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(pointer_json).unwrap() ), - "the orphaned generation must be recorded as pending" - ); - assert!( - !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), - "root pointer must never be deleted" + )); + let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); + let _path = PathPrepend::new(fake.path()); + + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + TEST_CONFIG_ID, + &AdapterPushContext::new(), ); + let Err(err) = result else { + panic!("corrupt chunk must error") + }; assert!( - oplog_has(&oplog, &format!("update {TEST_CONFIG_ID}")), - "root must be upserted with the direct value" + err.contains("SHA mismatch") || err.contains("mismatch"), + "error must mention hash mismatch: {err}" ); } #[cfg(unix)] #[test] - fn push_config_entries_writes_direct_entry_at_exactly_8000_chars() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn read_config_entry_errors_on_malformed_pointer() { let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); - let argv_log = dir.path().join("argv.txt"); - let fake = fake_fastly_argv_log(&argv_log); + // Root value is JSON but neither a BlobEnvelope nor a valid pointer. + let bad_json = r#"{"some_field":"not a pointer or envelope"}"#; + let item_json = format!( + r#"{{"item_value":{}}}"#, + serde_json::to_string(bad_json).unwrap() + ); + let fake = fake_fastly_returning(&item_json, "", 0); let _path = PathPrepend::new(fake.path()); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); - assert_eq!(envelope.len(), FASTLY_CONFIG_ENTRY_LIMIT); - - let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &entries, - &AdapterPushContext::new(), - false, - ) - .expect("push must succeed"); - // One physical entry written (direct). - let captured = fs::read_to_string(&argv_log).expect("argv log"); - assert!( - captured.contains(&format!("--key={TEST_CONFIG_ID}")), - "must write root key directly: {captured}" + let result = FastlyCliAdapter.read_config_entry( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + "cfg", + &AdapterPushContext::new(), ); + let Err(err) = result else { + panic!("malformed pointer must error") + }; assert!( - out[0].contains("1 physical entries (1 logical)"), - "summary reports 1 physical entry: {out:?}" + err.contains("neither a valid BlobEnvelope") || err.contains("chunk pointer"), + "error must describe parse failure: {err}" ); } - #[cfg(unix)] + // ---------- local read integration tests ---------- + #[test] - fn push_config_entries_writes_chunks_and_root_pointer_for_8001_chars() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let _lock = path_mutation_guard().lock().expect("guard"); + fn read_config_entry_local_resolves_direct_value() { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; let dir = tempdir().expect("tempdir"); - let argv_log = dir.path().join("argv.txt"); - let fake = fake_fastly_argv_log(&argv_log); - let _path = PathPrepend::new(fake.path()); + let fastly_toml = dir.path().join("fastly.toml"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - assert!(envelope.len() > FASTLY_CONFIG_ENTRY_LIMIT); + let envelope = BlobEnvelope::new(json!({"x": 1_i32}), "2026-06-22T00:00:00Z".into()); + let json_str = serde_json::to_string(&envelope).unwrap(); + // Write directly as a single entry (not via push_config_entries_local so we + // control the exact TOML content). + write_fastly_local_config_store( + &fastly_toml, + TEST_CONFIG_ID, + &[("cfg".to_owned(), json_str.clone())], + &[], + ) + .expect("write"); - let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; - let out = FastlyCliAdapter - .push_config_entries( + let result = FastlyCliAdapter + .read_config_entry_local( dir.path(), - None, + Some("fastly.toml"), None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &entries, + "cfg", &AdapterPushContext::new(), - false, ) - .expect("push must succeed"); - let captured = fs::read_to_string(&argv_log).expect("argv log"); - // At least one chunk key must appear before the root key. - assert!( - captured.contains(".__edgezero_chunks."), - "chunk keys must be written: {captured}" - ); - // Root pointer must also be written. - assert!( - captured.contains(&format!("--key={TEST_CONFIG_ID}")), - "root pointer must be written: {captured}" - ); - // Root key must be LAST in the log (chunk lines come before it). - let root_pos = captured.rfind(&format!("--key={TEST_CONFIG_ID}")).unwrap(); - let chunk_pos = captured.find(".__edgezero_chunks.").unwrap(); - assert!( - chunk_pos < root_pos, - "chunk writes must precede root pointer write: chunk_pos={chunk_pos} root_pos={root_pos}" - ); - assert!(out[0].contains("logical"), "summary line present: {out:?}"); + .expect("local read must succeed"); + let ReadConfigEntry::Present(value) = result else { + panic!("expected Present"); + }; + assert_eq!(value, json_str, "direct envelope passes through unchanged"); } - #[cfg(unix)] #[test] - fn push_config_entries_dry_run_reports_direct_vs_chunked() { + fn read_config_entry_local_reconstructs_chunked_envelope() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let dir = tempdir().expect("tempdir"); + let fastly_toml = dir.path().join("fastly.toml"); - let direct_envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); - let chunked_envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); + let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); + // Write all physical entries (chunks + pointer) to the local store. + write_fastly_local_config_store(&fastly_toml, TEST_CONFIG_ID, &physical, &[]) + .expect("write"); - let entries = vec![ - ("cfg_direct".to_owned(), direct_envelope), - ("cfg_chunked".to_owned(), chunked_envelope), - ]; - let out = FastlyCliAdapter - .push_config_entries( + let result = FastlyCliAdapter + .read_config_entry_local( dir.path(), - None, + Some("fastly.toml"), None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &entries, + TEST_CONFIG_ID, &AdapterPushContext::new(), - true, // dry_run ) - .expect("dry-run must not error"); - - // No shellout happens; output must describe intent. - let combined = out.join("\n"); - assert!( - combined.contains("would push `cfg_direct` as direct entry"), - "must report direct: {combined}" - ); - assert!( - combined.contains("would push `cfg_chunked` as chunked"), - "must report chunked: {combined}" + .expect("local chunked read must succeed"); + let ReadConfigEntry::Present(value) = result else { + panic!("expected Present"); + }; + assert_eq!( + value, envelope, + "reconstructed envelope must equal original" ); } - /// Spec 12.7: pushing two blobs under different root keys - /// (e.g. `app_config` + `app_config_staging`) must leave both - /// keys readable from the local fastly.toml so the runtime - /// `EDGEZERO__STORES__CONFIG__APP_CONFIG__KEY` override can - /// switch between them. Prior to the upsert fix the second - /// push wholesale-replaced the per-store contents table. + /// Spec 12.3 + 9.3: a second oversized push must converge the + /// runtime on the NEW envelope — chunk keys are content-addressed + /// by the full-envelope SHA, so push B writes a new chunk-set and + /// installs a new root pointer. + /// + /// The local fastly.toml writer upserts per-key (so a sibling + /// `--key app_config_staging` push leaves `app_config` intact per + /// spec 12.7). Within the SAME root key, GC on re-push prunes the + /// prior generation: after envelope B's push, envelope A's chunks — + /// now unreferenced by the `app_config` pointer — are removed from + /// the contents table. A read after push B follows the active + /// pointer and reconstructs envelope B, not A. #[cfg(unix)] #[test] - fn push_config_entries_local_preserves_sibling_keys() { + #[expect( + clippy::too_many_lines, + reason = "linear test scenario: push A, inspect, push B, inspect, read; splitting would obscure the chunk-set comparison" + )] + fn second_oversized_push_converges_runtime_on_new_envelope() { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; let dir = tempdir().expect("tempdir"); let fastly_toml = dir.path().join("fastly.toml"); fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); - let store = ResolvedStoreId::from_logical(TEST_CONFIG_ID); - let ctx = AdapterPushContext::new(); + // First push: envelope A. Records the chunk-key set so we can + // confirm they are pruned by the second push's GC. + let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); FastlyCliAdapter .push_config_entries_local( dir.path(), Some("fastly.toml"), None, - &store, - &[("app_config".to_owned(), "{\"envelope\":\"A\"}".to_owned())], - &ctx, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_a.clone())], + &AdapterPushContext::new(), false, ) - .expect("first push"); + .expect("first push must succeed"); + + let after_a = fs::read_to_string(&fastly_toml).expect("read"); + let doc_a: toml_edit::DocumentMut = after_a.parse().expect("parse"); + let contents_a = doc_a + .get("local_server") + .and_then(|ls| ls.get("config_stores")) + .and_then(|cs| cs.get(TEST_CONFIG_ID)) + .and_then(|st| st.get("contents")) + .and_then(toml_edit::Item::as_table) + .expect("contents table after push A"); + let chunks_a: Vec = contents_a + .iter() + .map(|(key, _)| key.to_owned()) + .filter(|key| key.contains(".__edgezero_chunks.")) + .collect(); + assert!( + !chunks_a.is_empty(), + "push A must have produced chunk entries: {after_a}" + ); + + // Second push: a DIFFERENT oversized envelope B. The + // content-addressed chunk keys must shift to B's sha; GC then + // prunes the old A-chunks. Build envelope B with a distinct + // payload key so its SHA differs from A's even at the same + // total length. + let envelope_b = { + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ "alt": "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:01Z".to_owned())) + .expect("envelope B serialises") + }; + assert_ne!(envelope_a, envelope_b, "test fixtures must differ"); FastlyCliAdapter .push_config_entries_local( dir.path(), Some("fastly.toml"), None, - &store, - &[( - "app_config_staging".to_owned(), - "{\"envelope\":\"B\"}".to_owned(), - )], - &ctx, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b.clone())], + &AdapterPushContext::new(), false, ) - .expect("second push (sibling key)"); + .expect("second push must succeed"); - let raw = fs::read_to_string(&fastly_toml).expect("read"); - let doc: toml_edit::DocumentMut = raw.parse().expect("parse"); - let contents = doc + let after_b = fs::read_to_string(&fastly_toml).expect("read"); + let doc_b: toml_edit::DocumentMut = after_b.parse().expect("parse"); + let contents_b = doc_b .get("local_server") .and_then(|ls| ls.get("config_stores")) .and_then(|cs| cs.get(TEST_CONFIG_ID)) .and_then(|st| st.get("contents")) .and_then(toml_edit::Item::as_table) - .expect("contents after sibling push"); - let app_config = contents - .get("app_config") - .and_then(toml_edit::Item::as_str) - .expect("default key must survive sibling push"); + .expect("contents table after push B"); + let chunks_b: Vec = contents_b + .iter() + .map(|(key, _)| key.to_owned()) + .filter(|key| key.contains(".__edgezero_chunks.")) + .collect(); + assert!( + !chunks_b.is_empty(), + "push B must have produced chunk entries: {after_b}" + ); + + // Chunk keys are content-addressed by envelope SHA, so the B + // push installs a fresh chunk-set whose keys are all distinct + // from A's. GC on re-push prunes the now-unreferenced A-chunks. + let new_b_chunks: Vec<&String> = chunks_b + .iter() + .filter(|key| !chunks_a.contains(*key)) + .collect(); + assert!( + !new_b_chunks.is_empty(), + "push B must have added at least one new content-addressed chunk: A-set={chunks_a:?} B-set={chunks_b:?}" + ); + // Old A-chunks are pruned: GC deletes the prior generation the + // old pointer referenced once B's pointer supersedes it. + for chunk_key in &chunks_a { + assert!( + !chunks_b.contains(chunk_key), + "old A-chunk `{chunk_key}` must be pruned from the local table after push B; B-set={chunks_b:?}" + ); + } + + // Runtime-correctness property: a fresh read after push B + // reconstructs envelope B (NOT envelope A). + let read = FastlyCliAdapter + .read_config_entry_local( + dir.path(), + Some("fastly.toml"), + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + TEST_CONFIG_ID, + &AdapterPushContext::new(), + ) + .expect("local read after push B"); + let ReadConfigEntry::Present(value) = read else { + panic!("expected Present after push B"); + }; assert_eq!( - app_config, "{\"envelope\":\"A\"}", - "default key value: {raw}" + value, envelope_b, + "read after second push must reconstruct envelope B, not A" + ); + assert_ne!( + value, envelope_a, + "old envelope A's chunks must be inert -- read must NOT return A" ); - let staging = contents - .get("app_config_staging") - .and_then(toml_edit::Item::as_str) - .expect("staging key must be present"); - assert_eq!(staging, "{\"envelope\":\"B\"}", "staging key value: {raw}"); } + // ---------- cloud chunk reclamation (store-derived) ---------- + + /// Envelope factory with a distinct payload per tag. #[cfg(unix)] - #[test] - fn push_config_entries_local_writes_literal_dotted_chunk_keys() { + fn gen_envelope(tag: &str) -> String { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:00Z".to_owned())) + .expect("envelope") + } + + /// THE eventual-consistency guarantee: the generation this push just + /// superseded is NEVER deleted, even with a zero grace window — POPs may + /// still be serving the pointer that references it. + #[cfg(unix)] + #[test] + fn push_config_entries_never_reclaims_the_just_superseded_generation() { + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); // even so, the prior generation is safe let dir = tempdir().expect("tempdir"); - let fastly_toml = dir.path().join("fastly.toml"); - fs::write(&fastly_toml, "name = \"demo\"\n").expect("write"); + let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + + // The store holds A (long-lived) and, post-commit, B. + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_a, 100_000); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer, b_pointer], + &listing, + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); FastlyCliAdapter - .push_config_entries_local( + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &entries, + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), false, ) - .expect("local push must succeed"); + .expect("push succeeds"); - let after = fs::read_to_string(&fastly_toml).expect("read back"); - // Chunk keys contain '.' and must appear as quoted string keys, - // not as TOML nested tables (which would look like [table.sub]). - assert!( - after.contains(".__edgezero_chunks."), - "chunk keys written to fastly.toml: {after}" - ); - // Parse with toml_edit and confirm chunk keys are string-keyed entries. - let doc: toml_edit::DocumentMut = after.parse().expect("must parse"); - let contents = doc - .get("local_server") - .and_then(|ls| ls.get("config_stores")) - .and_then(|cs| cs.get(TEST_CONFIG_ID)) - .and_then(|st| st.get("contents")) - .expect("contents table must exist"); - // At least one chunk key must be present as a string value (not a table). - let has_chunk_string = contents.as_table().is_some_and(|tbl| { - tbl.iter() - .any(|(key, val)| key.contains(".__edgezero_chunks.") && val.as_value().is_some()) - }); - assert!( - has_chunk_string, - "chunk keys must be literal string-valued entries, not nested tables: {after}" - ); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + for key in a_chunks.iter().chain(b_chunks.iter()) { + assert!( + !oplog_has(&oplog, &format!("delete {key}")), + "must not delete the just-superseded (or new) generation: `{key}`; log:\n{log}" + ); + } } + /// An OLDER orphan generation — superseded before the prior one, and aged + /// past the grace window — IS reclaimed. #[cfg(unix)] #[test] - fn push_config_entries_local_dry_run_reports_chunking_and_does_not_edit_fastly_toml() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn push_config_entries_reclaims_aged_orphan_generation() { + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("3600"); // 1h let dir = tempdir().expect("tempdir"); - let fastly_toml = dir.path().join("fastly.toml"); - let original = "name = \"demo\"\n"; - fs::write(&fastly_toml, original).expect("write"); + let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let entries = vec![(TEST_CONFIG_ID.to_owned(), envelope)]; - let out = FastlyCliAdapter - .push_config_entries_local( + let envelope_x = gen_envelope("gen_x"); // long-dead orphan + let envelope_a = gen_envelope("gen_a"); // prior (live before this push) + let envelope_b = gen_envelope("gen_b"); // what we push now + let (x_chunks, _x_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_x); + let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + + // X written 3 days ago; A written 2 days ago (so X was superseded 2 days + // ago -> well past the 1h grace); B written now. + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer, b_pointer], + &listing, + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &entries, + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), - true, // dry_run + false, ) - .expect("local dry-run must not error"); - - // File must be untouched. - let after = fs::read_to_string(&fastly_toml).expect("read back"); - assert_eq!(after, original, "dry-run must not edit fastly.toml"); + .expect("push succeeds"); - // Output must describe chunking intent. - let combined = out.join("\n"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + for key in &x_chunks { + assert!( + oplog_has(&oplog, &format!("delete {key}")), + "aged orphan `{key}` must be reclaimed; log:\n{log}" + ); + } + for key in a_chunks.iter().chain(b_chunks.iter()) { + assert!( + !oplog_has(&oplog, &format!("delete {key}")), + "live/just-superseded chunk `{key}` must survive" + ); + } assert!( - combined.contains("would set") && combined.contains("chunked"), - "must report chunked intent: {combined}" + !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), + "root pointer must never be deleted" ); + // Reclamation follows the commit. + let root_update = log + .lines() + .position(|line| line == format!("update {TEST_CONFIG_ID}")) + .expect("root update logged"); + let first_delete = log + .lines() + .position(|line| line.starts_with("delete ")) + .expect("a delete logged"); + assert!(first_delete > root_update, "deletes follow the commit"); } - // ---------- chunked read integration tests ---------- - + /// An orphan still inside the grace window is retained. #[cfg(unix)] #[test] - fn read_config_entry_resolves_direct_value_unchanged() { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; + fn push_config_entries_retains_orphan_within_grace() { let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("86400"); // 24h let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); - let envelope = BlobEnvelope::new(json!({"hello": "world"}), "2026-06-22T00:00:00Z".into()); - let json_str = serde_json::to_string(&envelope).unwrap(); - let item_json = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(&json_str).unwrap() + let envelope_x = gen_envelope("gen_x"); + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (x_chunks, _x) = chunked_parts(TEST_CONFIG_ID, &envelope_x); + let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + + // X superseded only 1h ago (when A was written) -> inside a 24h grace. + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 7_200); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 3_600)); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer, b_pointer], + &listing, + None, + false, + &oplog, ); - let fake = fake_fastly_returning(&item_json, "", 0); let _path = PathPrepend::new(fake.path()); - - let result = FastlyCliAdapter - .read_config_entry( + FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - "cfg", + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), + false, ) - .expect("read must succeed"); - let ReadConfigEntry::Present(value) = result else { - panic!("expected Present"); - }; - assert_eq!(value, json_str, "direct envelope passes through unchanged"); + .expect("push succeeds"); + + let log = fs::read_to_string(&oplog).unwrap_or_default(); + for key in &x_chunks { + assert!( + !oplog_has(&oplog, &format!("delete {key}")), + "orphan `{key}` is inside the grace window and must be retained; log:\n{log}" + ); + } } + /// Reclamation yields when a concurrent push has superseded us. #[cfg(unix)] #[test] - fn read_config_entry_reconstructs_chunked_envelope() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn push_config_entries_skips_gc_when_root_changed() { let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); - let (_, pointer_json) = physical.last().unwrap(); - // Build a key→response map for every physical entry. - let mut key_responses: Vec<(String, String)> = Vec::new(); - for (pk, pv) in &physical { - let resp = format!(r#"{{"item_value":{}}}"#, serde_json::to_string(pv).unwrap()); - key_responses.push((pk.clone(), resp)); - } - // The root key should return the pointer. - let ptr_resp = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(pointer_json).unwrap() - ); - key_responses.push((TEST_CONFIG_ID.to_owned(), ptr_resp)); + let envelope_x = gen_envelope("gen_x"); + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (_x, _xp) = chunked_parts(TEST_CONFIG_ID, &envelope_x); + let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); - let _path = PathPrepend::new(fake.path()); + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); - let result = FastlyCliAdapter - .read_config_entry( + // The read-back returns the OLD pointer: someone else won. + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer.clone(), a_pointer], + &listing, + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - TEST_CONFIG_ID, + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), + false, ) - .expect("chunked read must succeed"); - let ReadConfigEntry::Present(value) = result else { - panic!("expected Present"); - }; - assert_eq!( - value, envelope, - "reconstructed envelope must equal original" + .expect("push succeeds"); + + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "no deletes when the root changed under us; log:\n{log}" + ); + assert!( + out.iter().any(|line| line.contains("root changed")), + "must warn that reclamation was skipped: {out:?}" ); } + /// Every reclamation delete must pass `--key` + `--auto-yes` and NEVER + /// `--all` (which would wipe the whole store). #[cfg(unix)] #[test] - fn read_config_entry_errors_on_missing_chunk() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn push_config_entries_delete_uses_key_and_auto_yes_never_all() { let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("3600"); let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); - let (_, pointer_json) = physical.last().unwrap(); - // Only provide the root pointer; omit chunk responses so chunk fetch returns not-found. - let ptr_resp = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(pointer_json).unwrap() - ); - let key_responses = vec![(TEST_CONFIG_ID.to_owned(), ptr_resp)]; - let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); - let _path = PathPrepend::new(fake.path()); + let envelope_x = gen_envelope("gen_x"); + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let result = FastlyCliAdapter.read_config_entry( - dir.path(), - Some("fastly.toml"), - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + + let fake = fake_fastly_gc( TEST_CONFIG_ID, - &AdapterPushContext::new(), - ); - let Err(err) = result else { - panic!("missing chunk must error") - }; - assert!( - err.contains("missing chunk"), - "error must mention missing chunk: {err}" + &[a_pointer, b_pointer], + &listing, + None, + false, + &oplog, ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + + let log = fs::read_to_string(&oplog).unwrap_or_default(); + let argv_lines: Vec<&str> = log + .lines() + .filter(|line| line.starts_with("delete-argv ")) + .collect(); + assert!(!argv_lines.is_empty(), "a reclamation happened: {log}"); + for line in argv_lines { + assert!( + line.contains("--auto-yes"), + "delete passes --auto-yes: {line}" + ); + assert!(line.contains("--key="), "delete targets a --key: {line}"); + assert!( + !line.contains("--all"), + "delete must NEVER pass --all (store-wide wipe): {line}" + ); + } } + /// A failed reclamation delete warns; the push still succeeds. #[cfg(unix)] #[test] - fn read_config_entry_errors_on_corrupt_chunk_hash() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn push_config_entries_delete_failure_warns_but_succeeds() { let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("3600"); let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); - let (_, pointer_json) = physical.last().unwrap(); - let mut key_responses: Vec<(String, String)> = Vec::new(); - // Corrupt first chunk's content. - let (first_chunk_key, first_chunk_val) = &physical[0]; - let corrupted: String = first_chunk_val.chars().map(|_| 'Z').collect(); - let corrupt_resp = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(&corrupted).unwrap() - ); - key_responses.push((first_chunk_key.clone(), corrupt_resp)); - // Remaining chunks as normal. - for (pk, pv) in physical - .iter() - .take(physical.len().saturating_sub(1)) - .skip(1) - { - key_responses.push(( - pk.clone(), - format!(r#"{{"item_value":{}}}"#, serde_json::to_string(pv).unwrap()), - )); - } - key_responses.push(( - TEST_CONFIG_ID.to_owned(), - format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(pointer_json).unwrap() - ), - )); - let fake = fake_fastly_with_key_dispatch(dir.path(), &key_responses); - let _path = PathPrepend::new(fake.path()); + let envelope_x = gen_envelope("gen_x"); + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (x_chunks, _x) = chunked_parts(TEST_CONFIG_ID, &envelope_x); + let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let fail_key = x_chunks.first().expect("a chunk").clone(); - let result = FastlyCliAdapter.read_config_entry( - dir.path(), - Some("fastly.toml"), - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); + listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + + let fake = fake_fastly_gc( TEST_CONFIG_ID, - &AdapterPushContext::new(), + &[a_pointer, b_pointer], + &listing, + Some(&fail_key), + false, + &oplog, ); - let Err(err) = result else { - panic!("corrupt chunk must error") - }; + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push still succeeds despite a failed reclamation delete"); assert!( - err.contains("SHA mismatch") || err.contains("mismatch"), - "error must mention hash mismatch: {err}" + out.iter() + .any(|line| line.contains("could not reclaim orphan chunk") + && line.contains(&fail_key)), + "failed delete surfaces a warning: {out:?}" ); } + /// A suspicious prior pointer skips reclamation (we cannot tell what is + /// live), warns, and deletes nothing. #[cfg(unix)] #[test] - fn read_config_entry_errors_on_malformed_pointer() { + fn push_config_entries_suspicious_prior_warns_no_deletes() { let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); - // Root value is JSON but neither a BlobEnvelope nor a valid pointer. - let bad_json = r#"{"some_field":"not a pointer or envelope"}"#; - let item_json = format!( - r#"{{"item_value":{}}}"#, - serde_json::to_string(bad_json).unwrap() - ); - let fake = fake_fastly_returning(&item_json, "", 0); - let _path = PathPrepend::new(fake.path()); + let oplog = dir.path().join("ops.log"); - let result = FastlyCliAdapter.read_config_entry( - dir.path(), - Some("fastly.toml"), + let envelope_x = gen_envelope("gen_x"); + let envelope_b = gen_envelope("gen_b"); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); + + let listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[bad, b_pointer], + &listing, None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - "cfg", - &AdapterPushContext::new(), + false, + &oplog, ); - let Err(err) = result else { - panic!("malformed pointer must error") - }; + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( + dir.path(), + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], + &AdapterPushContext::new(), + false, + ) + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( - err.contains("neither a valid BlobEnvelope") || err.contains("chunk pointer"), - "error must describe parse failure: {err}" + !log.lines().any(|line| line.starts_with("delete ")), + "suspicious prior => no deletes; log:\n{log}" + ); + assert!( + out.iter().any(|line| line.contains("skipping chunk GC")), + "warns about the suspicious pointer: {out:?}" ); } - // ---------- local read integration tests ---------- - + /// A hard prior-read failure skips reclamation with a warning. + #[cfg(unix)] #[test] - fn read_config_entry_local_resolves_direct_value() { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; + fn push_config_entries_prior_read_failure_warns_no_deletes() { + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); - let fastly_toml = dir.path().join("fastly.toml"); + let oplog = dir.path().join("ops.log"); - let envelope = BlobEnvelope::new(json!({"x": 1_i32}), "2026-06-22T00:00:00Z".into()); - let json_str = serde_json::to_string(&envelope).unwrap(); - // Write directly as a single entry (not via push_config_entries_local so we - // control the exact TOML content). - write_fastly_local_config_store( - &fastly_toml, - TEST_CONFIG_ID, - &[("cfg".to_owned(), json_str.clone())], - &[], - ) - .expect("write"); + let envelope_x = gen_envelope("gen_x"); + let envelope_b = gen_envelope("gen_b"); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - let result = FastlyCliAdapter - .read_config_entry_local( + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[String::new(), b_pointer], + &listing, + None, + true, // first describe of each key fails hard + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + let out = FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - "cfg", + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), + false, ) - .expect("local read must succeed"); - let ReadConfigEntry::Present(value) = result else { - panic!("expected Present"); - }; - assert_eq!(value, json_str, "direct envelope passes through unchanged"); + .expect("push succeeds despite a prior-read failure"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "prior-read failure => no deletes; log:\n{log}" + ); + assert!( + out.iter() + .any(|line| line.contains("failed to read prior root")), + "must warn about the failed prior read: {out:?}" + ); } + /// Dry-run stays fully offline and reports reclamation intent. + #[cfg(unix)] #[test] - fn read_config_entry_local_reconstructs_chunked_envelope() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn push_config_entries_dry_run_reports_gc_intent_offline() { let dir = tempdir().expect("tempdir"); - let fastly_toml = dir.path().join("fastly.toml"); - - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let physical = prepare_fastly_config_entries(TEST_CONFIG_ID, &envelope).unwrap(); - // Write all physical entries (chunks + pointer) to the local store. - write_fastly_local_config_store(&fastly_toml, TEST_CONFIG_ID, &physical, &[]) - .expect("write"); - - let result = FastlyCliAdapter - .read_config_entry_local( + let envelope = gen_envelope("gen_a"); + // No fake on PATH: a dry-run must not shell out at all. + let out = FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - TEST_CONFIG_ID, + &[(TEST_CONFIG_ID.to_owned(), envelope)], &AdapterPushContext::new(), + true, ) - .expect("local chunked read must succeed"); - let ReadConfigEntry::Present(value) = result else { - panic!("expected Present"); - }; - assert_eq!( - value, envelope, - "reconstructed envelope must equal original" + .expect("dry-run must not error"); + assert!( + out.iter() + .any(|line| line.contains("would defer reclamation")), + "dry-run reports deferred-reclamation intent: {out:?}" ); } - /// Spec 12.3 + 9.3: a second oversized push must converge the - /// runtime on the NEW envelope — chunk keys are content-addressed - /// by the full-envelope SHA, so push B writes a new chunk-set and - /// installs a new root pointer. - /// - /// The local fastly.toml writer upserts per-key (so a sibling - /// `--key app_config_staging` push leaves `app_config` intact per - /// spec 12.7). Within the SAME root key, GC on re-push prunes the - /// prior generation: after envelope B's push, envelope A's chunks — - /// now unreferenced by the `app_config` pointer — are removed from - /// the contents table. A read after push B follows the active - /// pointer and reconstructs envelope B, not A. + /// Shrinking to a direct value orphans the whole chunk set — but the + /// just-superseded generation is still protected. #[cfg(unix)] #[test] - #[expect( - clippy::too_many_lines, - reason = "linear test scenario: push A, inspect, push B, inspect, read; splitting would obscure the chunk-set comparison" - )] - fn second_oversized_push_converges_runtime_on_new_envelope() { + fn push_config_entries_shrink_to_direct_defers_prior_chunks() { use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); - let fastly_toml = dir.path().join("fastly.toml"); - fs::write(&fastly_toml, "name = \"demo\"\n").expect("seed"); + let oplog = dir.path().join("ops.log"); - // First push: envelope A. Records the chunk-key set so we can - // confirm they are pruned by the second push's GC. - let envelope_a = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - FastlyCliAdapter - .push_config_entries_local( - dir.path(), - Some("fastly.toml"), - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_a.clone())], - &AdapterPushContext::new(), - false, - ) - .expect("first push must succeed"); + let chunked = gen_envelope("gen_a"); + let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); + let (prior_chunks, prior_pointer) = chunked_parts(TEST_CONFIG_ID, &chunked); + let listing = listed_generation(TEST_CONFIG_ID, &chunked, 172_800); - let after_a = fs::read_to_string(&fastly_toml).expect("read"); - let doc_a: toml_edit::DocumentMut = after_a.parse().expect("parse"); - let contents_a = doc_a - .get("local_server") - .and_then(|ls| ls.get("config_stores")) - .and_then(|cs| cs.get(TEST_CONFIG_ID)) - .and_then(|st| st.get("contents")) - .and_then(toml_edit::Item::as_table) - .expect("contents table after push A"); - let chunks_a: Vec = contents_a - .iter() - .map(|(key, _)| key.to_owned()) - .filter(|key| key.contains(".__edgezero_chunks.")) - .collect(); - assert!( - !chunks_a.is_empty(), - "push A must have produced chunk entries: {after_a}" + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[prior_pointer, direct.clone()], + &listing, + None, + false, + &oplog, ); - - // Second push: a DIFFERENT oversized envelope B. The - // content-addressed chunk keys must shift to B's sha; GC then - // prunes the old A-chunks. Build envelope B with a distinct - // payload key so its SHA differs from A's even at the same - // total length. - let envelope_b = { - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ "alt": "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:01Z".to_owned())) - .expect("envelope B serialises") - }; - assert_ne!(envelope_a, envelope_b, "test fixtures must differ"); + let _path = PathPrepend::new(fake.path()); FastlyCliAdapter - .push_config_entries_local( + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b.clone())], + &[(TEST_CONFIG_ID.to_owned(), direct)], &AdapterPushContext::new(), false, ) - .expect("second push must succeed"); - - let after_b = fs::read_to_string(&fastly_toml).expect("read"); - let doc_b: toml_edit::DocumentMut = after_b.parse().expect("parse"); - let contents_b = doc_b - .get("local_server") - .and_then(|ls| ls.get("config_stores")) - .and_then(|cs| cs.get(TEST_CONFIG_ID)) - .and_then(|st| st.get("contents")) - .and_then(toml_edit::Item::as_table) - .expect("contents table after push B"); - let chunks_b: Vec = contents_b - .iter() - .map(|(key, _)| key.to_owned()) - .filter(|key| key.contains(".__edgezero_chunks.")) - .collect(); - assert!( - !chunks_b.is_empty(), - "push B must have produced chunk entries: {after_b}" - ); + .expect("push succeeds"); - // Chunk keys are content-addressed by envelope SHA, so the B - // push installs a fresh chunk-set whose keys are all distinct - // from A's. GC on re-push prunes the now-unreferenced A-chunks. - let new_b_chunks: Vec<&String> = chunks_b - .iter() - .filter(|key| !chunks_a.contains(*key)) - .collect(); - assert!( - !new_b_chunks.is_empty(), - "push B must have added at least one new content-addressed chunk: A-set={chunks_a:?} B-set={chunks_b:?}" - ); - // Old A-chunks are pruned: GC deletes the prior generation the - // old pointer referenced once B's pointer supersedes it. - for chunk_key in &chunks_a { + let log = fs::read_to_string(&oplog).unwrap_or_default(); + for key in &prior_chunks { assert!( - !chunks_b.contains(chunk_key), - "old A-chunk `{chunk_key}` must be pruned from the local table after push B; B-set={chunks_b:?}" + !oplog_has(&oplog, &format!("delete {key}")), + "the just-superseded generation must be retained even on shrink-to-direct: `{key}`; log:\n{log}" ); } + assert!( + oplog_has(&oplog, &format!("update {TEST_CONFIG_ID}")), + "root upserted with the direct value" + ); + } - // Runtime-correctness property: a fresh read after push B - // reconstructs envelope B (NOT envelope A). - let read = FastlyCliAdapter - .read_config_entry_local( + /// A failing `config-store-entry list` degrades to a warning; the push + /// still succeeds and nothing is deleted. + #[cfg(unix)] + #[test] + fn push_config_entries_list_failure_warns_no_deletes() { + let _lock = path_mutation_guard().lock().expect("guard"); + let _grace = GraceGuard::set("0"); + let dir = tempdir().expect("tempdir"); + let oplog = dir.path().join("ops.log"); + let envelope_a = gen_envelope("gen_a"); + let envelope_b = gen_envelope("gen_b"); + let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + + // An empty listing means nothing can be grouped -> nothing reclaimed. + let fake = fake_fastly_gc( + TEST_CONFIG_ID, + &[a_pointer, b_pointer], + &[], + None, + false, + &oplog, + ); + let _path = PathPrepend::new(fake.path()); + FastlyCliAdapter + .push_config_entries( dir.path(), - Some("fastly.toml"), + None, None, &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - TEST_CONFIG_ID, + &[(TEST_CONFIG_ID.to_owned(), envelope_b)], &AdapterPushContext::new(), + false, ) - .expect("local read after push B"); - let ReadConfigEntry::Present(value) = read else { - panic!("expected Present after push B"); - }; - assert_eq!( - value, envelope_b, - "read after second push must reconstruct envelope B, not A" - ); - assert_ne!( - value, envelope_a, - "old envelope A's chunks must be inert -- read must NOT return A" + .expect("push succeeds"); + let log = fs::read_to_string(&oplog).unwrap_or_default(); + assert!( + !log.lines().any(|line| line.starts_with("delete ")), + "an empty listing reclaims nothing; log:\n{log}" ); } @@ -5361,12 +5362,16 @@ echo 'unexpected' >&2; exit 1 let _lock = path_mutation_guard().lock().expect("guard"); let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); + let _grace = GraceGuard::set("0"); let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); - // Prior == what this push writes; read-back also returns it. + // Prior == what this push writes; the store holds only that generation, + // so it is both the new keep-set AND the prior -> doubly protected. + let listing = listed_generation(TEST_CONFIG_ID, &envelope, 172_800); let fake = fake_fastly_gc( TEST_CONFIG_ID, &[pointer.clone(), pointer], + &listing, None, false, &oplog, From 481f5907c182e39f63ab9c957f602a8326838031 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Mon, 13 Jul 2026 12:54:54 -0700 Subject: [PATCH 23/25] docs: reconcile spec to the shipped store-derived design; mark the plan superseded Resolves the contradiction the review flagged: the spec declared deferred reclamation while its algorithm/cloud sections still specified eager deletion, and the plan was entirely eager-delete based. There is now ONE authoritative contract. The spec records both rejected designs (eager delete; metadata sidecar) and why, so the mistakes are not repeated, and documents the live-API findings that drove the final design -- including that the root entry's updated_at is NOT bumped by `update --upsert` and therefore cannot be used as a supersession clock. The plan is retained as a historical build guide with a prominent SUPERSEDED banner pointing at the spec. --- .../plans/2026-07-07-fastly-chunk-gc.md | 25 + .../specs/2026-07-07-fastly-chunk-gc.md | 509 +++++++----------- 2 files changed, 223 insertions(+), 311 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index 6796dbdc..edc5501d 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -1,5 +1,30 @@ # Fastly Chunk GC Implementation Plan +> ## ⚠️ SUPERSEDED — historical build guide, NOT the contract +> +> This plan was written against the ORIGINAL design, in which a cloud push +> **eagerly deleted** the generation it had just superseded. **That design is +> unsafe and was rejected during the PR #314 review**: Fastly's config store is +> eventually consistent, so POPs may still be serving the previous pointer and +> need its chunks. +> +> A second attempt (a metadata "pending record" sidecar) was also rejected — no +> compare-and-swap means a failed write or a concurrent push permanently loses a +> generation, and the record overflows the 8 000-char entry limit at ~71 +> generations. +> +> **The shipped cloud design derives everything from the store itself** (group +> the store's actual chunk keys into generations; a generation is superseded when +> the next one is written; reclaim only unreferenced generations that have aged +> past a grace window; never touch the just-superseded one). +> +> **The authoritative contract is the spec:** +> `docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md`. +> +> Everything below is retained only as a record of how the work was sequenced, +> and of the two designs that were tried and discarded. Do not implement from it. + + > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Reclaim obsolete Fastly chunked config-store entries after a successful re-push without changing push success semantics. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 7a2c4dd3..1d430cfe 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -31,10 +31,14 @@ behind, unreferenced and unreclaimed: `write_fastly_local_config_store` at `cli.rs:926`, which reads the doc, inserts each physical entry, and never removes stale keys). -Every chunked re-push leaks one generation of chunks. This spec adds -a **best-effort, post-commit garbage-collection sweep** that deletes -the chunk entries the *previous* root pointer referenced and the *new* -pointer no longer does. +Every chunked re-push leaks one generation of chunks. This spec reclaims them: + +- **Local** prunes eagerly, inside the same `fastly.toml` rewrite. Safe: one + file, read by Viceroy at startup — no propagation window. +- **Cloud** reclaims orphaned generations that are unreferenced by the live + pointer AND have been superseded for longer than a grace window, deriving + both facts from the store itself. It is **never** safe to delete the + just-superseded generation immediately (see "Cloud reclamation"). > **Code-layout note.** This spec targets the layout on `main`, where > the Fastly adapter is monolithic (`cli.rs` + `chunked_config.rs`). @@ -53,9 +57,9 @@ pointer no longer does. key: the pointer at `` is overwritten in place by the existing upsert and remains the stable, live entry. - Does not touch sibling logical keys (e.g. `app_config` vs - `app_config_staging`): the delete-set is scoped, per root key, to - chunk keys that match that root's own generated chunk prefix AND were - named by that root's previous pointer. + `app_config_staging`): every candidate key must parse as + `.__edgezero_chunks..` for *that* root, so the shared + string prefix of a sibling root never matches. ## Terminology @@ -86,66 +90,46 @@ pointer no longer does. prefix-scanning the flattened multi-root physical set (which would reintroduce infix inference and mis-handle shared prefixes or a free-form key). -- **Orphans**: `prior_chunk_keys − new_keep_set`. In practice this is - all prior chunk keys, because a changed config changes the SHA and - therefore every chunk key; but the set-difference is the correct, - idempotent formulation (a re-push of *identical* bytes re-derives the - same keys and deletes nothing). "Idempotent" here is about re-pushing - the same bytes, NOT concurrency safety — see "Concurrency model: - last-writer-wins". +- **Generation**: the set of chunk entries sharing one content address + (`envelope_sha256`). Chunk keys are grouped into generations by parsing them. +- **Orphan generation**: a generation not referenced by the **live** root + pointer. Cloud reclaims an orphan only once it has also aged past the grace + window; local prunes `prior_chunk_keys − new_keep_set` immediately. ## Invariants (MUST) -1. **Nothing is deleted until the new root pointer is committed.** The - root pointer write is the atomic cutover: before it lands, the live - pointer still references the prior chunks; after it lands, the prior - chunks are unreachable. Deletes run strictly after the pointer - commits. - - Ordering per root key: - - 1. Write all new chunk entries (new-SHA keys) — upsert - 2. Write the new root pointer — upsert ← **commit point** - 3. Delete `prior_chunk_keys − new_keep_set` — best-effort - - (Locally, steps 1–3 happen as one in-memory `DocumentMut` rewrite - followed by the existing single `fs::write` of `fastly.toml` - (`cli.rs:999`). That write is a plain overwrite, NOT a durable - atomic rename — but from GC's perspective there is no partial-commit - window: the new entries and the orphan removals land in the same - rewrite, so a reader never sees new chunks without the new pointer, - nor a swept chunk that is still referenced.) - -2. **Delete only validated, prefix-matched prior references.** The - delete-set is derived from `previous_pointer.chunks[].key`, filtered - to keys with the exact prefix `"{root_key}.__edgezero_chunks."`, and - only when the previous pointer is a valid v1 pointer. It is NOT - derived from a prefix scan or a store-wide enumeration. This bounds - the blast radius: a malformed or hand-edited pointer can never cause - GC to delete a key outside its own root's chunk namespace. - -3. **A key in the new keep-set is never deleted.** Even though a SHA - change makes overlap empty in practice, the sweep subtracts the - keep-set unconditionally so an identical-bytes re-push is a no-op. - -4. **GC failure never fails or blocks the push.** The push's success - criterion is unchanged: new chunks + new pointer committed. The - sweep is optimistic — any read/parse/delete failure degrades to a - warning folded into the returned status lines; the push still - reports success. A leaked chunk is harmless; a failed push is not. - -5. **Cloud deletes obey last-writer-wins via a post-commit read-back.** - A push reclaims a root's prior chunks only while it is still the last - writer of that root. After committing, cloud GC re-reads the raw root - and deletes only if it still equals *exactly* what this push wrote - (`new_root_value`); if a newer push has superseded it (root differs) - or the re-read fails, GC **yields** — deletes nothing, warns, moves - on. This keeps a superseded push from deleting the winner's live - chunks. The guard narrows but does not fully close the window (Fastly - has no compare-and-delete); see "Concurrency model: last-writer-wins" - for the residual and its bounded impact. The local path holds the - whole file in one in-memory rewrite and has no remote-concurrency - window. +1. **Nothing is deleted until the new root pointer is committed.** The pointer + write is the cutover: before it lands, the live pointer still references the + prior chunks. Deletes run strictly after the commit. + + (Locally, this is one in-memory `DocumentMut` rewrite followed by the single + existing `fs::write` — not a durable atomic rename, but from GC's perspective + there is no partial-commit window: the new entries and the removals land in + the same rewrite.) + +2. **The just-superseded generation is NEVER deleted by the push that + supersedes it (cloud).** Fastly's config store is eventually consistent: POPs + may still be serving the previous pointer. This is unconditional — it holds + even with a zero grace window. Local is exempt: one file, no POPs. + +3. **Only validated, root-scoped keys are ever delete candidates.** A key must + parse as `.__edgezero_chunks..` for that exact root. + Deletes target the store's **actual** keys — never keys re-derived from a + content address — so a hand-edited pointer cannot retarget a delete onto keys + it never referenced. + +4. **A key in the live keep-set is never deleted.** An identical-bytes re-push + is therefore a no-op. + +5. **GC failure never fails or blocks the push.** Any read/list/parse/delete + failure degrades to a warning; the push still succeeds. Reclamation is + **stateless and idempotent**, so a later push simply recomputes and retries. + A leaked chunk is harmless; a failed push is not. + +6. **Cloud reclamation obeys last-writer-wins.** It runs only while a post-commit + read-back confirms the root still holds exactly what this push wrote. If a + newer push has superseded us, we yield — costing nothing, because there is no + state to go stale. ## Two further invariants (PR #314 review) @@ -166,242 +150,158 @@ error** in both push paths, before any expansion or I/O. (Rejecting beats silently coalescing: a duplicated key is a caller bug, and picking a winner would hide it.) -## Cloud reclamation is DEFERRED (eventual consistency) - -> **Design correction (PR #314 review).** An earlier revision of this spec had -> the cloud push delete the superseded generation immediately, guarded only by a -> post-commit read-back. That is **unsafe**: Fastly Config Store is eventually -> consistent, and the read-back observes only the **control plane**. After we -> write pointer N, POPs may still be serving pointer N-1 — which references -> N-1's chunks. Deleting them immediately strips chunks out from under those -> POPs, breaking reads on *every* re-push, not merely under concurrent writers. -> Knowing precisely *what* to delete was never the problem; knowing *when* it is -> safe is, and no control-plane read can answer that. - -So a cloud push **never deletes the generation it just superseded**. It -*records* it, and a **later** push reclaims it once it has aged out: +## Cloud reclamation: store-derived and grace-gated + +> **Design history (PR #314 review).** Two earlier revisions were wrong and are +> recorded here so the mistakes are not repeated. +> +> 1. **Eager deletion** (delete the superseded generation right after the +> commit, guarded by a read-back) is **unsafe**. Fastly Config Store is +> eventually consistent and the read-back observes only the **control +> plane**. After we write pointer N, POPs may still serve pointer N-1, which +> references N-1's chunks. Deleting them immediately strips chunks from those +> POPs — breaking reads on *every* re-push, not merely under concurrency. +> Knowing *what* to delete was never the problem; knowing *when* it is safe is. +> 2. **A metadata sidecar** (record the superseded generation, reclaim it later) +> is **unsound**. Fastly has no compare-and-swap, so a failed sidecar write, a +> failed read-back, or a concurrent read-modify-write **permanently loses** a +> generation, which can never be rediscovered. And the record itself overflows +> the 8 000-char entry limit at ~71 generations, after which every subsequent +> generation is silently lost. + +The design carries **no metadata at all — the store IS the state.** + +**Verified against the live Fastly API:** + +- `fastly config-store-entry list --store-id= --json` returns, per item, + `item_key` and **`created_at`** (also `updated_at`, `store_id`, `item_value`). +- **The root entry's own `updated_at` is NOT usable.** Fastly does **not** bump + it on `update --upsert`. Observed live: a root whose `updated_at` read + `2026-07-07` was pointing at chunks created `2026-07-13`. Using it as the + supersession clock would conclude the current generation had been stable for + six days and reclaim the previous one immediately — the exact unsafe outcome + we are trying to avoid. +- Chunk entries' own `created_at` **is** accurate and monotonic. + +**The supersession clock, derived from the store.** A generation is superseded +exactly when **the next one is written**. So, ordering generations by +`created_at`: ``` -push N (root R): - 1. read prior pointer -> prior generation (N-1) - 2. read pending record -> generations superseded by EARLIER pushes - 3. commit N's chunks + N's pointer <- commit point - 4. LWW read-back: is the root still exactly what we wrote? (else yield) - 5. RECLAIM each pending generation g where - age(g.superseded_at) >= GRACE - and keys(g) disjoint from N's keep-set - and keys(g) disjoint from keys(N-1) <- may still be live at POPs - 6. RECORD pending += { N-1, superseded_at: now } +superseded_at(G) = created_at( successor(G) ) ``` -**Pending record** at `.__edgezero_gc.pending` (a reserved namespace — -`GC_PENDING_INFIX`, rejected in logical keys exactly like the chunk infix): +with no successor meaning the generation was never referenced by any pointer +(e.g. a partially-committed push), in which case it ages from its own creation. + +**Per root, after the commit:** -```json -{ "edgezero_kind": "fastly_config_gc_pending", "version": 1, - "generations": [ { "sha": "", "count": 12, "superseded_at": 1720000000 } ] } +``` +1. LWW read-back: does the root still hold exactly what we wrote? (else yield) +2. list the store; group THIS root's actual chunk keys by content address + -> generations, each with created_at = max(created_at of its chunks) +3. protected = keys(N, what we just wrote) ∪ keys(N-1, what we just superseded) +4. for each generation G, in created_at order: + if keys(G) ∩ protected ≠ ∅ -> skip (live, or still at POPs) + if now - superseded_at(G) < GRACE -> skip (inside the grace window) + else delete G's ACTUAL keys from the listing ``` -Two properties that are load-bearing: +Three properties this buys: -- **Compact, not raw keys.** Chunk keys are derivable - (`.__edgezero_chunks..`), so the record stores `(sha, count)`. - Storing the raw key list would exceed the record's own 8 000-char entry limit - after ~2 generations. -- **A list, not a slot.** If a push outpaces the grace window, the un-reclaimed - generation must not be overwritten — that would leak it *untracked*, forever. +- **N-1 is always protected**, unconditionally, regardless of the grace window. + That is the eventual-consistency guarantee: a POP still serving the previous + pointer keeps its chunks. +- **Deletes target the store's real keys**, never keys re-derived from a content + address — so a hand-edited pointer whose SHA and key suffixes disagree can + never retarget a delete onto keys the pointer never referenced. +- **The pre-existing backlog is reclaimed for free.** Orphans leaked before this + feature existed are ordinary unreferenced generations; no tracking was needed. **Grace window:** `EDGEZERO_FASTLY_GC_GRACE_SECS`, default **86 400 (24h)**. Fastly documents no propagation bound, so the default is deliberately generous; -operators on a faster cadence can lower it. Generation N-1 is retained -unconditionally regardless of the window. +operators on a faster cadence can lower it. + +**Key-shape validation.** `chunk_key_generation` only recognises +`.__edgezero_chunks..`. A foreign or malformed key is +never grouped, and therefore never becomes a delete target. -**Failure handling:** a failed delete leaves the generation pending, so a later -push retries (deletes are idempotent — not-found is success). A malformed -pending record warns and touches nothing rather than clobbering state it cannot -understand. +**Failure handling.** A failing `list`, a failing read-back, a suspicious prior +pointer, or a failing delete all degrade to a warning; the push still succeeds +and a later push simply retries (reclamation is idempotent and stateless — there +is nothing to lose). -**The LOCAL path still prunes eagerly** and that is correct: `fastly.toml` is a -single file Viceroy reads at startup — there is no propagation window and no POP -that could still be serving the previous pointer. +**The LOCAL path prunes eagerly** and that is correct: `fastly.toml` is a single +file Viceroy reads at startup — no propagation window, no POPs. ## Concurrency model: last-writer-wins -The config value follows **last-writer-wins (LWW)**. The root key is a -single Config Store entry and `update --upsert` means the push whose -root pointer lands *last* defines the live config. Concurrent pushes to -the same store are **supported** under this semantic — no single-writer -assumption, no lock, no blocking precondition. - -GC layers onto LWW with one rule: **a push reclaims prior-generation -chunks only while it is still the last writer of the root.** That is -exactly what the invariant-5 read-back checks — after committing, the -push re-reads the root and sweeps only if it still equals what the push -wrote (`new_root_value`). If a newer push has superseded it (the root -now differs), the push **yields**: it deletes nothing and lets the new -last writer own reclamation. This is what keeps LWW honest — a -superseded (losing) push never deletes the *winner's* live chunks, so -the winner's config stays readable. - -Best-effort boundary (stated honestly): Fastly exposes no -compare-and-delete, so a newer push can still supersede us *between* the -read-back and a delete. Deferral makes that residual far narrower than it -was under eager deletion, because the only thing a sweep can ever delete -is a generation that (a) was superseded at least one push ago, (b) has -aged past the grace window, and (c) is referenced by neither the new -generation nor the just-superseded one. A losing push therefore cannot -touch the winner's live chunks in the ordinary case; it would take a -concurrent revert to a *stale, aged-out* generation inside the sweep -window. The blast radius stays bounded to one store's chunk data — never -another store, never the root pointer — and a missing chunk surfaces as -an integrity error on read rather than wrong data, recoverable by -re-pushing (chunks are written before the pointer). GC remains a -best-effort reclaimer under LWW, not a transactional one. +The config value is last-writer-wins: the root key is one entry and +`update --upsert` means the push whose pointer lands last defines the live +config. Concurrent pushes are supported. -## `prior_chunk_keys` helper (`chunked_config.rs`) +Reclamation only runs while a post-commit read-back confirms the root still +holds exactly what this push wrote; if a newer push has superseded us, we yield +and let that push reclaim. Because the design is **stateless**, yielding costs +nothing — there is no record to go stale, and the next push recomputes +everything from the store. -Add next to the existing `FastlyChunkPointer` schema -(`chunked_config.rs:47`): - -```rust -/// Validate a prior root value and return the chunk keys it referenced, -/// scoped to `root_key`'s own chunk namespace. Used only for GC. -/// -/// Returns: -/// - `Ok(keys)` — value is a valid v1 chunk pointer; `keys` are its -/// `chunks[].key` entries that match `"{root_key}.__edgezero_chunks."`. -/// - `Ok(vec![])` — value is a direct `BlobEnvelope`, absent, or not -/// pointer-shaped at all (normal: first push, or was-direct). Silent. -/// - `Err(msg)` — value IS pointer-kind (`edgezero_kind == POINTER_KIND`) -/// but fails validation: unsupported `version`, or a referenced key -/// falls outside this root's chunk prefix. Caller logs `msg` as a -/// warning and skips GC for this root (deletes nothing). -pub(crate) fn prior_chunk_keys( - root_key: &str, - raw: &str, -) -> Result, String>; -``` +Residual (stated honestly): Fastly exposes no compare-and-delete, so a newer +push could in principle intervene between our read-back and a delete. The blast +radius is bounded by the same three gates that govern every delete — the key +must be unreferenced by the live pointer, not in the just-superseded generation, +and older than the grace window. GC is a best-effort reclaimer, not a +transactional one, and a missing chunk surfaces as an integrity error on read +rather than wrong data. -Rules — parse as `serde_json::Value` **first**, so a pointer-kind value -with missing or invalid fields still reaches the warning path instead -of being silently dropped by a failed struct deserialize (e.g. -`{"edgezero_kind":"fastly_config_chunks","version":2}` has no `chunks` -array and would fail a direct `FastlyChunkPointer` deserialize): - -1. Parse `raw` as `serde_json::Value`. If that fails, or the top-level - `edgezero_kind` field is absent or `!= POINTER_KIND` → `Ok(vec![])`, - silent. (Direct `BlobEnvelope`, unrelated JSON, or a first push.) -2. The value IS pointer-kind. From here every failure is `Err(...)` - (warn, delete nothing): - - required fields missing / wrong types (no `chunks` array, - `version` not an integer, …), or - - `version != 1`, or - - any `chunks[].key` does NOT start with - `format!("{root_key}{CHUNK_KEY_INFIX}")`. -3. Otherwise → `Ok(keys)`, the `chunks[].key` values (all prefix-matched - by step 2). - -Unit tests: valid pointer → its keys; direct envelope → `Ok([])`; -unrelated JSON → `Ok([])`; wrong `edgezero_kind` → `Ok([])`; pointer-kind -with missing `chunks` and `version:2` → `Err` (NOT silently `Ok([])`); -`version` ≠ 1 → `Err`; a chunk key with a foreign prefix → `Err`. +## `prior_chunk_keys` helper (`chunked_config.rs`) -## Algorithm +Unchanged from the original design and still load-bearing: it yields the +just-superseded generation's keys (validated, prefix-scoped to the root) which +the cloud path adds to `protected`, and which the local path prunes against. + +- `Ok(keys)` — a valid v1 chunk pointer, keys prefix-matched to this root. +- `Ok(vec![])` — a direct `BlobEnvelope`, absent, or not pointer-shaped (silent). +- `Err(msg)` — pointer-*kind* but invalid (bad version, foreign-prefix key): + warn and reclaim nothing for that root. -# --- validate first (both paths), before any expansion or I/O --- -reject the whole push if ANY logical key contains CHUNK_KEY_INFIX # hard error +Parsed `serde_json::Value`-first, so a pointer-kind value with missing fields +reaches the warning path instead of being silently dropped. -For each logical `(root_key, body)` entry in the push: +## Algorithm ``` -# --- before writing --- -prev = read_root_value(root_key) # may be absent -prior = prior_chunk_keys(root_key, prev) # Ok([]) unless prev is a valid v1 pointer -expanded = prepare_fastly_config_entries(root_key, body) # THIS root only -new_keys = { k for (k, _) in expanded } # includes root_key (its last entry) -new_root_value = value of expanded.last() # exactly what we write at root_key - -# --- write (existing behaviour, unchanged) --- -push expanded # chunks first, root pointer last (commit point) - -# --- sweep (best-effort, only after the WHOLE push succeeds) --- -match prior: - Err(msg): warn(msg) # suspicious pointer; skip GC - Ok(keys): - orphans = keys − new_keys - if orphans not empty: - # cloud only — read-back concurrency guard (invariant 5): - if read_root_value(root_key) != new_root_value: - warn("root `{root_key}` changed since this push wrote it; skipping GC"); continue - for k in orphans: - try delete_entry(k) except e: warn("could not reclaim orphan `{k}`: {e}") +# validate first (both paths), before any expansion or I/O +reject if any logical key contains CHUNK_KEY_INFIX # reserved namespace +reject if any logical key appears more than once # see the duplicate-root invariant + +for each logical (root_key, body): + expanded = prepare_fastly_config_entries(root_key, body) # this root only + new_keys = { k for (k, _) in expanded } # includes root_key + new_root_value = value of expanded.last() + +# --- cloud --- +prior_keys = prior_chunk_keys(root, read(root)) # BEFORE the commit +push expanded # chunks first, pointer last (commit) +reclaim_orphan_generations(root, new_keys, new_root_value, prior_keys, now, grace) + +# --- local --- +prune orphans (prior_keys - new_keys) inside the same fastly.toml rewrite ``` -The **shrink-to-direct** case (config drops back under 8 000 chars) is -handled for free: the new value is a direct envelope, -`new_keys = { root_key }`, and every prior chunk key is swept. - -## Cloud path (`push_config_entries`, `cli.rs:349`) - -- **Dry-run stays offline.** The `dry_run` early return (`cli.rs:385`) - is *before* `resolve_remote_config_store_id` (`cli.rs:410`) and must - stay that way — dry-run never resolves the store id and never fetches - remote state. It reports GC intent without a count, e.g. - `" would delete orphaned prior-generation chunks of `app_config` (count determined at push time)"`. - (Rationale: counting orphans needs the store id + a remote read, - which would make dry-run hit the network. Not worth it; the local - dry-run below gives an exact count offline for the common case.) -- **Read prior value (real push only).** After `resolve_remote_config_store_id` - succeeds, for each logical root call the existing - `fetch_remote_config_store_entry(store_id, root_key)` (`cli.rs:671`) - to get `prev`. `Ok(None)` → no prior chunks; `Err` → warn, skip GC - for that root. This read happens *before* the committer loop. -- **Delete helper.** New `delete_config_store_entry(store_id, key)` - shelling `fastly config-store-entry delete --store-id= - --key= --auto-yes`, mirroring the spawn/stderr handling of - `create_config_store_entry` (`cli.rs:1073`). `--auto-yes` suppresses - any interactive confirmation (non-interactive shell-out). Only ever - pass `--key`: the subcommand also accepts `-a/--all`, which deletes - **every** entry in the store — never construct that flag. Treat a - "not found" / "does not exist" / "404" stderr as success (already - gone). -- **A failed delete is informational, not actionable.** The push has - already succeeded and the new pointer is live, so an un-deleted orphan - is inert. Word the warning accordingly (e.g. "could not reclaim orphan - chunk … ; it is inert and will be removed by a future `config gc`") — - do NOT imply the operator should retry the push. There is no reclaim - command in v1, so such an orphan persists as a pre-existing leak (see - Non-goals) until that future tool ships. -- **Sweep after the whole commit.** The cloud writer flattens every - logical root's physical entries into one `physical_entries` vec - (`cli.rs:380`) and commits them in a single `push_entries_with_committer` - loop (`cli.rs:411`, `:1022`). Prior-value reads for every root happen - *before* that loop; the per-root sweep runs only *after* it returns - `Ok` — i.e. after ALL roots' chunks + pointers are committed. Fold - delete failures and prior-read/parse warnings into the returned status - `Vec` (the push still returns `Ok`). -- **Read-back concurrency guard (invariant 5).** Each root's GC record - carries `new_root_value` — the exact value written at the root key - (its new pointer, or the direct envelope; i.e. the last expanded - entry's value). In the post-commit sweep, before deleting a root's - orphans, re-read the raw root via `fetch_remote_config_store_entry` - and delete only if it equals `new_root_value`; otherwise warn ("root - changed since this push wrote it") and skip that root. -- **Partial-commit failure ⇒ no GC.** If the committer fails partway, - `push_config_entries` returns `Err` and no sweep runs for any root. - Safe by construction: nothing is deleted, so the store is left with - (at worst) a mix of old and new chunks, each still referenced by - whichever pointer actually committed. Reclamation waits for the next - successful push. Leak-safe over corrupt-safe, per invariant 4. -- **Cost**: per logical root, one `describe` before the push; then, for - each root that actually has orphans, a second `describe` after commit - (the read-back guard); then one `delete` per orphan. A root with no - prior chunks costs only the first `describe` and no read-back. No - store-wide `list` call. Deletes are sequential `fastly` subprocess - spawns — a large prior generation (e.g. 50 chunks) adds ~50 sequential - shell-outs of post-push latency. Fastly exposes no bulk delete-by-key - (`--all` is store-wide), so v1 accepts this cost rather than - parallelising spawns. +## Cloud path (`push_config_entries`) + +- **Dry-run stays offline** — no store-id resolution, no remote read. It reports + reclamation intent without a count (the count depends on the listing). +- **Pre-commit:** read each root's prior pointer → `prior_keys` (the generation + about to be superseded). `Err` → warn, reclaim nothing for that root. +- **Commit:** unchanged — all roots' physical entries in one committer loop. +- **Post-commit:** `reclaim_orphan_generations` per root, as specified above. +- **Delete helper:** `fastly config-store-entry delete --store-id --key + --auto-yes`. Only ever `--key`; the subcommand also accepts `-a/--all`, which + would wipe the whole store — never construct that flag. +- **Cost:** one `describe` per root pre-commit, one read-back per root, one + `list` per root post-commit, plus one `delete` per reclaimed chunk. ## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) @@ -485,37 +385,25 @@ in the `cli.rs` test module. ### Cloud (`push_config_entries`) -- **Deletes prior, keeps new**: fake fastly serves a prior pointer for - the root; after push assert `config-store-entry delete --auto-yes` is - invoked for each prior chunk key and NOT for any new chunk key or the - root key. -- **Shrink-to-direct**: prior is a chunk pointer, new value is direct - (≤ 8 000 chars) → all prior chunk keys deleted, root key upserted - (not deleted). -- **First push (no prior)**: prior read returns not-found → zero delete - calls. -- **Identical re-push**: prior and new reference the same keys → zero - delete calls. -- **Delete failure degrades to warning**: delete exits non-zero with a - non-"not found" stderr → push returns `Ok`, status includes a warning - naming the failed chunk key. -- **Prior read failure degrades to warning**: `describe` on the root - errors (non-not-found) → push `Ok`, GC skipped, warning present. -- **Suspicious pointer degrades to warning**: prior value is - pointer-kind with `version` 2 → push `Ok`, no deletes, warning. -- **Concurrency guard skips on changed root**: the fake serves the root - `describe` twice — prior pointer pre-commit, then a *different* value - on the post-commit read-back → GC skipped for that root with a "root - changed" warning and **no** `delete` calls. (The happy-path - delete test conversely serves the newly-written value on read-back so - the guard passes.) -- **Reserved key rejected**: `push_config_entries` with a logical key - containing `.__edgezero_chunks.` returns `Err` before any `fastly` - invocation (no `list`/`describe`/`update`/`delete`). -- **Ordering**: extend the argv-log fake to assert every `delete` argv - appears strictly after the root-pointer `update` argv. -- **Dry-run stays offline**: dry-run makes no `list`/`describe`/`delete` - calls and prints the no-count GC intent line. +- **Never reclaims the just-superseded generation** — even with `GRACE=0`. This + is the eventual-consistency guarantee and the single most important test. +- **Reclaims an aged orphan generation**: an older generation, superseded before + the prior one and past the grace window, IS deleted; the live and + just-superseded generations survive. +- **Retains an orphan inside the grace window**: nothing is deleted. +- **Yields when the root changed**: a differing read-back → no deletes, warning. +- **Identical re-push**: the sole generation is both the keep-set and the prior + → doubly protected, no deletes. +- **Shrink-to-direct**: the whole chunk set is orphaned, but the just-superseded + generation is still retained. +- **Suspicious prior pointer** / **prior-read failure** / **listing failure**: + warn, delete nothing, push still succeeds. +- **Delete failure**: warns, push succeeds. +- **Delete argv**: every delete passes `--key` + `--auto-yes` and **never** + `--all` (store-wide wipe). +- **Dry-run stays offline**: no `list`/`describe`/`delete`, reports intent only. +- **Payload redaction**: schema-drift stdout *and* failing stderr never echo the + stored value (sentinel-secret tests on both the read and push paths). ### Local (`push_config_entries_local` / `write_fastly_local_config_store`) @@ -568,9 +456,8 @@ in the `cli.rs` test module. | File | Change | | --- | --- | -| `crates/edgezero-adapter-fastly/src/chunked_config.rs` | Add `pub(crate) fn prior_chunk_keys(root_key, raw) -> Result, String>` (validates v1 + prefix-scopes) + unit tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (helpers) | `reject_reserved_root_keys`, `expand_root` (exact per-root keep-set + `new_root_value`), `orphan_chunk_keys`, `delete_config_store_entry` (`--key --auto-yes`, never `--all`) | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries`, `:349`) | Reject reserved keys; per-root prior read via `fetch_remote_config_store_entry`; post-commit sweep with read-back concurrency guard; offline dry-run GC-intent line; tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`push_config_entries_local`, `:421`) | Reject reserved keys; pass exact per-root keep-sets to the writer; best-effort dry-run orphan counts (degrade to "unknown", never newly fail); tests | -| `crates/edgezero-adapter-fastly/src/cli.rs` (`write_fastly_local_config_store`, `:926`) | New `gc_roots: &[(String, HashSet)]` param (empty ⇒ no GC); snapshot old values, prune orphans in the same in-memory rewrite before the existing `fs::write` (`:999`); update all 10 call sites | -| `crates/edgezero-adapter-fastly/src/cli.rs` (test `:3317`) | Invert: assert old chunks are deleted after re-push; drop "no GC in v1" commentary | +| `crates/edgezero-adapter-fastly/src/chunked_config.rs` | `prior_chunk_keys` (validated, prefix-scoped) and `chunk_key_generation` (parses/validates a chunk key into its content address) + unit tests | +| `crates/edgezero-adapter-fastly/src/cli.rs` (cloud) | `reject_reserved_root_keys`, `reject_duplicate_root_keys`, `expand_root`, `list_config_store_entries`, `parse_rfc3339_secs`, `reclaim_orphan_generations`, `delete_config_store_entry` (`--key --auto-yes`, never `--all`), `gc_grace_secs` / `unix_now_secs` | +| `crates/edgezero-adapter-fastly/src/cli.rs` (local) | `write_fastly_local_config_store` takes exact per-root keep-sets and prunes in the same rewrite; `local_contents_table` + best-effort dry-run counts | +| `crates/edgezero-adapter-fastly/src/cli.rs` (diagnostics) | `redact_describe_response` + `redact_stderr` — diagnostics never echo a config payload | +| `crates/edgezero-adapter-fastly/Cargo.toml` | `handlebars` dev-dependency (fake-`fastly` test shim) | From e5ac9d09589a3127d8856918b01e4c543e3e9ad4 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 14 Jul 2026 11:08:20 -0700 Subject: [PATCH 24/25] fix(fastly)!: remove automatic cloud chunk deletion; add operator-invoked `config gc` A cloud push no longer deletes anything. Safe automatic reclamation is not achievable with Fastly's primitives, and three designs failed review proving it: 1. eager delete -> unsafe (eventual consistency: POPs may still serve the previous pointer, and the read-back only sees the control plane) 2. metadata sidecar -> unsound (no CAS => lost updates; 8000-char overflow) 3. store-derived clock -> unsound (chunk created_at is NOT pointer-supersession time: a chunked -> direct -> direct transition leaves the old generation with no 'successor' at all, so it ages from its own creation and gets deleted while POPs still serve it) The missing fact -- 'the pointer that referenced these chunks has stopped being served everywhere' -- is one Fastly does not record (updated_at is not bumped by an upsert; verified live) and that we cannot safely synthesise. Only the OPERATOR knows their deploy history. So reclamation becomes an explicit `config gc`: - new Adapter::gc_config_entries (default: unimplemented) + Fastly impl - one `config-store-entry list`; roots are classified via their pointers, so live chunks are known exactly - deletes only unreferenced chunk entries older than the operator's --older-than - FAILS CLOSED: an unclassifiable root or an unreadable created_at aborts with nothing deleted - a dry-run prints every key and age it would delete, so the operator's assertion is reviewable The local path keeps eager pruning (one file, no POPs) and is unchanged. --- crates/edgezero-adapter-fastly/src/cli.rs | 952 ++++++---------------- crates/edgezero-adapter/src/registry.rs | 37 + 2 files changed, 306 insertions(+), 683 deletions(-) diff --git a/crates/edgezero-adapter-fastly/src/cli.rs b/crates/edgezero-adapter-fastly/src/cli.rs index 179b8750..ef981ca6 100644 --- a/crates/edgezero-adapter-fastly/src/cli.rs +++ b/crates/edgezero-adapter-fastly/src/cli.rs @@ -1,4 +1,4 @@ -use std::collections::{HashMap, HashSet}; +use std::collections::HashSet; use std::env; use std::fs; use std::io::{ErrorKind, Write as _}; @@ -149,11 +149,16 @@ enum ConfigStoreLookup { SchemaDrift(String), } -/// One `config-store-entry list` item. The stored VALUE is deliberately not -/// captured: it is the config payload and must never be held or logged. +/// One `config-store-entry list` item. +/// +/// `item_value` IS captured — `config gc` must parse root pointers to learn +/// which chunks are live, and one listing avoids a `describe` per root. It is +/// the config payload: it may be read in memory but must NEVER be logged or +/// surfaced (see `redact_describe_response` / `redact_stderr`). struct ConfigStoreItem { created_at: String, item_key: String, + item_value: String, } /// Per-root plan for the LOCAL path's eager prune. @@ -220,6 +225,19 @@ impl Adapter for FastlyCliAdapter { } } + fn gc_config_entries( + &self, + _manifest_root: &Path, + _adapter_manifest_path: Option<&str>, + _component_selector: Option<&str>, + store: &ResolvedStoreId, + _push_ctx: &AdapterPushContext<'_>, + older_than_secs: u64, + dry_run: bool, + ) -> Result, String> { + gc_fastly_config_store(store.platform.as_str(), older_than_secs, dry_run) + } + fn name(&self) -> &'static str { "fastly" } @@ -434,54 +452,33 @@ impl Adapter for FastlyCliAdapter { body.len() )); } - out.push(format!( - " would defer reclamation of `{key}`'s prior generation (reclaimed on a later push, once aged past the grace window)" - )); } return Ok(out); } let resolved_id = resolve_remote_config_store_id(name)?; - let now = unix_now_secs(); - let grace = gc_grace_secs(); - // BEFORE the commit, capture the generation each root is about to - // supersede. It stays protected: POPs may still be serving it. - let mut priors: Vec, String>> = Vec::with_capacity(roots.len()); - for (root_key, _, _) in &roots { - priors.push( - match fetch_remote_config_store_entry(&resolved_id, root_key) { - Ok(Some(raw)) => prior_chunk_keys(root_key, &raw), - Ok(None) => Ok(Vec::new()), - Err(err) => Err(format!( - "failed to read prior root `{root_key}` for chunk GC: {err}; skipping GC for this root" - )), - }, - ); - } - // Commit all physical entries (each root's chunks first, its root - // pointer last). + // NOTE: a cloud push does NOT reclaim orphaned chunks. + // + // Fastly's config store is eventually consistent, so a generation may + // only be deleted once the pointer that referenced it has stopped being + // served everywhere. Fastly records no pointer-supersession time + // (`updated_at` is NOT bumped by `update --upsert` -- verified against + // the live API), offers no compare-and-swap with which to record one + // safely, and chunk `created_at` is NOT a proxy for it (a chunked -> + // direct -> direct transition leaves the old generation with no + // "successor" at all). Every attempt to synthesise that fact is unsound. + // + // So reclamation is an explicit, operator-invoked `config gc`: the + // operator supplies the one fact the platform cannot -- that the current + // config has been live long enough that nothing is serving the old + // pointers. See the spec's "Cloud reclamation". push_entries_with_committer(&physical_entries, |key, value| { create_config_store_entry(&resolved_id, key, value) })?; - // Post-commit reclamation, derived entirely from the store itself. - let mut warnings = Vec::new(); - for ((root_key, new_keys, new_root_value), prior) in roots.into_iter().zip(priors) { - warnings.extend(reclaim_orphan_generations( - &resolved_id, - &root_key, - &new_keys, - &new_root_value, - prior, - now, - grace, - )); - } - let mut out = vec![format!( + Ok(vec![format!( "pushed {} physical entries ({} logical) to fastly config-store `{name}` (logical id `{logical}`, id={resolved_id})", physical_entries.len(), entries.len() - )]; - out.extend(warnings); - Ok(out) + )]) } fn push_config_entries_local( @@ -1180,23 +1177,6 @@ fn unix_now_secs() -> u64 { .map_or(0, |elapsed| elapsed.as_secs()) } -/// How long a superseded generation must age before it may be reclaimed. -/// -/// Fastly's config store is eventually consistent and documents no -/// propagation bound, so eager deletion can strip chunks from a POP that is -/// still serving the previous pointer. We therefore retain generation N-1 -/// unconditionally and only reclaim older generations once they have aged -/// past this window. Override with `EDGEZERO_FASTLY_GC_GRACE_SECS` -/// (e.g. `0` in tests, or a smaller value for a fast deploy cadence). -fn gc_grace_secs() -> u64 { - /// 24h — generous against any realistic propagation lag. - const DEFAULT_GRACE_SECS: u64 = 86_400; - env::var("EDGEZERO_FASTLY_GC_GRACE_SECS") - .ok() - .and_then(|raw| raw.parse::().ok()) - .unwrap_or(DEFAULT_GRACE_SECS) -} - /// Reject a batch that names the same logical root key more than once. /// /// The adapter trait takes an entry slice and does not enforce uniqueness, @@ -1367,9 +1347,14 @@ fn list_config_store_entries(store_id: &str) -> Result, Str .get("created_at") .and_then(serde_json::Value::as_str) .unwrap_or_default(); + let item_value = entry + .get("item_value") + .and_then(serde_json::Value::as_str) + .unwrap_or_default(); items.push(ConfigStoreItem { created_at: created_at.to_owned(), item_key: item_key.to_owned(), + item_value: item_value.to_owned(), }); } Ok(items) @@ -1381,113 +1366,113 @@ fn parse_rfc3339_secs(raw: &str) -> Option { u64::try_from(stamp.timestamp()).ok() } -/// Post-commit reclamation for ONE root, derived entirely FROM THE STORE. -/// -/// Fastly's config store is eventually consistent: after the control plane -/// accepts pointer N, POPs may still serve pointer N-1 and need N-1's chunks. -/// So a generation may only be deleted once it has been superseded for longer -/// than the grace window. The question is how to know WHEN it was superseded. +/// `config gc` for Fastly: delete chunk entries that no LIVE root pointer +/// references and that are older than the operator's `older_than_secs`. /// -/// A generation is superseded exactly when the NEXT one is written, so -/// `superseded_at(G) = created_at(successor(G))` — read straight off -/// `config-store-entry list`. There is no metadata sidecar: the store IS the -/// state, so there is nothing to lose on a failed write, nothing to overflow -/// the 8 000-char entry limit, and no lost update under concurrent pushes. +/// Why this is a separate, operator-invoked command rather than part of `config +/// push`: see `Adapter::gc_config_entries`. The operator's `--older-than` is the +/// safety assertion the platform cannot make. A dry-run prints exactly which +/// keys would go, with ages, so the assertion is reviewable. /// -/// NOTE: the root entry's own `updated_at` is NOT usable — Fastly does not bump -/// it on `update --upsert` (verified against the live API: a root last -/// "updated" 2026-07-07 pointed at chunks created 2026-07-13). -fn reclaim_orphan_generations( - store_id: &str, - root_key: &str, - new_keys: &HashSet, - new_root_value: &str, - prior: Result, String>, - now: u64, - grace: u64, -) -> Vec { - let mut warnings = Vec::new(); - // Last-writer-wins: only reclaim while the root still holds our write. - match fetch_remote_config_store_entry(store_id, root_key) { - Ok(Some(current)) if current == new_root_value => {} - Ok(_) => { - warnings.push(format!( - "note: skipped chunk reclamation for `{root_key}`: root changed since this push wrote it (concurrent push?); a later push will reclaim" - )); - return warnings; +/// Fails CLOSED: if the listing is unreadable, or a root's value cannot be +/// classified, nothing is deleted. +fn gc_fastly_config_store( + store_name: &str, + older_than_secs: u64, + dry_run: bool, +) -> Result, String> { + let resolved_id = resolve_remote_config_store_id(store_name)?; + let items = list_config_store_entries(&resolved_id)?; + let now = unix_now_secs(); + + // LIVE chunk keys = those named by any root pointer currently in the store. + // A root is any entry that is not itself a chunk key. + let mut live: HashSet = HashSet::new(); + let mut roots = 0_usize; + for item in &items { + if chunk_key_generation_any(&item.item_key).is_some() { + continue; // a chunk, not a root } - Err(err) => { - warnings.push(format!( - "note: skipped chunk reclamation for `{root_key}`: could not re-read the root ({err}); a later push will reclaim" - )); - return warnings; + roots = roots.saturating_add(1); + match prior_chunk_keys(&item.item_key, &item.item_value) { + Ok(keys) => live.extend(keys), + Err(err) => { + // Cannot classify this root -> we do not know what it references. + // Fail closed rather than risk deleting something live. + return Err(format!( + "refusing to reclaim: could not classify root `{}` ({err}); nothing was deleted", + item.item_key + )); + } } } - let prior_keys = match prior { - Ok(keys) => keys, - Err(err) => { - warnings.push(format!("warning: {err}")); - return warnings; - } - }; - let items = match list_config_store_entries(store_id) { - Ok(items) => items, - Err(err) => { - warnings.push(format!( - "note: skipped chunk reclamation for `{root_key}`: {err}" - )); - return warnings; - } - }; - // Group the store's ACTUAL chunk keys into generations. We delete only keys - // the store really has — never keys re-derived from a content address. - let mut grouped: HashMap)> = HashMap::new(); - for item in items { - let Some(sha) = chunk_key_generation(root_key, &item.item_key) else { + // Candidates: chunk entries not referenced by any live pointer, older than + // the operator's threshold. + let mut doomed: Vec<(String, u64)> = Vec::new(); + let mut retained_recent = 0_usize; + for item in &items { + if chunk_key_generation_any(&item.item_key).is_none() { continue; - }; - let created = parse_rfc3339_secs(&item.created_at).unwrap_or(0); - let slot = grouped.entry(sha).or_insert((0, Vec::new())); - slot.0 = slot.0.max(created); // the generation is complete at its newest chunk - slot.1.push(item.item_key); - } - if grouped.is_empty() { - return warnings; - } - let mut ordered: Vec<(String, u64, Vec)> = grouped - .into_iter() - .map(|(sha, (created, keys))| (sha, created, keys)) - .collect(); - ordered.sort_by(|left, right| left.1.cmp(&right.1).then_with(|| left.0.cmp(&right.0))); - - // Never touch the generation we just wrote, nor the one we just superseded - // (still live at POPs that have not caught up). - let mut protected: HashSet<&str> = new_keys.iter().map(String::as_str).collect(); - protected.extend(prior_keys.iter().map(String::as_str)); - - for (idx, (_, created, keys)) in ordered.iter().enumerate() { - if keys.iter().any(|key| protected.contains(key.as_str())) { + } + if live.contains(&item.item_key) { continue; } - // Superseded when the NEXT generation was written. With no successor it - // was never referenced by any pointer (e.g. a partially-committed push), - // so age it from its own creation. - let superseded_at = ordered - .get(idx.saturating_add(1)) - .map_or(*created, |next| next.1); - if now.saturating_sub(superseded_at) < grace { + let Some(created) = parse_rfc3339_secs(&item.created_at) else { + // Unparseable timestamp on a DELETE path -> fail closed. + return Err(format!( + "refusing to reclaim: entry `{}` has an unreadable `created_at`; nothing was deleted", + item.item_key + )); + }; + let age = now.saturating_sub(created); + if age < older_than_secs { + retained_recent = retained_recent.saturating_add(1); continue; } - for key in keys { - if let Err(err) = delete_config_store_entry(store_id, key) { - warnings.push(format!( - "note: could not reclaim orphan chunk `{key}` for `{root_key}` ({err}); it is inert and a later push will retry" - )); - } + doomed.push((item.item_key.clone(), age)); + } + + let mut out = vec![format!( + "fastly config-store `{store_name}` (id={resolved_id}): {} entries, {roots} root(s), {} live chunk(s), {} orphan(s) older than {older_than_secs}s, {retained_recent} orphan(s) too recent", + items.len(), + live.len(), + doomed.len(), + )]; + if doomed.is_empty() { + out.push("nothing to reclaim".to_owned()); + return Ok(out); + } + for (key, age) in &doomed { + let verb = if dry_run { "would delete" } else { "deleting" }; + out.push(format!(" {verb} `{key}` (age {age}s)")); + } + if dry_run { + out.push(format!( + "dry-run: {} orphan chunk(s) would be deleted; re-run without --dry-run to apply", + doomed.len() + )); + return Ok(out); + } + let mut deleted = 0_usize; + for (key, _) in &doomed { + match delete_config_store_entry(&resolved_id, key) { + Ok(()) => deleted = deleted.saturating_add(1), + Err(err) => out.push(format!(" warning: could not delete `{key}` ({err})")), } } - warnings + out.push(format!( + "reclaimed {deleted} of {} orphan chunk entries", + doomed.len() + )); + Ok(out) +} + +/// Is this key a chunk key of ANY root? (`config gc` scans the whole store, so +/// it cannot scope to one root up front.) Validates the canonical shape. +fn chunk_key_generation_any(key: &str) -> Option { + let (root, _rest) = key.split_once(CHUNK_KEY_INFIX)?; + chunk_key_generation(root, key) } /// Drive a sequential per-entry commit loop and produce the @@ -2009,32 +1994,6 @@ mod tests { } } - /// RAII guard for the GC grace-window override. Callers hold - /// `path_mutation_guard()` too, which serialises env mutation. - #[cfg(unix)] - struct GraceGuard { - original: Option, - } - - #[cfg(unix)] - impl GraceGuard { - fn set(secs: &str) -> Self { - let original = env::var("EDGEZERO_FASTLY_GC_GRACE_SECS").ok(); - env::set_var("EDGEZERO_FASTLY_GC_GRACE_SECS", secs); - Self { original } - } - } - - #[cfg(unix)] - impl Drop for GraceGuard { - fn drop(&mut self) { - match self.original.take() { - Some(prev) => env::set_var("EDGEZERO_FASTLY_GC_GRACE_SECS", prev), - None => env::remove_var("EDGEZERO_FASTLY_GC_GRACE_SECS"), - } - } - } - #[test] fn finds_closest_manifest_when_multiple_exist() { let dir = tempdir().unwrap(); @@ -2827,14 +2786,10 @@ build = \"cargo build --release\" true, ) .expect("dry-run succeeds"); - // First line names the resolve+publish flow; then per key a preview - // line plus an offline GC-intent line (so callers can eyeball the - // keyset before running for real). - assert_eq!( - out.len(), - 1 + entries.len() * 2, - "header + per-entry preview + per-entry GC-intent" - ); + // First line names the resolve+publish flow; then one preview line per + // key. A push no longer reclaims anything (see `config gc`), so there is + // no GC-intent line. + assert_eq!(out.len(), 1 + entries.len(), "header + per-entry preview"); assert!( out[0].contains("would resolve fastly config-store `app_config`") && out[0].contains("push entries"), @@ -3373,7 +3328,7 @@ build = \"cargo build --release\" fn fake_fastly_gc( root_key: &str, root_describe_seq: &[String], - entry_list: &[(String, String)], + entry_list: &[(String, String, String)], fail_delete_key: Option<&str>, describe_hard_error: bool, oplog: &Path, @@ -3444,14 +3399,14 @@ echo 'unexpected' >&2; exit 1 /// A `config-store-entry list --json` payload. The item VALUE is a /// placeholder: reclamation must only ever use keys and timestamps. #[cfg(unix)] - fn entry_list_json(items: &[(String, String)]) -> String { + fn entry_list_json(items: &[(String, String, String)]) -> String { let entries: Vec = items .iter() - .map(|(key, created)| { + .map(|(key, created, value)| { serde_json::json!({ "item_key": key, "created_at": created, - "item_value": "PLACEHOLDER", + "item_value": value, }) }) .collect(); @@ -3468,12 +3423,38 @@ echo 'unexpected' >&2; exit 1 .to_rfc3339_opts(chrono::SecondsFormat::Secs, true) } - /// Every chunk key of `envelope` paired with a creation stamp. + /// Every chunk key of `envelope` as the listing would return it. #[cfg(unix)] - fn listed_generation(root_key: &str, envelope: &str, secs_ago: u64) -> Vec<(String, String)> { + fn listed_generation( + root_key: &str, + envelope: &str, + secs_ago: u64, + ) -> Vec<(String, String, String)> { let (chunks, _) = chunked_parts(root_key, envelope); let stamp = stamp_secs_ago(secs_ago); - chunks.into_iter().map(|key| (key, stamp.clone())).collect() + chunks + .into_iter() + .map(|key| (key, stamp.clone(), "CHUNK-PAYLOAD".to_owned())) + .collect() + } + + /// The ROOT entry as the listing would return it: its value is the pointer, + /// which is how `config gc` learns which chunks are live. + #[cfg(unix)] + fn listed_root(root_key: &str, envelope: &str, secs_ago: u64) -> (String, String, String) { + let (_, pointer) = chunked_parts(root_key, envelope); + (root_key.to_owned(), stamp_secs_ago(secs_ago), pointer) + } + + /// A chunked envelope with a distinct payload per tag. + #[cfg(unix)] + fn gen_envelope(tag: &str) -> String { + use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + use edgezero_core::blob_envelope::BlobEnvelope; + use serde_json::json; + let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); + serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:00Z".to_owned())) + .expect("envelope") } /// Split a chunked envelope into (chunk keys, root pointer value). @@ -4337,560 +4318,208 @@ echo 'unexpected' >&2; exit 1 ); } - // ---------- cloud chunk reclamation (store-derived) ---------- + // ---------- config gc (operator-invoked reclamation) ---------- - /// Envelope factory with a distinct payload per tag. #[cfg(unix)] - fn gen_envelope(tag: &str) -> String { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - use edgezero_core::blob_envelope::BlobEnvelope; - use serde_json::json; - let data = json!({ tag: "x".repeat(FASTLY_CONFIG_ENTRY_LIMIT) }); - serde_json::to_string(&BlobEnvelope::new(data, "2026-06-22T00:00:00Z".to_owned())) - .expect("envelope") + fn run_gc(dir: &Path, older_than_secs: u64, dry_run: bool) -> Result, String> { + FastlyCliAdapter.gc_config_entries( + dir, + None, + None, + &ResolvedStoreId::from_logical(TEST_CONFIG_ID), + &AdapterPushContext::new(), + older_than_secs, + dry_run, + ) } - /// THE eventual-consistency guarantee: the generation this push just - /// superseded is NEVER deleted, even with a zero grace window — POPs may - /// still be serving the pointer that references it. + /// gc never deletes a chunk the LIVE root pointer references, however old. #[cfg(unix)] #[test] - fn push_config_entries_never_reclaims_the_just_superseded_generation() { + fn gc_never_deletes_live_chunks() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); // even so, the prior generation is safe let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - - // The store holds A (long-lived) and, post-commit, B. - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_a, 100_000); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + let live = gen_envelope("live"); + let (live_chunks, _) = chunked_parts(TEST_CONFIG_ID, &live); + // The live generation is ANCIENT, but it is referenced by the root. + let mut listing = vec![listed_root(TEST_CONFIG_ID, &live, 999_999)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &live, 999_999)); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &listing, - None, - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); + let out = run_gc(dir.path(), 0, false).expect("gc succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); - for key in a_chunks.iter().chain(b_chunks.iter()) { + for key in &live_chunks { assert!( !oplog_has(&oplog, &format!("delete {key}")), - "must not delete the just-superseded (or new) generation: `{key}`; log:\n{log}" + "live chunk `{key}` must never be reclaimed; log:\n{log}\nout: {out:?}" ); } } - /// An OLDER orphan generation — superseded before the prior one, and aged - /// past the grace window — IS reclaimed. + /// gc reclaims unreferenced chunks older than the operator's threshold. #[cfg(unix)] #[test] - fn push_config_entries_reclaims_aged_orphan_generation() { + fn gc_reclaims_unreferenced_chunks_older_than_threshold() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("3600"); // 1h let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_x = gen_envelope("gen_x"); // long-dead orphan - let envelope_a = gen_envelope("gen_a"); // prior (live before this push) - let envelope_b = gen_envelope("gen_b"); // what we push now - let (x_chunks, _x_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_x); - let (a_chunks, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (b_chunks, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let live = gen_envelope("live"); + let dead = gen_envelope("dead"); + let (live_chunks, _) = chunked_parts(TEST_CONFIG_ID, &live); + let (dead_chunks, _) = chunked_parts(TEST_CONFIG_ID, &dead); - // X written 3 days ago; A written 2 days ago (so X was superseded 2 days - // ago -> well past the 1h grace); B written now. - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + let mut listing = vec![listed_root(TEST_CONFIG_ID, &live, 3_600)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &live, 3_600)); + listing.extend(listed_generation(TEST_CONFIG_ID, &dead, 604_800)); // a week old - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &listing, - None, - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - for key in &x_chunks { + // Operator asserts: nothing older than 1 day is still being served. + let out = run_gc(dir.path(), 86_400, false).expect("gc succeeds"); + for key in &dead_chunks { assert!( oplog_has(&oplog, &format!("delete {key}")), - "aged orphan `{key}` must be reclaimed; log:\n{log}" + "orphan `{key}` older than the threshold must be reclaimed; out: {out:?}" ); } - for key in a_chunks.iter().chain(b_chunks.iter()) { + for key in &live_chunks { assert!( !oplog_has(&oplog, &format!("delete {key}")), - "live/just-superseded chunk `{key}` must survive" + "live chunk `{key}` must survive" ); } - assert!( - !oplog_has(&oplog, &format!("delete {TEST_CONFIG_ID}")), - "root pointer must never be deleted" - ); - // Reclamation follows the commit. - let root_update = log - .lines() - .position(|line| line == format!("update {TEST_CONFIG_ID}")) - .expect("root update logged"); - let first_delete = log - .lines() - .position(|line| line.starts_with("delete ")) - .expect("a delete logged"); - assert!(first_delete > root_update, "deletes follow the commit"); } - /// An orphan still inside the grace window is retained. + /// Orphans younger than the operator's threshold are retained. #[cfg(unix)] #[test] - fn push_config_entries_retains_orphan_within_grace() { + fn gc_retains_orphans_younger_than_threshold() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("86400"); // 24h let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_x = gen_envelope("gen_x"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (x_chunks, _x) = chunked_parts(TEST_CONFIG_ID, &envelope_x); - let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); + let live = gen_envelope("live"); + let recent = gen_envelope("recent"); + let (recent_chunks, _) = chunked_parts(TEST_CONFIG_ID, &recent); - // X superseded only 1h ago (when A was written) -> inside a 24h grace. - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 7_200); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 3_600)); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + let mut listing = vec![listed_root(TEST_CONFIG_ID, &live, 60)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &live, 60)); + listing.extend(listed_generation(TEST_CONFIG_ID, &recent, 120)); // 2 min old - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &listing, - None, - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); + run_gc(dir.path(), 86_400, false).expect("gc succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); - for key in &x_chunks { + for key in &recent_chunks { assert!( !oplog_has(&oplog, &format!("delete {key}")), - "orphan `{key}` is inside the grace window and must be retained; log:\n{log}" + "orphan `{key}` is younger than the threshold and must be retained; log:\n{log}" ); } } - /// Reclamation yields when a concurrent push has superseded us. + /// A dry-run lists exactly what it would delete, and deletes nothing. #[cfg(unix)] #[test] - fn push_config_entries_skips_gc_when_root_changed() { + fn gc_dry_run_lists_but_deletes_nothing() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_x = gen_envelope("gen_x"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (_x, _xp) = chunked_parts(TEST_CONFIG_ID, &envelope_x); - let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); + let live = gen_envelope("live"); + let dead = gen_envelope("dead"); + let (dead_chunks, _) = chunked_parts(TEST_CONFIG_ID, &dead); - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); + let mut listing = vec![listed_root(TEST_CONFIG_ID, &live, 3_600)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &live, 3_600)); + listing.extend(listed_generation(TEST_CONFIG_ID, &dead, 604_800)); - // The read-back returns the OLD pointer: someone else won. - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer.clone(), a_pointer], - &listing, - None, - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); + let out = run_gc(dir.path(), 86_400, true).expect("dry-run succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( !log.lines().any(|line| line.starts_with("delete ")), - "no deletes when the root changed under us; log:\n{log}" + "a dry-run must not delete; log:\n{log}" ); + let rendered = out.join("\n"); assert!( - out.iter().any(|line| line.contains("root changed")), - "must warn that reclamation was skipped: {out:?}" + rendered.contains("would delete"), + "lists intent: {rendered}" ); - } - - /// Every reclamation delete must pass `--key` + `--auto-yes` and NEVER - /// `--all` (which would wipe the whole store). - #[cfg(unix)] - #[test] - fn push_config_entries_delete_uses_key_and_auto_yes_never_all() { - let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("3600"); - let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let envelope_x = gen_envelope("gen_x"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); - - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &listing, - None, - false, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - - let log = fs::read_to_string(&oplog).unwrap_or_default(); - let argv_lines: Vec<&str> = log - .lines() - .filter(|line| line.starts_with("delete-argv ")) - .collect(); - assert!(!argv_lines.is_empty(), "a reclamation happened: {log}"); - for line in argv_lines { - assert!( - line.contains("--auto-yes"), - "delete passes --auto-yes: {line}" - ); - assert!(line.contains("--key="), "delete targets a --key: {line}"); - assert!( - !line.contains("--all"), - "delete must NEVER pass --all (store-wide wipe): {line}" - ); + for key in &dead_chunks { + assert!(rendered.contains(key.as_str()), "names `{key}`: {rendered}"); } } - /// A failed reclamation delete warns; the push still succeeds. + /// An unreadable `created_at` on a DELETE path fails CLOSED. #[cfg(unix)] #[test] - fn push_config_entries_delete_failure_warns_but_succeeds() { + fn gc_fails_closed_on_unreadable_timestamp() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("3600"); let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let envelope_x = gen_envelope("gen_x"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (x_chunks, _x) = chunked_parts(TEST_CONFIG_ID, &envelope_x); - let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let fail_key = x_chunks.first().expect("a chunk").clone(); - - let mut listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_a, 172_800)); - listing.extend(listed_generation(TEST_CONFIG_ID, &envelope_b, 0)); + let live = gen_envelope("live"); + let dead = gen_envelope("dead"); + let mut listing = vec![listed_root(TEST_CONFIG_ID, &live, 3_600)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &live, 3_600)); + // An orphan whose timestamp is garbage. + let (dead_chunks, _) = chunked_parts(TEST_CONFIG_ID, &dead); + for key in dead_chunks { + listing.push((key, "not-a-timestamp".to_owned(), "X".to_owned())); + } - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &listing, - Some(&fail_key), - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push still succeeds despite a failed reclamation delete"); - assert!( - out.iter() - .any(|line| line.contains("could not reclaim orphan chunk") - && line.contains(&fail_key)), - "failed delete surfaces a warning: {out:?}" - ); - } - /// A suspicious prior pointer skips reclamation (we cannot tell what is - /// live), warns, and deletes nothing. - #[cfg(unix)] - #[test] - fn push_config_entries_suspicious_prior_warns_no_deletes() { - let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); - let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let envelope_x = gen_envelope("gen_x"); - let envelope_b = gen_envelope("gen_b"); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); - - let listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[bad, b_pointer], - &listing, - None, - false, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "suspicious prior => no deletes; log:\n{log}" - ); + let err = run_gc(dir.path(), 86_400, false).expect_err("must fail closed"); assert!( - out.iter().any(|line| line.contains("skipping chunk GC")), - "warns about the suspicious pointer: {out:?}" + err.contains("unreadable") && err.contains("nothing was deleted"), + "must refuse to reclaim: {err}" ); - } - - /// A hard prior-read failure skips reclamation with a warning. - #[cfg(unix)] - #[test] - fn push_config_entries_prior_read_failure_warns_no_deletes() { - let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); - let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - - let envelope_x = gen_envelope("gen_x"); - let envelope_b = gen_envelope("gen_b"); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - let listing = listed_generation(TEST_CONFIG_ID, &envelope_x, 259_200); - - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[String::new(), b_pointer], - &listing, - None, - true, // first describe of each key fails hard - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds despite a prior-read failure"); let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( !log.lines().any(|line| line.starts_with("delete ")), - "prior-read failure => no deletes; log:\n{log}" - ); - assert!( - out.iter() - .any(|line| line.contains("failed to read prior root")), - "must warn about the failed prior read: {out:?}" + "nothing may be deleted when the state is unreadable; log:\n{log}" ); } - /// Dry-run stays fully offline and reports reclamation intent. + /// A root whose pointer cannot be classified fails CLOSED — we cannot know + /// what it references, so nothing may be deleted. #[cfg(unix)] #[test] - fn push_config_entries_dry_run_reports_gc_intent_offline() { - let dir = tempdir().expect("tempdir"); - let envelope = gen_envelope("gen_a"); - // No fake on PATH: a dry-run must not shell out at all. - let out = FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], - &AdapterPushContext::new(), - true, - ) - .expect("dry-run must not error"); - assert!( - out.iter() - .any(|line| line.contains("would defer reclamation")), - "dry-run reports deferred-reclamation intent: {out:?}" - ); - } - - /// Shrinking to a direct value orphans the whole chunk set — but the - /// just-superseded generation is still protected. - #[cfg(unix)] - #[test] - fn push_config_entries_shrink_to_direct_defers_prior_chunks() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; + fn gc_fails_closed_on_unclassifiable_root() { let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); let dir = tempdir().expect("tempdir"); let oplog = dir.path().join("ops.log"); - let chunked = gen_envelope("gen_a"); - let direct = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT); - let (prior_chunks, prior_pointer) = chunked_parts(TEST_CONFIG_ID, &chunked); - let listing = listed_generation(TEST_CONFIG_ID, &chunked, 172_800); + let dead = gen_envelope("dead"); + // Root value is pointer-kind but invalid. + let bad = r#"{"edgezero_kind":"fastly_config_chunks","version":2}"#.to_owned(); + let mut listing = vec![(TEST_CONFIG_ID.to_owned(), stamp_secs_ago(3_600), bad)]; + listing.extend(listed_generation(TEST_CONFIG_ID, &dead, 604_800)); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[prior_pointer, direct.clone()], - &listing, - None, - false, - &oplog, - ); + let fake = fake_fastly_gc(TEST_CONFIG_ID, &[], &listing, None, false, &oplog); let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), direct)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - for key in &prior_chunks { - assert!( - !oplog_has(&oplog, &format!("delete {key}")), - "the just-superseded generation must be retained even on shrink-to-direct: `{key}`; log:\n{log}" - ); - } + let err = run_gc(dir.path(), 86_400, false).expect_err("must fail closed"); assert!( - oplog_has(&oplog, &format!("update {TEST_CONFIG_ID}")), - "root upserted with the direct value" + err.contains("could not classify root") && err.contains("nothing was deleted"), + "must refuse to reclaim: {err}" ); - } - - /// A failing `config-store-entry list` degrades to a warning; the push - /// still succeeds and nothing is deleted. - #[cfg(unix)] - #[test] - fn push_config_entries_list_failure_warns_no_deletes() { - let _lock = path_mutation_guard().lock().expect("guard"); - let _grace = GraceGuard::set("0"); - let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let envelope_a = gen_envelope("gen_a"); - let envelope_b = gen_envelope("gen_b"); - let (_a, a_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_a); - let (_b, b_pointer) = chunked_parts(TEST_CONFIG_ID, &envelope_b); - - // An empty listing means nothing can be grouped -> nothing reclaimed. - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[a_pointer, b_pointer], - &[], - None, - false, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope_b)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); let log = fs::read_to_string(&oplog).unwrap_or_default(); assert!( !log.lines().any(|line| line.starts_with("delete ")), - "an empty listing reclaims nothing; log:\n{log}" + "nothing may be deleted when a root is unclassifiable; log:\n{log}" ); } @@ -5352,49 +4981,6 @@ echo 'unexpected' >&2; exit 1 } } - /// Cloud identical-bytes re-push reclaims nothing (new keys equal prior - /// keys). Read-back returns our own value so the guard would PASS if - /// orphans were miscomputed — making "no deletes" non-vacuous. - #[cfg(unix)] - #[test] - fn push_config_entries_identical_repush_issues_no_deletes() { - use crate::chunked_config::FASTLY_CONFIG_ENTRY_LIMIT; - let _lock = path_mutation_guard().lock().expect("guard"); - let dir = tempdir().expect("tempdir"); - let oplog = dir.path().join("ops.log"); - let _grace = GraceGuard::set("0"); - let envelope = make_test_envelope(FASTLY_CONFIG_ENTRY_LIMIT.saturating_add(1)); - let (_chunks, pointer) = chunked_parts(TEST_CONFIG_ID, &envelope); - // Prior == what this push writes; the store holds only that generation, - // so it is both the new keep-set AND the prior -> doubly protected. - let listing = listed_generation(TEST_CONFIG_ID, &envelope, 172_800); - let fake = fake_fastly_gc( - TEST_CONFIG_ID, - &[pointer.clone(), pointer], - &listing, - None, - false, - &oplog, - ); - let _path = PathPrepend::new(fake.path()); - FastlyCliAdapter - .push_config_entries( - dir.path(), - None, - None, - &ResolvedStoreId::from_logical(TEST_CONFIG_ID), - &[(TEST_CONFIG_ID.to_owned(), envelope)], - &AdapterPushContext::new(), - false, - ) - .expect("push succeeds"); - let log = fs::read_to_string(&oplog).unwrap_or_default(); - assert!( - !log.lines().any(|line| line.starts_with("delete ")), - "identical re-push must delete nothing; log:\n{log}" - ); - } - // ---- chunk GC helpers ---- #[test] diff --git a/crates/edgezero-adapter/src/registry.rs b/crates/edgezero-adapter/src/registry.rs index fcc5bfa4..28d734af 100644 --- a/crates/edgezero-adapter/src/registry.rs +++ b/crates/edgezero-adapter/src/registry.rs @@ -239,6 +239,43 @@ pub trait Adapter: Sync + Send { /// Returns an error string if the requested adapter action fails. fn execute(&self, action: AdapterAction, args: &[String]) -> Result<(), String>; + /// Reclaim chunk entries that no LIVE config pointer references. + /// + /// Deliberately NOT part of `config push`. On an eventually-consistent + /// store, a chunk may only be deleted once the pointer that referenced it + /// has stopped being served everywhere — and the platform may record no + /// pointer-supersession time (Fastly does not: `updated_at` is not bumped + /// by an upsert) and offer no compare-and-swap with which to record one + /// safely. Only the OPERATOR knows their deploy history, so `older_than` + /// carries that assertion: "nothing created before this is still being + /// served". A `dry_run` lists exactly what would be deleted, for review. + /// + /// # Errors + /// + /// Returns `Err` if the adapter has no `config gc` impl, if the platform + /// state cannot be read, or if it cannot be classified with confidence — + /// reclamation FAILS CLOSED: when in doubt, nothing is deleted. + #[inline] + #[expect( + clippy::too_many_arguments, + reason = "mirrors `push_config_entries`: manifest root, adapter manifest path, component selector, resolved store, push-time overlay, the operator's age assertion, and dry-run — each distinct; an aggregate struct is a worse ergonomic trade for implementers." + )] + fn gc_config_entries( + &self, + _manifest_root: &Path, + _adapter_manifest_path: Option<&str>, + _component_selector: Option<&str>, + _store: &ResolvedStoreId, + _push_ctx: &AdapterPushContext<'_>, + _older_than_secs: u64, + _dry_run: bool, + ) -> Result, String> { + Err(format!( + "adapter `{}` does not implement `config gc`", + self.name() + )) + } + /// Store kinds whose logical-id namespaces the adapter merges into /// a single backend at runtime — declaring the SAME logical id /// under two merged kinds causes silent write collisions because From 6e909bd0bf44d815553257fa91fb46c9cd251382 Mon Sep 17 00:00:00 2001 From: Aram Grigoryan <132480+aram356@users.noreply.github.com> Date: Tue, 14 Jul 2026 11:10:25 -0700 Subject: [PATCH 25/25] docs: spec `config gc`; record why automatic cloud reclamation is impossible Resolves the spec's self-contradiction (it claimed pre-existing leaks were both reclaimed and never reclaimed) and makes it the single authoritative contract. Records all three rejected automatic designs (eager delete; metadata sidecar; store-derived clock) with the concrete counterexample that killed each, and the live-API findings behind them -- notably that Fastly does not bump updated_at on an upsert, and that chunk created_at is not a pointer-supersession time. States the conclusion plainly: the fact needed to delete safely is one the platform does not record and we cannot synthesise, so the operator supplies it via `config gc --older-than`. Plan banner updated to match. --- .../plans/2026-07-07-fastly-chunk-gc.md | 35 +- .../specs/2026-07-07-fastly-chunk-gc.md | 330 ++++++++---------- 2 files changed, 165 insertions(+), 200 deletions(-) diff --git a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md index edc5501d..52638a6f 100644 --- a/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/plans/2026-07-07-fastly-chunk-gc.md @@ -3,26 +3,33 @@ > ## ⚠️ SUPERSEDED — historical build guide, NOT the contract > > This plan was written against the ORIGINAL design, in which a cloud push -> **eagerly deleted** the generation it had just superseded. **That design is -> unsafe and was rejected during the PR #314 review**: Fastly's config store is -> eventually consistent, so POPs may still be serving the previous pointer and -> need its chunks. +> **eagerly deleted** the generation it had just superseded. > -> A second attempt (a metadata "pending record" sidecar) was also rejected — no -> compare-and-swap means a failed write or a concurrent push permanently loses a -> generation, and the record overflows the 8 000-char entry limit at ~71 -> generations. +> **Three automatic cloud designs were built and each was demolished in review:** > -> **The shipped cloud design derives everything from the store itself** (group -> the store's actual chunk keys into generations; a generation is superseded when -> the next one is written; reclaim only unreferenced generations that have aged -> past a grace window; never touch the just-superseded one). +> 1. **eager delete** — unsafe: the store is eventually consistent, so POPs may +> still serve the previous pointer, and the read-back only sees the control plane; +> 2. **metadata sidecar** — unsound: no compare-and-swap ⇒ lost updates, plus an +> 8 000-char overflow at ~71 generations; +> 3. **store-derived clock** — unsound: chunk `created_at` is not a pointer +> transition (chunked → direct → direct leaves the old generation with no +> "successor" at all). +> +> **Conclusion: safe AUTOMATIC cloud reclamation is not achievable with Fastly's +> primitives.** The needed fact — "the pointer that referenced these chunks has +> stopped being served everywhere" — is one Fastly does not record and we cannot +> safely synthesise. Only the operator knows it. +> +> **What shipped:** a cloud push reclaims **nothing**; the local path prunes +> eagerly (safe — one file, no POPs); and reclamation is an explicit, operator- +> invoked **`config gc --older-than `** that fails closed and offers a +> reviewable dry-run. > > **The authoritative contract is the spec:** > `docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md`. > -> Everything below is retained only as a record of how the work was sequenced, -> and of the two designs that were tried and discarded. Do not implement from it. +> Everything below is retained only as a record of how the work was sequenced and +> of the designs that were tried and discarded. Do not implement from it. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md index 1d430cfe..9dd033ec 100644 --- a/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md +++ b/docs/superpowers/specs/2026-07-07-fastly-chunk-gc.md @@ -98,38 +98,30 @@ Every chunked re-push leaks one generation of chunks. This spec reclaims them: ## Invariants (MUST) -1. **Nothing is deleted until the new root pointer is committed.** The pointer - write is the cutover: before it lands, the live pointer still references the - prior chunks. Deletes run strictly after the commit. - - (Locally, this is one in-memory `DocumentMut` rewrite followed by the single - existing `fs::write` — not a durable atomic rename, but from GC's perspective - there is no partial-commit window: the new entries and the removals land in - the same rewrite.) - -2. **The just-superseded generation is NEVER deleted by the push that - supersedes it (cloud).** Fastly's config store is eventually consistent: POPs - may still be serving the previous pointer. This is unconditional — it holds - even with a zero grace window. Local is exempt: one file, no POPs. - -3. **Only validated, root-scoped keys are ever delete candidates.** A key must - parse as `.__edgezero_chunks..` for that exact root. - Deletes target the store's **actual** keys — never keys re-derived from a - content address — so a hand-edited pointer cannot retarget a delete onto keys - it never referenced. - -4. **A key in the live keep-set is never deleted.** An identical-bytes re-push - is therefore a no-op. - -5. **GC failure never fails or blocks the push.** Any read/list/parse/delete - failure degrades to a warning; the push still succeeds. Reclamation is - **stateless and idempotent**, so a later push simply recomputes and retries. - A leaked chunk is harmless; a failed push is not. - -6. **Cloud reclamation obeys last-writer-wins.** It runs only while a post-commit - read-back confirms the root still holds exactly what this push wrote. If a - newer push has superseded us, we yield — costing nothing, because there is no - state to go stale. +1. **A cloud push never deletes.** It writes chunks, then the pointer. Any + reclamation is a separate, operator-invoked `config gc`. + +2. **Nothing is deleted that a live pointer references.** `config gc` computes + the live set by parsing every root's pointer. Local prunes only + `prior_chunk_keys − new_keep_set`. + +3. **Only canonical, root-scoped keys are delete candidates.** A key must parse + as `.__edgezero_chunks..`. Deletes target the store's + ACTUAL keys — never keys re-derived from a content address. + +4. **Destructive paths FAIL CLOSED.** If `config gc` cannot classify a root, or + cannot read an entry's `created_at`, it aborts and deletes **nothing**. An + unreadable state must never fail open into deletion. + +5. **The operator's `--older-than` is the safety assertion.** The machine cannot + know when a pointer stopped being served; the operator can. A `--dry-run` + prints every key and age it would delete so that assertion is reviewable. + +6. **Local eager pruning is safe and stays.** One file, read at Viceroy startup: + no propagation window, no POPs. + +7. **Push failure semantics are unchanged.** Reclamation is never part of a + push, so it can never fail one. ## Two further invariants (PR #314 review) @@ -150,158 +142,132 @@ error** in both push paths, before any expansion or I/O. (Rejecting beats silently coalescing: a duplicated key is a caller bug, and picking a winner would hide it.) -## Cloud reclamation: store-derived and grace-gated +## Cloud reclamation: NOT automatic. `config gc`, invoked by the operator. -> **Design history (PR #314 review).** Two earlier revisions were wrong and are -> recorded here so the mistakes are not repeated. +> **This is the single most important section. Three automatic designs were +> built and each was demolished in review. They are recorded so nobody tries a +> fourth.** > -> 1. **Eager deletion** (delete the superseded generation right after the -> commit, guarded by a read-back) is **unsafe**. Fastly Config Store is -> eventually consistent and the read-back observes only the **control -> plane**. After we write pointer N, POPs may still serve pointer N-1, which -> references N-1's chunks. Deleting them immediately strips chunks from those -> POPs — breaking reads on *every* re-push, not merely under concurrency. -> Knowing *what* to delete was never the problem; knowing *when* it is safe is. -> 2. **A metadata sidecar** (record the superseded generation, reclaim it later) -> is **unsound**. Fastly has no compare-and-swap, so a failed sidecar write, a -> failed read-back, or a concurrent read-modify-write **permanently loses** a -> generation, which can never be rediscovered. And the record itself overflows -> the 8 000-char entry limit at ~71 generations, after which every subsequent -> generation is silently lost. - -The design carries **no metadata at all — the store IS the state.** - -**Verified against the live Fastly API:** - -- `fastly config-store-entry list --store-id= --json` returns, per item, - `item_key` and **`created_at`** (also `updated_at`, `store_id`, `item_value`). -- **The root entry's own `updated_at` is NOT usable.** Fastly does **not** bump - it on `update --upsert`. Observed live: a root whose `updated_at` read - `2026-07-07` was pointing at chunks created `2026-07-13`. Using it as the - supersession clock would conclude the current generation had been stable for - six days and reclaim the previous one immediately — the exact unsafe outcome - we are trying to avoid. -- Chunk entries' own `created_at` **is** accurate and monotonic. - -**The supersession clock, derived from the store.** A generation is superseded -exactly when **the next one is written**. So, ordering generations by -`created_at`: +> 1. **Eager delete** (remove the superseded generation right after the commit, +> guarded by a read-back) — **unsafe**. The store is eventually consistent and +> the read-back only observes the **control plane**. POPs may still be serving +> the previous pointer, which references the chunks being deleted. This breaks +> reads on *every* re-push, not merely under concurrency. +> 2. **Metadata sidecar** (record the superseded generation; reclaim it later) — +> **unsound**. Fastly has no compare-and-swap, so a failed write, a failed +> read-back, or a concurrent push **permanently loses** a generation; and the +> record overflows the 8 000-char entry limit at ~71 generations. +> 3. **Store-derived clock** (`superseded_at(G) = created_at(successor(G))`) — +> **unsound**. Chunk creation is not a pointer transition. Counterexample: +> chunked **A** → direct **B** → direct **C**. During C, A is the only chunk +> generation listed, so it has *no successor* and ages from its own creation — +> and is deleted, though B superseded it seconds ago and POPs may still serve +> A. Partial pushes and chunk/pointer write gaps break it identically. + +**The impossibility, stated plainly.** To delete a chunk safely you must know +that *the pointer which referenced it has stopped being served everywhere, for +longer than the propagation window*. Fastly: + +- **does not record** that fact — `updated_at` is **not** bumped by + `update --upsert` (verified against the live API: a root reading `updated_at = + 2026-07-07` was pointing at chunks created `2026-07-13`); +- **offers no CAS** with which we could record it ourselves safely; +- and its chunk `created_at` **is not a proxy** for it (design 3 above). + +The fact simply is not available to the machine. **It is available to the +operator**, who knows their own deploy history. So the operator supplies it. + +### What a cloud `config push` does + +**It reclaims nothing.** It writes chunks, then the pointer. That is all. Cloud +storage therefore accretes orphaned generations exactly as it does today — this +is **not a regression**, and it is the only safe automatic behaviour. + +### `config gc` (`Adapter::gc_config_entries`) ``` -superseded_at(G) = created_at( successor(G) ) +config gc --adapter fastly [--older-than ] [--dry-run] ``` -with no successor meaning the generation was never referenced by any pointer -(e.g. a partially-committed push), in which case it ages from its own creation. +1. One `config-store-entry list --json`. +2. Classify every non-chunk entry as a **root**; parse its value with + `prior_chunk_keys` → the chunk keys that root's pointer **references**. The + union over all roots is the **live** set. (The listing already carries + `item_value`, so this costs no extra `describe` calls.) +3. Candidates = chunk entries **not** in the live set, whose `created_at` is + older than `--older-than`. +4. Delete them. -**Per root, after the commit:** +**`--older-than` is the operator's safety assertion**: *"nothing created before +this is still being served."* Only they can make it. -``` -1. LWW read-back: does the root still hold exactly what we wrote? (else yield) -2. list the store; group THIS root's actual chunk keys by content address - -> generations, each with created_at = max(created_at of its chunks) -3. protected = keys(N, what we just wrote) ∪ keys(N-1, what we just superseded) -4. for each generation G, in created_at order: - if keys(G) ∩ protected ≠ ∅ -> skip (live, or still at POPs) - if now - superseded_at(G) < GRACE -> skip (inside the grace window) - else delete G's ACTUAL keys from the listing -``` +**It fails CLOSED.** If a root's value cannot be classified, or an entry's +`created_at` cannot be read, `config gc` **aborts and deletes nothing** — an +unreadable state must never fail open on a destructive path. + +**A `--dry-run` prints every key it would delete, with its age**, so the +assertion is reviewable before it is acted on. + +Only ever `--key` on delete; the subcommand also accepts `-a/--all`, which would +wipe the store — never construct that flag. + +### Local is different, and eager pruning there is correct -Three properties this buys: - -- **N-1 is always protected**, unconditionally, regardless of the grace window. - That is the eventual-consistency guarantee: a POP still serving the previous - pointer keeps its chunks. -- **Deletes target the store's real keys**, never keys re-derived from a content - address — so a hand-edited pointer whose SHA and key suffixes disagree can - never retarget a delete onto keys the pointer never referenced. -- **The pre-existing backlog is reclaimed for free.** Orphans leaked before this - feature existed are ordinary unreferenced generations; no tracking was needed. - -**Grace window:** `EDGEZERO_FASTLY_GC_GRACE_SECS`, default **86 400 (24h)**. -Fastly documents no propagation bound, so the default is deliberately generous; -operators on a faster cadence can lower it. - -**Key-shape validation.** `chunk_key_generation` only recognises -`.__edgezero_chunks..`. A foreign or malformed key is -never grouped, and therefore never becomes a delete target. - -**Failure handling.** A failing `list`, a failing read-back, a suspicious prior -pointer, or a failing delete all degrade to a warning; the push still succeeds -and a later push simply retries (reclamation is idempotent and stateless — there -is nothing to lose). - -**The LOCAL path prunes eagerly** and that is correct: `fastly.toml` is a single -file Viceroy reads at startup — no propagation window, no POPs. - -## Concurrency model: last-writer-wins - -The config value is last-writer-wins: the root key is one entry and -`update --upsert` means the push whose pointer lands last defines the live -config. Concurrent pushes are supported. - -Reclamation only runs while a post-commit read-back confirms the root still -holds exactly what this push wrote; if a newer push has superseded us, we yield -and let that push reclaim. Because the design is **stateless**, yielding costs -nothing — there is no record to go stale, and the next push recomputes -everything from the store. - -Residual (stated honestly): Fastly exposes no compare-and-delete, so a newer -push could in principle intervene between our read-back and a delete. The blast -radius is bounded by the same three gates that govern every delete — the key -must be unreferenced by the live pointer, not in the just-superseded generation, -and older than the grace window. GC is a best-effort reclaimer, not a -transactional one, and a missing chunk surfaces as an integrity error on read -rather than wrong data. +`fastly.toml` is a single file Viceroy reads at startup: there is no propagation +window and no POP that could still be serving the previous pointer. The local +path prunes the prior generation immediately, inside the same rewrite. + +## Concurrency model + +The config value is last-writer-wins: the root is one entry and `update --upsert` +means the push whose pointer lands last defines the live config. Concurrent +pushes are supported. + +Because a push **never deletes**, there is no push-time reclamation race to +reason about at all. `config gc` is a separate, operator-timed action; it deletes +only what no live pointer references and what the operator has asserted is old +enough. Running it concurrently with a push is still the operator's call — the +`--older-than` assertion is what makes it safe. ## `prior_chunk_keys` helper (`chunked_config.rs`) -Unchanged from the original design and still load-bearing: it yields the -just-superseded generation's keys (validated, prefix-scoped to the root) which -the cloud path adds to `protected`, and which the local path prunes against. +Load-bearing in both paths: it yields the chunk keys a pointer **references**, +validated and prefix-scoped to that root. -- `Ok(keys)` — a valid v1 chunk pointer, keys prefix-matched to this root. +- `Ok(keys)` — a valid v1 chunk pointer. - `Ok(vec![])` — a direct `BlobEnvelope`, absent, or not pointer-shaped (silent). -- `Err(msg)` — pointer-*kind* but invalid (bad version, foreign-prefix key): - warn and reclaim nothing for that root. +- `Err(msg)` — pointer-*kind* but invalid (bad version, foreign-prefix key). Parsed `serde_json::Value`-first, so a pointer-kind value with missing fields -reaches the warning path instead of being silently dropped. +reaches the error path instead of being silently dropped. In `config gc` an +`Err` here **aborts the whole run** (we cannot know what that root references). + +`chunk_key_generation` recognises only the canonical shape +`.__edgezero_chunks..`, so a foreign or hand-edited key is +never a delete candidate. ## Algorithm ``` -# validate first (both paths), before any expansion or I/O +# validate first (both push paths), before any expansion or I/O reject if any logical key contains CHUNK_KEY_INFIX # reserved namespace -reject if any logical key appears more than once # see the duplicate-root invariant +reject if any logical key appears more than once # duplicate-root invariant for each logical (root_key, body): - expanded = prepare_fastly_config_entries(root_key, body) # this root only - new_keys = { k for (k, _) in expanded } # includes root_key - new_root_value = value of expanded.last() - -# --- cloud --- -prior_keys = prior_chunk_keys(root, read(root)) # BEFORE the commit -push expanded # chunks first, pointer last (commit) -reclaim_orphan_generations(root, new_keys, new_root_value, prior_keys, now, grace) + expanded = prepare_fastly_config_entries(root_key, body) # this root only + new_keys = { k for (k, _) in expanded } # includes root_key -# --- local --- -prune orphans (prior_keys - new_keys) inside the same fastly.toml rewrite -``` +# --- cloud push --- +commit expanded (chunks first, pointer last). NO deletes. -## Cloud path (`push_config_entries`) +# --- local push --- +prune (prior_chunk_keys - new_keys) inside the same fastly.toml rewrite -- **Dry-run stays offline** — no store-id resolution, no remote read. It reports - reclamation intent without a count (the count depends on the listing). -- **Pre-commit:** read each root's prior pointer → `prior_keys` (the generation - about to be superseded). `Err` → warn, reclaim nothing for that root. -- **Commit:** unchanged — all roots' physical entries in one committer loop. -- **Post-commit:** `reclaim_orphan_generations` per root, as specified above. -- **Delete helper:** `fastly config-store-entry delete --store-id --key - --auto-yes`. Only ever `--key`; the subcommand also accepts `-a/--all`, which - would wipe the whole store — never construct that flag. -- **Cost:** one `describe` per root pre-commit, one read-back per root, one - `list` per root post-commit, plus one `delete` per reclaimed chunk. +# --- config gc (operator) --- +list -> live = union of prior_chunk_keys(root, root_value) over all roots +doomed = chunk entries not in live, older than --older-than +delete doomed # fails closed on any unclassifiable/unreadable state +``` ## Local path (`push_config_entries_local` / `write_fastly_local_config_store`) @@ -383,28 +349,25 @@ in the `cli.rs` test module. - A `chunks[].key` outside `"{root_key}.__edgezero_chunks."` → `Err`, returns no keys. -### Cloud (`push_config_entries`) - -- **Never reclaims the just-superseded generation** — even with `GRACE=0`. This - is the eventual-consistency guarantee and the single most important test. -- **Reclaims an aged orphan generation**: an older generation, superseded before - the prior one and past the grace window, IS deleted; the live and - just-superseded generations survive. -- **Retains an orphan inside the grace window**: nothing is deleted. -- **Yields when the root changed**: a differing read-back → no deletes, warning. -- **Identical re-push**: the sole generation is both the keep-set and the prior - → doubly protected, no deletes. -- **Shrink-to-direct**: the whole chunk set is orphaned, but the just-superseded - generation is still retained. -- **Suspicious prior pointer** / **prior-read failure** / **listing failure**: - warn, delete nothing, push still succeeds. -- **Delete failure**: warns, push succeeds. -- **Delete argv**: every delete passes `--key` + `--auto-yes` and **never** - `--all` (store-wide wipe). -- **Dry-run stays offline**: no `list`/`describe`/`delete`, reports intent only. +### Cloud push (`push_config_entries`) + +- **Never deletes anything** — a push writes chunks + pointer, nothing else. +- Reserved-key and duplicate-root keys are hard errors before any I/O. +- Dry-run stays offline (no `list`/`describe`/`delete`). - **Payload redaction**: schema-drift stdout *and* failing stderr never echo the stored value (sentinel-secret tests on both the read and push paths). +### `config gc` (`gc_config_entries`) + +- **Never deletes a live chunk**, however old it is (referenced by a root + pointer ⇒ untouchable). +- **Reclaims unreferenced chunks older than `--older-than`.** +- **Retains unreferenced chunks younger than `--older-than`.** +- **Dry-run** names every key + age it would delete, and deletes nothing. +- **Fails closed on an unreadable `created_at`** — aborts, deletes nothing. +- **Fails closed on an unclassifiable root** — aborts, deletes nothing. +- Delete argv passes `--key` + `--auto-yes` and **never** `--all`. + ### Local (`push_config_entries_local` / `write_fastly_local_config_store`) - **Prunes orphan chunk keys**: seed a `fastly.toml` whose `contents` @@ -439,15 +402,10 @@ in the `cli.rs` test module. ## Non-goals -- **Reclaiming pre-existing leaks.** This sweep only deletes the - generation the *current* live pointer references. Chunks orphaned by - pushes that predate this feature (or by a prior sweep that partially - failed) are not enumerated and not reclaimed. Steady state going - forward: each push cleans the immediately-prior generation, so at most - one stale generation exists between pushes. A one-time full reclaim - (prefix-scan the store and delete unreferenced `__edgezero_chunks` - keys) is deferred to a possible future `config gc` command and is out - of scope here. +- **Automatic cloud reclamation.** Out of scope because it is not achievable + safely (see "Cloud reclamation"). Cloud orphans — including those that predate + this feature — are reclaimed by the operator running `config gc`, which is IN + scope and implemented. - **Transactional multi-key GC.** Each root key is swept independently; there is no cross-key atomicity beyond what each path already provides (per-entry for cloud, whole-file for local).