Harden storage recovery and cloud serving

.github/workflows/ci.yml

@@ -49,13 +49,35 @@ jobs:
           toolchain: stable
       - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
       - uses: ./.github/actions/setup-protobuf
+      - name: Start MinIO and create the integration bucket
+        run: |
+          docker run --detach --name pb-minio \
+            --publish 9100:9000 \
+            --env MINIO_ROOT_USER=minioadmin \
+            --env MINIO_ROOT_PASSWORD=minioadmin \
+            minio/minio:RELEASE.2025-07-23T15-54-02Z server /data
+          for attempt in $(seq 1 30); do
+            if curl --fail --silent http://127.0.0.1:9100/minio/health/ready >/dev/null; then
+              break
+            fi
+            if [ "$attempt" -eq 30 ]; then
+              docker logs pb-minio
+              exit 1
+            fi
+            sleep 1
+          done
+          docker run --rm --network host --entrypoint /bin/sh \
+            minio/mc:RELEASE.2025-07-21T05-28-08Z \
+            -c 'mc alias set local http://127.0.0.1:9100 minioadmin minioadmin && mc mb --ignore-existing local/poly-book-test'
       # The #[ignore]d cases: ClickHouse roundtrips, Parquet-vs-ClickHouse
       # cross-backend equivalence, and the MinIO S3 roundtrip — the strongest
       # consistency evidence in the suite, previously never enforced in CI.
       # Ubuntu runners ship a working Docker daemon for testcontainers.
       # Single-threaded: the containers are memory-hungry and the tests share
       # fixed host ports.
       - run: cargo test -p pb-integration-tests -- --ignored --test-threads=1
+        env:
+          PB_TEST_S3_ENDPOINT: http://127.0.0.1:9100
 
   clippy:
     name: Clippy
@@ -102,18 +124,24 @@ jobs:
   audit:
     name: Security Audit
     runs-on: ubuntu-latest
-    permissions:
-      checks: write
-      contents: read
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
-      - uses: rustsec/audit-check@69366f33c96575abad1ee0dba8212993eecbe998 # v2
+      - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
         with:
-          token: ${{ secrets.GITHUB_TOKEN }}
-          # quick-xml DoS advisories with no upgrade path yet (object_store
-          # pins quick-xml < 0.41). Rationale and removal condition are
-          # documented on the matching ignore entries in deny.toml.
-          ignore: RUSTSEC-2026-0194,RUSTSEC-2026-0195
+          toolchain: stable
+      # Keep cargo-audit's own dependency graph locked. Installing without
+      # --locked can resolve transitive crates whose MSRV exceeds the stable
+      # compiler selected above, failing before the repository is audited.
+      - name: Install cargo-audit
+        run: cargo install cargo-audit --version 0.22.2 --locked
+      - name: Audit Rust dependencies
+        # quick-xml DoS advisories with no upgrade path yet (object_store pins
+        # quick-xml < 0.41). Rationale and removal condition are documented on
+        # the matching ignore entries in deny.toml.
+        run: >-
+          cargo audit
+          --ignore RUSTSEC-2026-0194
+          --ignore RUSTSEC-2026-0195
 
   monitoring:
     name: Monitoring Rules

CLAUDE.md

@@ -57,7 +57,7 @@ design notes, and a **Docs to Update After Changes** table.
 - `thiserror` for library crate errors, `anyhow` only in pb-bin
 - `tracing` for structured logging (not `log` or `println!`)
 - Wire types borrow from raw buffers (`&'a str`) for zero-copy deserialization
-- Storage uses `object_store` trait for filesystem abstraction (local FS / S3 / GCS)
+- Storage uses the `object_store` trait for filesystem abstraction (local FS / S3)
 - `mimalloc` as global allocator in pb-bin for lower p99 latency
 - `FxHashMap` (rustc-hash) for internal hot-path lookups in trusted-data contexts
 - `proptest` for property-based invariant testing in pb-types and pb-book

Cargo.toml

@@ -73,7 +73,7 @@ bincode = "1"
 
 # Storage
 arrow = { version = "58", features = ["prettyprint"] }
-parquet = { version = "58", features = ["async", "zstd"] }
+parquet = { version = "58", features = ["async", "object_store", "zstd"] }
 object_store = { version = "0.13", features = ["aws"] }
 clickhouse = { version = "0.15", features = ["inserter"] }
 

README.md

@@ -237,6 +237,22 @@ just backfill <TOKEN_ID>
 cargo run -- backfill --tokens <TOKEN_ID> --interval-secs 60
 ```
 
+### Recover complete Parquet hours from WAL
+
+```bash
+# Stop ingest and every other Parquet writer first. The command verifies the
+# exclusive WAL lease; standalone backfill/append writers are an operator check.
+cargo run -- reconcile
+```
+
+Recovery fails closed on WAL corruption and partial boundary hours. For every
+fully covered receive-time-partitioned UTC hour (`book_events`, `trade_events`,
+and `ingest_events`) it writes and verifies a new Parquet object, publishes an
+authoritative manifest, and only then cleans superseded objects. Checkpoint,
+validation, and execution partitions are left untouched because their partition
+timestamps do not provide the same WAL coverage proof. Local files and S3 use
+the same object-store reader and manifest semantics.
+
 ### Inspect local Parquet output
 
 ```bash

TESTING.md

@@ -15,13 +15,13 @@ commands so they can be re-checked instead of trusted.
 
 | # | Failure mode | Primary defense | Backstop | Runs in CI? |
 |---|--------------|-----------------|----------|-------------|
-| 1 | Numeric precision loss or rounding drift in prices/sizes | 16 proptest properties on `FixedPrice`/`FixedSize` (roundtrip, ordering, serde) | `fuzz_fixed_price` (parse + serde roundtrip assert), 170 unit tests in pb-types | yes (`test`, `fuzz`) |
+| 1 | Numeric precision loss or rounding drift in prices/sizes | 16 proptest properties on `FixedPrice`/`FixedSize` (roundtrip, ordering, serde) | `fuzz_fixed_price` (parse + serde roundtrip assert), 173 unit tests in pb-types | yes (`test`, `fuzz`) |
 | 2 | Order-book invariant violations (mis-sorted levels, wrong size accounting) | 17 proptest properties on `L2Book` | `fuzz_book_delta` (arbitrary snapshots + deltas, ordering asserted after every step), 74 unit tests in pb-book | yes (`test`, `fuzz`) |
 | 3 | Malformed venue input crashing ingest | `fuzz_ws_deser` (arbitrary bytes into the wire parser: must error, never panic) | 67 unit tests in pb-feed (dispatcher normalization, reconnect paths) | yes (`fuzz`, `test`) |
-| 4 | WAL corruption on disk (bit flips, torn writes, zero-filled tails) | `fuzz_wal_corruption` (write, corrupt bytes, assert reader never panics and returns only CRC-valid records) | pb-wal unit tests for truncated-tail recovery, CRC/length corruption, prune safety (80 test attributes) + 2 proptest properties | yes (`fuzz`, `test`) |
+| 4 | WAL corruption on disk (bit flips, torn writes, zero-filled tails) | `fuzz_wal_corruption` (write, corrupt bytes, assert reader never panics and returns only CRC-valid records) | pb-wal unit tests for truncated-tail recovery, CRC/length corruption, prune safety (82 test attributes) + 2 proptest properties | yes (`fuzz`, `test`) |
 | 5 | Silent WAL byte-format drift between builds | Golden-bytes fixture `golden_codec_book_v2_bytes_are_stable` (pb-wal): any codec change that alters encoded bytes fails the build | `fuzz_codec_decode` (arbitrary bytes into `codec::decode`: error, never panic); version byte rejected on mismatch | yes (`test`, `fuzz`) |
-| 6 | Non-deterministic replay (same events, different book) | Determinism fixture `tests/integration/book_determinism.rs` (3 tests) | Golden replay regression `golden_replay_produces_expected_book` (pb-replay), 47 pb-replay test attributes | yes (`test`) |
-| 7 | Parquet and ClickHouse answering the same query differently | 3 cross-backend equivalence tests (replay, integrity, execution) in `tests/integration/cross_backend_service.rs` | 5 ClickHouse round-trip tests, 1 S3/MinIO round-trip; `reconcile` rebuilds Parquet from the WAL when they do diverge | yes (`integration-docker` runs the `#[ignore]`d Docker-backed cases via testcontainers) |
+| 6 | Non-deterministic replay (same events, different book) | Determinism fixture `tests/integration/book_determinism.rs` (3 tests) | Golden replay regression `golden_replay_produces_expected_book` (pb-replay), 51 pb-replay test attributes | yes (`test`) |
+| 7 | Parquet and ClickHouse answering the same query differently | 3 cross-backend equivalence tests (replay, integrity, execution) in `tests/integration/cross_backend_service.rs` | 5 ClickHouse round-trip tests; S3/MinIO write + production-reader round-trip; manifest-switch and partial-hour refusal tests for `reconcile` | yes (`integration-docker` runs the `#[ignore]`d Docker-backed cases) |
 | 8 | SQL escaping the read-only query workbench | `fuzz_query_guard` (guarded SQL must be a fixed point of the guard) | Guard unit tests in pb-service/pb-api; server-side readonly + LIMIT enforcement | yes (`fuzz`, `test`) |
 | 9 | Crash/restart data loss across the ingest/serve boundary | 2 checkpoint + WAL hydration integration tests (`checkpoint_wal_hydration.rs`) | pb-wal reopen-recovery and position-file tests; standby-writer flock takeover test | yes (`test`) |
 | 10 | Undefined behavior / memory unsafety | Miri on pb-types and pb-book unit tests | Workspace is overwhelmingly safe Rust; `clippy -D warnings` | yes (`miri`) |
@@ -46,14 +46,14 @@ Test attributes (`#[test]` + `#[tokio::test]`) per crate at this revision:
 
 | Crate | Tests | Crate | Tests |
 |-------|------:|-------|------:|
-| pb-types | 170 | pb-replay | 47 |
-| pb-api | 81 | pb-store | 36 |
-| pb-wal | 80 | pb-grpc | 25 |
+| pb-types | 173 | pb-replay | 51 |
+| pb-api | 83 | pb-store | 41 |
+| pb-wal | 82 | pb-grpc | 25 |
 | pb-book | 74 | pb-metrics | 14 |
-| pb-bin | 74 | pb-service | 72 |
+| pb-bin | 75 | pb-service | 72 |
 | pb-feed | 67 | `tests/integration` | 22 |
 
-Total: 762 test attributes. These counts include `#[ignore]`d tests: one in
+Total: 779 test attributes. These counts include `#[ignore]`d tests: one in
 pb-wal (a failover timing drill, run via `just failover-drill`) and nine in the
 integration package (see below).
 
@@ -62,10 +62,12 @@ The integration package (`tests/integration/`) splits cleanly:
 - **Run in CI** (13 tests): book determinism (3), checkpoint + WAL hydration
   (2), dispatcher pipeline (2), Parquet round-trip (2), replay engine (2),
   schema conversion (2).
-- **`#[ignore]`d, Docker-backed, not run in CI** (9 tests): ClickHouse
+- **`#[ignore]`d, Docker-backed, run by the `integration-docker` CI job** (9 tests): ClickHouse
   round-trips (5), cross-backend Parquet/ClickHouse equivalence for replay,
-  integrity, and execution (3), and an S3/MinIO round-trip (1). They use
-  `testcontainers` and require a local Docker daemon:
+  integrity, and execution (3), and an S3/MinIO round-trip through the production
+  `ParquetReader` (1). The ClickHouse cases use `testcontainers`; the S3 case
+  requires a running MinIO/S3-compatible endpoint in `PB_TEST_S3_ENDPOINT` (CI
+  starts MinIO explicitly). Run all nine locally with:
   `cargo test -p pb-integration-tests -- --ignored`.
 
 ### Property-based tests
@@ -187,11 +189,21 @@ npx vite build && npx playwright test               # what the e2e job runs
 
 Being explicit about what is *not* covered:
 
-- **No kill -9 crash-recovery end-to-end test.** Torn-tail truncation, CRC
-  skipping, and reopen recovery are unit-tested by constructing damaged
-  segments directly, and the flock standby-takeover path has a test — but no
+- **No kill -9 crash-recovery end-to-end test.** Torn-tail truncation, live-reader
+  CRC skipping, strict recovery rejection, manifest publication, and reopen
+  recovery are unit-tested by constructing damaged segments and storage views
+  directly, and the flock standby-takeover path has a test — but no
   harness SIGKILLs a live `ingest` mid-append and asserts the full
   recover-and-reconcile path end to end.
+- **Recovery is intentionally scoped.** Crash-consistent replacement is tested
+  for receive-time-partitioned book/trade/ingest hours. Checkpoint, validation,
+  and execution partitions are rejected because WAL endpoint timestamps cannot
+  prove those independently partitioned datasets complete.
+- Recovery reader coverage includes an asset containing `/` and `%`, catching
+  double-encoding bugs across object filenames, manifests, and cleanup paths.
+- Reconcile validates the whole retained WAL without retaining it, rejects an
+  eligible hour above 2,000,000 records or 128 MiB of encoded WAL payload, then
+  bounds decoded recovery memory to one eligible hour at a time.
 - **No soak or load testing.** There is no sustained multi-hour run under
   production-like message rates; channel capacities are sized by reasoning
   plus the exported depth gauge, not by a measured burst profile.

crates/pb-api/README.md

@@ -75,6 +75,12 @@ AnyQueryService  ──▶ query workbench handlers (datasets, SQL)
   delta, and checkpoint apply it runs `L2Book::check_integrity`; a crossed/locked
   book increments `pb_crossed_books_total` and surfaces a `crossed_book`
   continuity warning instead of being served silently.
+- Separated `serve` hydration inventories Parquet checkpoints once for the full
+  active-asset set and replays the retained WAL with a distinct checkpoint
+  offset per asset. An asset without a checkpoint forces replay from the
+  earliest retained WAL record; a legacy checkpoint without a WAL cut is
+  ignored when WAL replay is configured. `/health/ready` stays at 503 when checkpoint
+  inventory, WAL open/read, or record decoding fails.
 - The projector keeps a cached published projection and only rebuilds
   `AssetReadView` snapshots for assets touched by a record, instead of
   re-materializing every tracked book on each update.
@@ -108,6 +114,6 @@ AnyQueryService  ──▶ query workbench handlers (datasets, SQL)
 
 ## Tests
 
-68 tests covering health states, error format consistency, depth and time parameter
+83 tests covering health states, error format consistency, depth and time parameter
 validation, execution limits, `ServiceError` to `ApiError` mapping,
 `PerAssetBroadcast` unit tests, and `LiveReadModel` tests.