Harden storage recovery and cloud serving

[weiminglong]

Summary

• make Parquet replay, API services, hydration, and WAL resync use the configured object-store backend for local and S3 storage
• make WAL reconciliation coverage-aware, strict, memory-bounded, and crash-consistent with staged publication and manifest-aware cleanup
• fix encoded object paths, checkpoint inventory scaling, object-disappearance retries, and bounded Parquet reads
• make hydration and startup readiness fail closed, preserve per-asset WAL cutoffs, and use the effective historical backend after ClickHouse fallback
• make doctor perform a write/read/delete probe and redact cloud credential details from failures
• run the MinIO round-trip in CI instead of silently skipping it, and propagate the updated behavior through crate docs, operations guidance, monitoring, IAM, and the archived OpenSpec

Why

The prior recovery and cloud-serving paths had several correctness gaps:

• reconciliation could replace a partially covered hour and temporarily delete the only visible generation
• the writer supported S3 while production readers still assumed a local filesystem
• already encoded asset keys could be encoded again
• startup could report ready after checkpoint or WAL recovery failed
• checkpoint discovery could issue an excessive number of object-store requests
• the S3 integration test passed without exercising S3 when no endpoint was configured

These changes make the failure modes explicit, preserve direct-consumer-compatible storage layout, and add regression coverage for interrupted publication, corrupt WAL data, incomplete coverage, encoded asset IDs, backend fallback, and per-asset hydration cutoffs.

Validation

• cargo test --locked --workspace --exclude pb-integration-tests
• cargo test --locked -p pb-integration-tests - 13 non-Docker tests passed; Docker-backed tests remained ignored locally
• cargo test --locked -p pb-api hydration - 6 passed
• cargo check --locked -p pb-api -p pb-bin
• cargo clippy --locked -p pb-api -p pb-bin --all-targets -- -D warnings
• cargo fmt --all -- --check
• git diff --check
• cargo test --locked -p pb-integration-tests --test s3_minio_roundtrip --no-run
• local doctor --skip-network write/read/delete probe passed
• invalid S3 doctor probe failed closed with a sanitized error

The local Docker daemon was unavailable, so the live MinIO round-trip was compiled but not executed locally. CI now provisions MinIO and requires that test.

Remaining production work

• add a distributed storage-prefix lease shared by every Parquet writer before allowing concurrent cloud reconciliation
• repair the checked-in Compose startup contract for external auth and required token configuration
• run live deployment, failover, cloud-denial, and restore drills before production sign-off