fix(responses): bound database work under concurrent bursts

[pjb157]

Summary

• cache API-key policy for /responses authentication while retaining live
  routing validation and spend-cap child-key correctness
• batch response lifecycle creates behind durable acknowledgements and collapse owned response retrieval to one Fusillade query
• share bounded read/write admission across request and response-step stores, reserving pool headroom for unrelated database work
• cover the production-shaped path with a constrained-pool burst regression

Closes COR-550.

Why

Concurrent response creates and reads independently acquired database
connections and repeated several queries per request. A burst could therefore
fill the Fusillade pool, delay authentication queries, and surface 500/502
responses even while inference remained healthy.

Database work

For the cached, verified, non-ZDR, image-normalisation-disabled Flex path:

• Flex create before: five main-database statements per request, then one
  Fusillade transaction with two inserts per request.
• Flex create after: zero main-database statements for a cached cap-child
  execution key, or one child-existence lookup for an uncapped/shared-key
  request; one Fusillade transaction with two bulk INSERT ... UNNEST
  statements per microbatch.
• Owned GET before: one main-database auth lookup plus four Fusillade reads.
• Owned GET after: zero main-database auth lookups plus one owner-scoped
  Fusillade query.

Uncapped/shared resolution deliberately does not memoize a negative child
lookup: if a spend-cap child is created after the metadata snapshot, the next
request must discover it rather than execute against the shared key. Cold
hidden-key creation and unverified-volume enforcement retain their required
database work.

Correctness and load shedding

• A Flex create is acknowledged only after the batch transaction commits.
• Queued creates whose callers disconnect are discarded before the first write
  and before every retry, so an abandoned acknowledgement cannot create
  executable orphan work.
• Invalid items are isolated without failing valid neighbours.
• The request and response-step stores use one aggregate admission budget.
• Admission happens before pool acquisition and is reacquired for each retry;
  permits are released before backoff and cancellation-safe.
• The aggregate budget reserves two physical connections where available.
• Read and write class limits remain independently configurable.
• Metrics expose bounded operation labels for admitted/in-flight/waiting work.
• API-key metadata refresh runs independently of Onwards target sync, so key
  revocation continues to remove response GET/DELETE authorization when target
  synchronization is disabled.

Verification

• 100 concurrent Flex creates with a four-connection Fusillade pool and
  two connections deliberately held: 100 × HTTP 202, unique durable rows, one
  create transaction attempt
• 100 concurrent authorized GETs under the same constraint: 100 × HTTP 200
  with matching IDs
• 100-operation mixed create/read/completion pass drains cleanly and
  persists every expected row
• canceled callers are discarded before initial write and during retry
  backoff (14/14 writer tests)
• revoked keys lose GET and DELETE authorization with Onwards sync disabled
• focused Fusillade Arsenal, response writer, middleware, cache, and handler
  tests
• SQLx prepared-query, migration, and checksum checks
• formatting and Clippy

The workspace Rust run completed 1,854 tests successfully. Fifteen unrelated
tests failed in the local environment: organization-email tests could not
initialize the local DNS resolver, and a small set of parallel database/listener
tests passed when rerun in isolation. All response/Fusillade tests touched by
this PR pass.

Rollout

The response archive mappings are explicit. Deploy the additive archive/schema
support before enabling code paths that depend on the new mapped columns.

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	d07b9b0
Status:	✅  Deploy successful!
Preview URL:	https://f12a0c54.control-layer.pages.dev
Branch Preview URL:	https://peter-fix-cor-550-responses.control-layer.pages.dev

View logs

[Copilot]

Pull request overview

This PR reduces database contention and improves correctness under concurrent bursts for the OpenAI-compatible /responses APIs by (1) caching API-key metadata for auth/ownership decisions, (2) batching and commit-acknowledging response lifecycle creates via a singleton writer, and (3) adding bounded admission to keep response traffic from exhausting the Fusillade pool.

Changes:

• Add an API-key metadata snapshot + resolver to keep /responses auth/ownership and Flex hidden-batch-key resolution off the main DB hot path.
• Introduce commit-acknowledged, microbatched durable response creates (and best-effort completions) via RequestsWriter, and wire it through inference paths and outlet.
• Add new daemon/storage config for response read concurrency and expand schema-parity tests to support explicit archive mappings.

Reviewed changes

Copilot reviewed 33 out of 37 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
fusillade/src/manager/postgres.rs	Plumbs new daemon config into PostgresStorageConfig.
fusillade/src/daemon/config.rs	Adds max_concurrent_response_reads with defaults + serde round-trip tests.
fusillade-core/src/request/query.rs	Introduces ResolvedResponseDetail and CreateResponseInput for batching/ownership resolution.
fusillade-core/src/request/mod.rs	Re-exports new request query types.
fusillade-core/src/manager.rs	Extends Storage trait with owner-scoped response lookup + batched response creates.
fusillade-arsenal/tests/archive_schema_parity.rs	Updates parity contract to name/shape-based checks + explicit forward/reverse mappings.
fusillade-arsenal/src/response_step.rs	Adds admission-aware writes and ensures paired construction shares admission identity.
fusillade-arsenal/src/lib.rs	Adds max_concurrent_response_reads to storage config + defaults/tests.
fusillade-arsenal/src/db.rs	Adds admission-aware pool acquisition + admitted transactions.
dwctl/src/test/utils.rs	Updates test app builder wiring (writer handle, API-key cache, Flex resolver).
dwctl/src/test/responses.rs	Adds regression tests for cache refresh/revocation behavior and lifecycle persistence.
dwctl/src/test/multi_step_executor.rs	Updates multi-step tests to use shared step manager + writer gating behavior.
dwctl/src/test/mod.rs	Updates router tests to pass writer/cache/resolver and updated build_router signature.
dwctl/src/sync/mod.rs	Replaces ZDR-only cache module with unified api_key_cache module export.
dwctl/src/sync/api_key_cache.rs	New lock-free API-key metadata snapshot + Flex hidden-key resolver + sync loop/tests.
dwctl/src/metrics/errors.rs	Renames component label from zdr_key_sync to api_key_cache_sync.
dwctl/src/lib.rs	Wires writer handle + API-key cache + Flex resolver through AppState and BackgroundServices; adjusts outlet attachment conditions.
dwctl/src/inference/zdr.rs	Switches per-key ZDR decision to use API-key metadata cache.
dwctl/src/inference/streaming.rs	Streams Flex requests only after commit-backed admission via writer handle; adds test.
dwctl/src/inference/store.rs	Uses writer for durable creates; adds owner-scoped response retrieval path.
dwctl/src/inference/handler.rs	Moves /responses auth/ownership checks to cached metadata + single Fusillade query; adds constrained-pool test.
dwctl/src/inference/engine/writer.rs	Implements batched create commands with commit acknowledgements + bounded retry/cancellation semantics; expands tests.
dwctl/src/inference/engine/outlet_handler.rs	Routes completion persistence through writer handle and cached attribution (no dwctl DB lookup).
dwctl/src/inference/engine/mod.rs	Re-exports RequestsWriterHandle.
dwctl/src/error_enrichment.rs	Removes Flex-specific modality enforcement helper/tests as response routing/auth model changes.
dwctl/src/db/handlers/api_keys.rs	Refactors hidden-key upsert to avoid update-trigger path; now uses INSERT + SELECT fallback.
dwctl/src/config.rs	Adds response admission config + writer linger config; documents defaults and mapping tests.
dwctl/src/auth/middleware.rs	Updates auth middleware tests to include new AppState fields.
docs/src/reference/configuration.md	Documents new response admission settings and aggregate budget behavior.
config.yaml	Adds documented defaults for response admission caps and aggregate budget note.
.sqlx/query-fa5f7c95862e2679ae0196cf76afa24ac9af59569eaaf6dd27e338ced4e6db34.json	Updates SQLx metadata for archive move-back query aliasing.
.sqlx/query-d6379af554c098413dc98b8e4fdda47376b36fb19f02c5a060b649a355593785.json	Adds SQLx metadata for bulk insert into request_templates via UNNEST.
.sqlx/query-9f238e2dbe568b27e6e0a33f7ff5a7eac465ae5bb2e0c9b2b59322fed2c21e48.json	Updates SQLx metadata for archive retry move-back query aliasing.
.sqlx/query-3251ec3d0700f85fe59bb3153a8cfb74fc3cf24790a2ddeef00878d8a98d5355.json	Adds SQLx metadata for bulk insert into requests via UNNEST.

Files not reviewed (2)

• .sqlx/query-3251ec3d0700f85fe59bb3153a8cfb74fc3cf24790a2ddeef00878d8a98d5355.json: Generated file
• .sqlx/query-d6379af554c098413dc98b8e4fdda47376b36fb19f02c5a060b649a355593785.json: Generated file

Comments suppressed due to low confidence (1)

dwctl/src/db/handlers/api_keys.rs:389

• Same as above: using sqlx::query_scalar here skips SQLx’s compile-time query validation. Using the query_scalar! macro keeps this query covered by cargo sqlx prepare --check and avoids runtime-only failures.

        let existing_secret = sqlx::query_scalar::<_, String>(
            r#"
            SELECT secret
            FROM api_keys
            WHERE user_id = $1