feat!: add spare-capacity background workload processing

[pjb157]

What this adds

This adds a no-SLA background inference class that uses spare live-model
capacity without reducing the capacity available to completion-SLA traffic.
It supports both asynchronous batchless Responses requests and file-backed
Batch API jobs.

The implementation now lives in Control Layer alongside the Fusillade crates
and replaces doublewordai/fusillade#355.

Submission contract

Batchless work uses the OpenAI Responses API extension:

{
  "service_tier": "background",
  "background": true
}

The request is persisted asynchronously, returns 202 Accepted, and is
available through GET /v1/responses/{id}.

File-backed batches use:

{
  "completion_window": "background"
}

Both paths map to the same internal background service tier. Attempts to use
the tier synchronously or on unsupported inference endpoints are rejected with
an actionable error explaining the supported Responses contract.

Submission requires BackgroundInferenceUser. The role is included in the
default roles for newly created platform users; existing users can receive it
through the existing role-management API. Background batches also retain the
normal Batch API permission requirement.

Admission and priority

background_concurrency_limit is a per-model foreground threshold, not a
limit on background work itself.

For a model with an ordinary limit of 100 and a background threshold of 50:

• 70 foreground requests in flight: do not send background work
• 50 foreground requests in flight: do not send background work
• 49 foreground requests in flight: background work becomes eligible

Already-dispatched background requests never count towards model or user
foreground in-flight limits. Background workers read foreground headroom but do
not reserve it, so a later foreground claim can still use the full ordinary
model capacity.

Every background request is sent with Dynamo priority i32::MIN. SLA priorities
are clamped above that reserved value, leaving queued background work behind all
completion-SLA traffic at the node.

Background work is only claimed when:

• the model's latest global filter event is explicitly live
• there is no immediately schedulable foreground request for that model
• foreground in-flight work is below the configured threshold

Background requests have no completion deadline, expiry, deadline escalation,
or deadline-based retry cutoff.

Daemon topology

Background work runs in the same process as the foreground daemon but in
dedicated async claim workers:

• RequestOnly: foreground request worker + background request worker
• BatchOnly: foreground batch worker + background batch worker
• Both: both foreground workers + both background workers

The background workers do not acquire the foreground claim mutex. A large
background database query therefore cannot delay the next customer claim cycle.
The request worker uses request claim cadence/size configuration; the batch
worker uses batch cadence, row-count, and batch-count configuration.

Setting background_concurrency_limit = 0 disables both background workers
without disabling submission, status inspection, cancellation, or pending
counts.

Pending requests and persistence

• background demand is exposed in a separate background pending bucket per model
• batched and batchless background demand share that bucket but are claimed by their modality-specific workers
• ordinary pending buckets and foreground claim queries exclude background rows
• active background rows do not hide remaining eligible background demand
• one atomic Fusillade Arsenal migration adds the tier, no-deadline constraints, archive compatibility, and scheduling indexes

Representing a no-SLA batch makes completion windows, expiry timestamps, and
request-state batch deadlines optional in Fusillade Core, which is why this PR
is marked as breaking.

Verification

• full local Rust CI via just ci rust
• SQLx metadata and migration checksum verification
• background admission, modality, pending-count, authorization, and API contract tests
• end-to-end coverage for batchless and file-backed background dispatch
• regression coverage proving foreground dispatch continues while background work remains in flight
• all GitHub checks passing

[cloudflare-workers-and-pages]

Deploying control-layer with    Cloudflare Pages

Latest commit:	c4bd963
Status:	✅  Deploy successful!
Preview URL:	https://dd6619c3.control-layer.pages.dev
Branch Preview URL:	https://peter-background-tier.control-layer.pages.dev

View logs

[Copilot]

Pull request overview

This PR introduces a new no-SLA background service tier to Fusillade’s scheduler and storage model, enabling spare-capacity processing for both file-backed batches and batchless async requests while preserving strict priority ordering under SLA workloads.

Changes:

• Adds background-tier request/batch APIs and updates core types to support “no deadline” semantics (e.g., batch_expires_at: Option<_>, batch expires_at: Option<_>).
• Implements background claiming in Postgres storage with DB-wide per-model concurrency ceilings, liveness gating, and due-SLA gating.
• Adds concurrent index migrations plus a hardened migration runner (fusillade_arsenal::run_migrations) and wires config/support through dwctl.

Reviewed changes

Copilot reviewed 33 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
fusillade/tests/integration.rs	Adds an end-to-end integration test covering background batches + background batchless requests and priority injection behavior.
fusillade/src/manager/postgres.rs	Wires background_concurrency_limit into Arsenal Postgres storage config mapping.
fusillade/src/lib.rs	Re-exports daemon types (now includes BackgroundDaemon).
fusillade/src/daemon/mod.rs	Adds background claim loop kind, priority injection helpers, and background capacity accounting + metrics.
fusillade/src/daemon/config.rs	Introduces background_concurrency_limit daemon setting with default.
fusillade/README.md	Documents background tier semantics, requirements, and new configuration option.
fusillade-core/src/request/types.rs	Makes request deadline fields optional to support no-SLA background requests.
fusillade-core/src/request/transitions.rs	Updates retry cutoff logic to ignore deadline gating when no SLA exists; adds tests.
fusillade-core/src/request/query.rs	Adds CreateBackgroundInput and clarifies ServiceTierFilter behavior w.r.t. background isolation.
fusillade-core/src/request/mod.rs	Re-exports CreateBackgroundInput.
fusillade-core/src/manager.rs	Extends the Storage trait with background batch creation and background claim APIs (+ capability flag).
fusillade-core/src/batch/mod.rs	Adds background batch input type; makes batch deadline fields optional and adds batch-tier filtering.
fusillade-arsenal/src/postgres.rs	Implements background batch creation + background claiming SQL; isolates background in counts/claims; adds extensive tests.
fusillade-arsenal/src/lib.rs	Adds concurrent-index repair + run_migrations wrapper with advisory-lock serialization; adds test.
fusillade-arsenal/migrations/20260722000000_add_background_service_tier.up.sql	Adds background tier constraints + batch tier column and SLA/background cross-field checks.
fusillade-arsenal/migrations/20260722000000_add_background_service_tier.down.sql	Adds guarded rollback that refuses to drop tier while background data exists.
fusillade-arsenal/migrations/20260722000100_add_pending_background_batchless_index.up.sql	Adds concurrent partial index to support background batchless claim path.
fusillade-arsenal/migrations/20260722000100_add_pending_background_batchless_index.down.sql	Drops the background batchless index concurrently.
fusillade-arsenal/migrations/20260722000200_add_pending_background_batched_index.up.sql	Adds concurrent partial index to support background batched claim path.
fusillade-arsenal/migrations/20260722000200_add_pending_background_batched_index.down.sql	Drops the background batched index concurrently.
fusillade-arsenal/migrations/20260722000300_add_pending_batchless_sla_index.up.sql	Adds concurrent partial index to prevent SLA batchless claim scanning background backlog.
fusillade-arsenal/migrations/20260722000300_add_pending_batchless_sla_index.down.sql	Drops the SLA batchless index concurrently.
fusillade-arsenal/migrations/20260722000400_add_active_sla_counts_index.up.sql	Adds concurrent partial index for active SLA counts used in background headroom computation.
fusillade-arsenal/migrations/20260722000400_add_active_sla_counts_index.down.sql	Drops the active SLA counts index concurrently.
dwctl/src/notifications.rs	Updates batch notification shaping for background batches and adds a unit test.
dwctl/src/lib.rs	Switches Fusillade migrations to fusillade_arsenal::run_migrations.
dwctl/src/config.rs	Adds background_concurrency_limit to daemon config mapping + validation and tests.
dwctl/src/api/handlers/batches.rs	Updates OpenAI-shaped batch responses to avoid fabricating deadlines for background batches; adds tests.
config.yaml	Documents the new background_concurrency_limit setting in config template comments.
.sqlx/query-da8baa843a03ed4d629f0e7d261c1f5f99e8550ee7b1d67343788d82ae1fbe87.json	Adds SQLx metadata for background advisory-lock acquisition query.
.sqlx/query-cdfe08c68420d8a20b3774ddc793bc26e3cb36ae136e72de03c07cecaed1a75c.json	Adds SQLx metadata for background claim query.
.sqlx/query-9bbac1373d23dcc7ca20e80cd3da18dd026c458439287917d94071b51f10a9c0.json	Updates SQLx metadata for batch claim query (background exclusion + optional fields).
.sqlx/query-93ef9d752806dec21bcf3093a862aab809ce4a39b043cd2c1c17378773c84e31.json	Updates SQLx metadata for poller/batch selection query to include background batch fields.
.sqlx/query-7b5c4f3704754a54f9aec1a241c988f8a438853d6ea3cc0d21676f1bfc444fce.json	Updates SQLx metadata to include service_tier when validating batch population guards.
.sqlx/query-7ad6cb48dc2d641dc987551149ca77eddcbe460a62a77b16ec00d5b81f0bab32.json	Adds SQLx metadata for updated batchless claim query excluding background rows.
.sqlx/query-70c1dc1778bb4992c309906214e4269e4c6323b3e208ecc8169fac979b0bb98c.json	Updates SQLx metadata for batch insert including service_tier.
.sqlx/query-6a72b5443378c08f0537bc5054f27fc411160712183b14f79c16fd5218c63a24.json	Updates SQLx metadata nullability for background-compatible schema changes.
.sqlx/query-5df509f23d27e0490b43fd51f261f9e3fc701212f91a87327d72991464886903.json	Updates SQLx metadata nullability for background-compatible schema changes.
.sqlx/query-1f462341d942b5604366c772ca10a8467945f94bfef73e8351209eae2aab21e8.json	Updates SQLx metadata nullability for background-compatible schema changes.
.sqlx/query-0380fc8b4337f9c187cc6efe1a9a4a9d65ccb3da5927d5c679356c95023bb60f.json	Updates SQLx metadata nullability for background-compatible schema changes.
.sqlx/query-dcb02848cb222d2d375e295f3815f4f3e4dce812b0a0426dc6ff7496e9f9bba1.json	Removes stale SQLx metadata that no longer matches updated claim query shape.
.sqlx/query-882a6bbc38799ef64766802f2bdd5ad772ffff3fca8b32087cfaa9ec5eeb9178.json	Removes stale SQLx metadata superseded by updated batch claim query shape.
.github/fixtures/fusillade-migration-sha384.txt	Updates migration fixture hashes for the new Arsenal migrations.

Files not reviewed (10)

• .sqlx/query-088e9a03a4addb97b9146d1d52a0a3cb2191eab8ffeb78c08b5b0dc02776bb9a.json: Generated file
• .sqlx/query-1f462341d942b5604366c772ca10a8467945f94bfef73e8351209eae2aab21e8.json: Generated file
• .sqlx/query-5df509f23d27e0490b43fd51f261f9e3fc701212f91a87327d72991464886903.json: Generated file
• .sqlx/query-6a72b5443378c08f0537bc5054f27fc411160712183b14f79c16fd5218c63a24.json: Generated file
• .sqlx/query-7ad6cb48dc2d641dc987551149ca77eddcbe460a62a77b16ec00d5b81f0bab32.json: Generated file
• .sqlx/query-882a6bbc38799ef64766802f2bdd5ad772ffff3fca8b32087cfaa9ec5eeb9178.json: Generated file
• .sqlx/query-9bbac1373d23dcc7ca20e80cd3da18dd026c458439287917d94071b51f10a9c0.json: Generated file
• .sqlx/query-cdfe08c68420d8a20b3774ddc793bc26e3cb36ae136e72de03c07cecaed1a75c.json: Generated file
• .sqlx/query-da8baa843a03ed4d629f0e7d261c1f5f99e8550ee7b1d67343788d82ae1fbe87.json: Generated file
• .sqlx/query-dcb02848cb222d2d375e295f3815f4f3e4dce812b0a0426dc6ff7496e9f9bba1.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 34 out of 49 changed files in this pull request and generated 3 comments.

Files not reviewed (15)

• .sqlx/query-088e9a03a4addb97b9146d1d52a0a3cb2191eab8ffeb78c08b5b0dc02776bb9a.json: Generated file
• .sqlx/query-1f462341d942b5604366c772ca10a8467945f94bfef73e8351209eae2aab21e8.json: Generated file
• .sqlx/query-2f0199ae0d1a7a2528b18f7495814143b2adfa9c5b3bd7dba4d7837078421a78.json: Generated file
• .sqlx/query-594743bcc97f1e22290a71a362ad2aa7908beac588c90aed843f34f27a5051cf.json: Generated file
• .sqlx/query-5df509f23d27e0490b43fd51f261f9e3fc701212f91a87327d72991464886903.json: Generated file
• .sqlx/query-673f3e821d90c236494d1bf0f93b298d866f307a0e5ce2acdf41617a366bb4b0.json: Generated file
• .sqlx/query-6a72b5443378c08f0537bc5054f27fc411160712183b14f79c16fd5218c63a24.json: Generated file
• .sqlx/query-7ad6cb48dc2d641dc987551149ca77eddcbe460a62a77b16ec00d5b81f0bab32.json: Generated file
• .sqlx/query-83cae930b67a8d80b717561c064809f9f22bb74f16ca5f7508f328de15462568.json: Generated file
• .sqlx/query-882a6bbc38799ef64766802f2bdd5ad772ffff3fca8b32087cfaa9ec5eeb9178.json: Generated file
• .sqlx/query-9711ed1872cfea482fb5ada741800bd22a6a06748d4288cad4866740e0bbac73.json: Generated file
• .sqlx/query-9bbac1373d23dcc7ca20e80cd3da18dd026c458439287917d94071b51f10a9c0.json: Generated file
• .sqlx/query-b6a80dc19ff5a03bce5b91be733221051a86dd9e08abce2b92c7a14a8dafff4b.json: Generated file
• .sqlx/query-dcb02848cb222d2d375e295f3815f4f3e4dce812b0a0426dc6ff7496e9f9bba1.json: Generated file
• .sqlx/query-e7f4ef788ddc7096630ad812db808458b3951e006a0e64fe74e3a6312ca095a5.json: Generated file

Comments suppressed due to low confidence (2)

fusillade/README.md:275

• The README lists a single background_daemon value for the daemon label, but the code uses background_request_daemon and background_batch_daemon. Update the docs so users can match metrics labels to the actual claim loops.
  fusillade/src/daemon/mod.rs:693
• Similarly, the background claim loop dual-emits the unlabeled legacy histograms. Because background claims have different semantics (and can run in parallel with SLA claims), writing the unlabeled series makes it harder to interpret and can break alerts that assume SLA-only claim timings/sizes. Prefer emitting only the daemon-labeled histograms for background loops.