Test harness: run the suite under both storage engines (engine-aware helpers + engine guard)

[hardbyte]

Motivation

awa's caller-facing contract (enqueue / claim / complete / cancel / admin reads) is engine-invariant — it must behave identically whether the active engine is canonical or queue_storage. But the default integration harness forces canonical (awa_testing::setup::reset_runtime_backend), so the queue-storage implementations of that contract are only covered by the dedicated queue_storage_runtime_test suite, not by the broad integration suite.

This blind spot is how the queue-storage cancel_by_unique_key candidate query ({schema}.leases.unique_key, a column that does not exist on leases) shipped — every cancel_by_unique_key test ran under canonical, so the queue-storage SQL branch was dead code to CI. (Fixed in #359 / 0.6.0-rc.4.)

Experiment result

Running integration_test under a queue-storage activation (AWA_TEST_ENGINE=queue_storage) produces 31 pass / 13 fail, and all 13 failures are test artifacts, in three buckets:

• Harness uses canonical-only raw SQL. TestClient::work_one_in_queue / work_one hand-roll the claim→run→complete loop with raw UPDATE awa.jobs, and several tests manufacture preconditions via raw UPDATE/INSERT awa.jobs. The view's INSTEAD-OF trigger rejects these under queue_storage by design (0A000). The real executor uses the substrate (runtime.store.*), so these helpers never exercised the real engine path.
• Admin-metadata cache assertions (queue_state_counts / dirty-queue bookkeeping) — canonical-trigger-maintained; differ/no-op under queue_storage.
• Test isolation — clean_queue's DELETE FROM awa.jobs WHERE queue = $1 does not release canonical job_unique_claims rows under queue_storage, so claims accumulate across the run and later inserts collide on idx_awa_jobs_unique. Each affected test passes fresh + in isolation.

Net: zero real queue-storage engine bugs surfaced — but the harness can't currently run the broad suite under queue_storage.

Proposed work

1. Engine-aware harness helpers. Make work_one_in_queue / work_one / clean_queue (and any other raw-awa.jobs test helper) route through the real runtime/substrate path when the active engine is queue_storage, instead of raw UPDATE/INSERT/DELETE awa.jobs. Cleanup must release job_unique_claims under both engines.
2. Engine guard for canonical-specific tests. Provide a one-line skip/guard (e.g. an AWA_TEST_ENGINE-aware helper) for tests that assert canonical-only mechanics (admin-metadata caches, *_canonical_* notification tests). These stay canonical-only by design.
3. CI matrix dimension. Once the suite is green under both engines, run the correctness suites twice (engine: [canonical, queue_storage]). Scope the second pass to correctness suites; exclude soak/bench/chaos (already engine-specific).

Outcome

The engine-invariant contract is asserted under both engines in CI, closing the class of "queue-storage branch is dead code to CI" defects (the leases.unique_key family).

A spike branch with the AWA_TEST_ENGINE parameterization and the 31/44 triage exists locally and can seed the implementation.

---

Also: forward-compatibility (old binary against the newest schema)

Related coverage gap on the same axis. A consumer running an old awa version (e.g. 0.5.7, canonical) often keeps running against a database that has already been migrated to the newest schema (V39) — the mixed-version window during a rolling upgrade, and any deployment where migrations run ahead of the worker roll.

Current coverage:

• The default harness always migrates to CURRENT_VERSION and runs canonical, so the whole canonical suite exercises current canonical code against V39.
• migration_test.rs covers fresh-install→V39, triple-run idempotency, V1→V39 data survival, legacy normalization, per-version backfills, and auto-finalize-on-fresh-install (+ its refusal guards).
• The old 11-arg insert_job_compat positional signature is explicitly called against V39 (current Rust passes 12 args), proving old-caller overload resolution still works — but only for the single-insert path.

Gap: no test runs a genuinely old binary against a V39 database. Everything compiles/imports from the current tree; the only mixed-fleet test uses the current Python binding and is #[ignore]/nightly-only. sql_only_storage_upgrade_test.rs drives the engine transition but never claims/completes a job, so it does not assert the canonical work surface post-migration.

Proposed: a version-pinned forward-compat test — enqueue → claim → complete with awa 0.5.7 (pinned dev-dependency, or pip install awa==0.5.7 for the Python path) against a freshly-V39-migrated database. This is the executable form of the existing analytical sign-off (0.5.7 DB-surface audit + migration-safety audit) and the strongest guard for the rolling-upgrade window.