test(scripts): cover busy → timeout path via direct Redis manipulation

Script 15 exercises mutateWithFallback's safety-net cap: HSET-forces the
entry into each of the three busy-triggering states (DRAINING, FAILED,
materialised=true) and verifies the mutation API returns 503 within the
~2s safetyNetMs window. Also asserts the wait is bounded — fails if the
response comes back faster than 1s (would imply busy wasn't hit) or
slower than 5s (would imply the wait is unbounded).

The remaining uncovered slice of busy is the "drainer succeeds mid-wait
and pgMutation runs" branch, which requires injecting a PG row from
outside the webapp during the wait window. Documented as unit-test-only.

Author: d-cs

scripts/mollifier-challenge/15-busy-timeout.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# 15 — mutateWithFallback "busy" path → safety-net timeout → 503.
+# When mutateSnapshot returns busy (entry DRAINING / FAILED /
+# materialised=true) the helper polls the PG writer for ~2s, then
+# 503s if the row never materialises. We force the busy state by
+# HSET-ing the entry hash directly, then call a mutation endpoint
+# and expect 503 within ~2.5s.
+#
+# Required: drainer OFF (so the entry stays in whatever state we set).
+#          : redis-cli or `docker exec redis redis-cli`.
+
+source "$(dirname "$0")/00-lib.sh"
+
+header "mutateWithFallback busy → safety-net timeout"
+
+if [[ -z "${REDIS_CLI:-}" ]]; then
+  if command -v redis-cli >/dev/null 2>&1; then
+    REDIS_CLI=(redis-cli)
+  elif docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^redis$'; then
+    REDIS_CLI=(docker exec -i redis redis-cli)
+  else
+    fail "no redis-cli; set REDIS_CLI='docker exec -i NAME redis-cli'"
+    summary
+  fi
+else
+  read -ra REDIS_CLI <<< "$REDIS_CLI"
+fi
+
+# Test each of the three "busy" trigger states. Each one buffers a fresh
+# run, mutates the entry into the target state via redis-cli, then calls
+# a mutation API and expects 503 (not 5xx, not 200 — explicit timeout).
+test_busy_state() {
+  local label=$1 hset_args=("${@:2}")
+
+  BUFFERED_ID=$(capture_buffered_run_id)
+  if [[ -z "$BUFFERED_ID" ]]; then
+    fail "[$label] could not buffer a run"
+    return
+  fi
+
+  # Verify the entry is initially mutable.
+  api POST "/api/v1/runs/$BUFFERED_ID/tags" '{"tags":["pre-busy"]}'
+  if ! last_status_ok; then
+    fail "[$label] pre-busy tags status=$(cat "$WORK/last.status")"
+    return
+  fi
+
+  # Force the busy state.
+  "${REDIS_CLI[@]}" HSET "mollifier:entries:$BUFFERED_ID" "${hset_args[@]}" >/dev/null
+  info "[$label] HSET ${hset_args[*]} on $BUFFERED_ID"
+
+  # Fire a mutation. Should 503 after ~2s of polling.
+  local t0 t1
+  t0=$(date +%s)
+  api POST "/api/v1/runs/$BUFFERED_ID/tags" '{"tags":["during-busy"]}'
+  t1=$(date +%s)
+  local elapsed=$((t1 - t0))
+  local status
+  status=$(cat "$WORK/last.status")
+
+  if [[ "$status" == "503" ]]; then
+    pass "[$label] returned 503 in ${elapsed}s (expected ~2s)"
+  else
+    fail "[$label] expected 503, got $status in ${elapsed}s — body=$(last_body | head -c 200)"
+  fi
+
+  if (( elapsed >= 1 && elapsed <= 5 )); then
+    pass "[$label] wait time in [1, 5]s window (safetyNetMs=2000)"
+  else
+    fail "[$label] wait time ${elapsed}s outside expected [1, 5]s window"
+  fi
+}
+
+header "busy state 1: status=DRAINING"
+test_busy_state "DRAINING" status DRAINING
+
+header "busy state 2: status=FAILED"
+test_busy_state "FAILED" status FAILED
+
+header "busy state 3: materialised=true"
+test_busy_state "materialised" materialised true
+
+summary

scripts/mollifier-challenge/README.md

@@ -46,11 +46,12 @@ export TASK_ID=hello-world
 | 12 | `12-state3-replay.sh` | OFF + redis-cli | Direct Redis HSET status=FAILED to manufacture state 3 (Q2). Replay still produces a fresh run. |
 | 13 | `13-resume-parent-guard.sh` | OFF | triggerAndWait with an idempotency key matching a buffered run produces a fresh PG run (B6b guard). |
 | 14 | `14-dashboard-routes.sh` | OFF + session cookie | D1 cancel, D2 replay, D3 idempotencyKey reset via the `/resources/...` dashboard routes (session-cookie auth). |
+| 15 | `15-busy-timeout.sh` | OFF + redis-cli | Force entry into DRAINING / FAILED / materialised=true via direct HSET; verify the mutation API returns 503 after the ~2s safety-net wait. |
 
 **Toggling the drainer:** restart the webapp with `TRIGGER_MOLLIFIER_DRAINER_ENABLED=1`
 for scripts that need it. Scripts 05 and 06 are the only ones that need it ON.
 
-**Script 12 prerequisites:** sets `REDIS_CLI` env var, or has `redis-cli` on PATH,
+**Script 12 / 15 prerequisites:** sets `REDIS_CLI` env var, or has `redis-cli` on PATH,
 or a docker container named `redis` reachable via `docker exec`.
 
 **Script 14 prerequisites:** session-cookie value (the `__session` cookie from a
@@ -72,9 +73,10 @@ These behaviours exist in the implementation but aren't covered by the bash
 suite. They're documented here so future readers know what's checked elsewhere
 vs what's genuinely uncovered.
 
-- **`mutateWithFallback` "busy" wait-and-bounce path.** Triggers only when an
-  entry is in DRAINING state — racy from bash since only the drainer can flip
-  the status. Covered by unit tests in `apps/webapp/test/mollifierMutateWithFallback.test.ts`.
+- **`mutateWithFallback` busy → PG-row-arrives-mid-wait path.** Script 15 covers
+  the timeout side of busy. The "drainer succeeds while the API is waiting"
+  side requires injecting a PG row mid-flight; covered by unit tests in
+  `apps/webapp/test/mollifierMutateWithFallback.test.ts`.
 - **Buffer outage / fail-open.** Stopping Redis takes down the run engine,
   queue, and locks too — the system can't function for a meaningful end-to-end
   scenario. Covered by unit tests that pass a buffer that throws.