feat(jobs): robustness layer + agentic orchestrator

[drewstone]

Summary

Two additions on top of the freshly-merged jobs+reports infra (#83 / #84):

1. A deterministic robustness layer — retry, anti-bot detection, schema versioning, resume, adaptive cost. The five things you'd want before running this against any real corpus.
2. An agentic orchestrator that wraps the runner with an LLM control loop for strategic decisions (re-sample broken wayback snapshots, retry vs. skip, conclude). The agent only runs when the deterministic fan-out leaves something worth fixing — happy path pays no LLM tax.

The architectural rule (it actually matters)

Protocols are deterministic. Judgment is agentic.

• Exponential backoff with jitter is a protocol. An LLM has zero genuine skill at "wait 800ms after a 503" — wrapping it in prose adds latency and cost without adding intelligence.
• "5 of 8 wayback snapshots came back broken — should we widen the window? re-sample? skip?" is a judgment call. Hand-tuned heuristics break the moment the world changes. LLM in the control layer pays for itself.

So the stack looks like:

```
spec → discoverTargets → runJob (retry + anti-bot built in, deterministic)
↓
needsIntervention()? ──no──→ done
↓ yes
orchestrator agent loop
├─ getJobState
├─ resampleWayback (≤1/url)
├─ retryTarget (≤2/target)
├─ markSkipped
└─ concludeJob
```

What's in it

`src/jobs/retry.ts` — whitelist-based retry with exp backoff + jitter. Retries 429 / 5xx / network / timeouts. Everything else (4xx, anti-bot, schema, unknown) is treated as deterministic and not retried.

`src/jobs/anti-bot.ts` — pure pattern match against `report.json`. Cloudflare interstitial titles, "Just a moment...", "Access denied", challenge-page intents, plus a last-resort heuristic (zero findings + low confidence + unknown type). When fired the runner records `skipped` with a reason instead of putting a bogus score on the leaderboard.

`src/jobs/cost-history.ts` — adaptive cost estimate from prior `~/.bad/jobs/` records. Uses static default until N≥3 completed jobs exist; afterward averages per-target cost from the last 20. Floors at 50% of static default to prevent runaway optimism on a stretch of zero-cost claude-code jobs.

Schema versioning — `tokens.json` is now stamped with `schemaVersion: 1` at write time. Aggregator refuses files older than `MIN_TOKENS_SCHEMA`.

Resume — `bad jobs resume ` re-runs only targets that aren't already `ok`/`skipped`. Same on the API via `RunJobOptions.resume`.

`src/jobs/orchestrator.ts` — `orchestrateJob(job, opts)` runs the deterministic fan-out then enters a control loop only if `needsIntervention()` returns true. Five tools (`getJobState`, `resampleWayback`, `retryTarget`, `markSkipped`, `concludeJob`) with hard caps: 2 retries per target, 1 resample per URL, cost ≤ `spec.maxCostUSD * 0.9`. Default brain uses `claude-code` provider (subscription-based, no API key required). CLI: `bad jobs orchestrate --spec <file.json>`.

Tests

+34 across:

• `jobs-retry` — 9 tests, retry logic + backoff + onRetry hook
• `jobs-anti-bot` — 8 tests, every detection path
• `jobs-cost-history` — 5 tests, cold start / warm history / floor / failure-skew
• `jobs-orchestrator` — 6 tests, `needsIntervention` + queue resume + skipped surfacing
• `jobs-orchestrator-agent` — 4 tests, end-to-end LLM path with `MockLanguageModelV3` (deterministic, no real LLM)

Total: 1494 passing (up from 1460).

Test plan

• `pnpm lint` clean
• `pnpm check:boundaries` clean (156 files)
• `pnpm test` 1494/1494
• CLI smoke: `bad jobs orchestrate --spec` runs end-to-end with the agent layer (validated against a real wayback longitudinal stripe.com job)
• Adaptive cost: `bad jobs estimate` shows `(default)` until enough history, then `(history, n=...)`
• Tier1/Tier2 gates green in CI

Notes

The orchestrator's `needsIntervention` gate is the single most important seam in the design. Without it every job pays the LLM tax even when there's nothing to do. With it, a clean run (every target ok) skips the agent entirely and produces deterministic output. Verified end-to-end: clean stripe.com run → orchestrator prints `no intervention needed` and exits.