Worker registry: track active containers, ensure idempotency, handle stale workers

[mfittko]

Scope

Central bookkeeper between event ingestion and container execution. Owns the registry layer only — not webhook receipt, not lease acquisition, not container spawn.

Owns:

• Track active containers: worker ID, target identity (repo+PR), start time, last heartbeat, status
• Idempotency: prevent duplicate spawns for same target+event
• Assignment: assign(target) → worker slot or capacity-rejected signal
• Stale detection: heartbeat-based liveness; dead worker → release assignments, mark target unassigned
• Visibility: status() snapshot of all active workers without polling individual containers
• Graceful shutdown: drain mode — worker marked draining, no new assignments, exits after current loop completes

Flow

Webhook → endpoint (#726) → worker registry (#729)
                                 ├─ already claimed? → skip (idempotent)
                                 ├─ no capacity? → reject
                                 └─ assigned → lease check (#727) → spawn (#723)
                                                  └─ heartbeat loop → registry updates liveness

Acceptance Criteria

• AC1 Registry holds per-worker record: worker ID, target identity, start time, last heartbeat timestamp, status (active | draining | dead)
• AC2 claim(target, eventId) returns already_claimed when same target+eventId already active — no second worker spawned
• AC3 assign(target) returns available worker slot; same target maps to same worker while that worker is active; returns capacity-rejected signal when at limit
• AC4 Worker missing N consecutive heartbeat intervals is marked dead; all its assignments released; target set to unassigned
• AC5 Worker can be set to draining: no new assignments accepted, current loop allowed to complete, transitions to dead on clean exit
• AC6 status() returns consistent snapshot of all tracked workers with current state; no container-by-container queries needed

Definition of Done

• DoD1 Unit tests cover: idempotency (duplicate claim rejected), assignment consistency, heartbeat expiry → dead transition, drain → exit transition
• DoD2 In-memory implementation sufficient for MVP; no external DB dependency introduced
• DoD3 Registry exposed via injectable interface (not a global singleton); tests use fakes/test doubles
• DoD4 Heartbeat interval and stale threshold are named configurable constants (not magic numbers)
• DoD5 All AC items verified by automated tests
• DoD6 Code reviewed and merged to feature branch; CI green

AC / DoD Matrix

AC	Required DoD
AC1 — per-worker record	DoD3 (interface), DoD5 (tests pass)
AC2 — idempotency	DoD1 (unit tests), DoD5
AC3 — assignment	DoD1, DoD3, DoD5
AC4 — stale detection	DoD1, DoD4 (configurable threshold), DoD5
AC5 — drain mode	DoD1, DoD5
AC6 — visibility query	DoD3, DoD5

Non-Goals

• Receiving, validating, or normalizing webhook payloads → GitHub webhook endpoint: validate, normalize, route events to target identity #726
• Acquiring or releasing container leases/locks → Container lease/lock manager: prevent concurrent runs on same target #727
• Polling GitHub for missed or delayed events → Fallback reconcile probe: detect missed webhooks via polling #728
• Spawning, supervising, or tearing down Docker containers → Container lifecycle manager: spawn, wait, teardown for dev-loop sessions #723
• Persistent storage beyond process lifetime (MVP: in-memory is sufficient)
• Full job queue semantics (priority, retry, dead-letter) — K8s handles that later