fix(orchestrator): handle push events directly and break probe deadlock

- Handle WebhookDispatch::Push by calling handle_push() directly
  instead of enqueuing to dispatcher. Push events now bypass the
  reconcile_tick timeout, so build-runner spawns immediately.
- Set probed_at at START of probe_all() so is_stale() returns false
  even if tick is cancelled. Breaks the deadlock where probes never
  complete, circuit breakers never open, probes re-run every tick.
- Update circuit breakers inline as each probe result arrives.
- Process group kill via kill -9 -<pgid> kills opencode children.
- Runtime orphan reaping: pkill .opencode run every 12 ticks.

Refs #1721

Author: Alex

.docs/design-adf-reconciler-timeout.md

@@ -0,0 +1,156 @@
+# Design & Implementation Plan: ADF Reconciler Timeout and Process Leak Fix
+
+**Date:** 2026-05-19
+**Status:** Phase 2 design
+**Research:** `.docs/research-adf-reconciler-timeout.md`
+
+## 1. Summary of Target Behaviour
+
+After implementation:
+
+| Outcome | Before | After |
+|---------|--------|-------|
+| Build-runner spawns on push | Never (queue stuck behind probes) | Within 1-2 s of webhook receipt |
+| reconcile_tick timeout | Every tick | Only when TTL expires and probes re-run |
+| Orphaned opencode processes | 60+ accumulated over 50 min | Zero orphans — killed on timeout or periodic cleanup |
+| Circuit breaker for timeout providers | Never opens (deadlock) | Opens after 5 consecutive failures within cooldown window |
+
+## 2. Key Invariants and Acceptance Criteria
+
+| # | Criterion | Verification |
+|---|-----------|-------------|
+| AC1 | `handle_push()` called directly from `handle_webhook_dispatch()` for Push events, NOT via dispatcher queue | Code review: match arm in `handle_webhook_dispatch` calls `self.handle_push(task)` |
+| AC2 | `provider_health.probed_at` updated at START of `probe_all()`, not end | Code review; `is_stale()` returns false after first probe attempt |
+| AC3 | Probe process group (not just bash) killed on timeout | Manual: insert `sleep 300 &` into probe action, verify grandchild also killed |
+| AC4 | Orphaned opencode processes reaped in `adf-cleanup.sh` at startup AND periodically during runtime | Manual: verify `pkill` matches running opencode processes |
+| AC5 | Circuit breaker opens for providers that time out 5 times consecutively | Log output shows "probe skipped: circuit breaker open" after 5th timeout |
+| AC6 | `reconcile_tick` completes within timeout when probes are skipped (circuit breaker open) | No "reconcile_tick exceeded timeout" in logs after fix |
+| AC7 | Dispatch queue still drained for non-Push tasks (ReviewPr, AutoMerge, etc.) | Other task types still processed in Step 17 |
+
+## 3. High-Level Design
+
+### 3.1 Change 1: Direct push handling (bypass dispatcher)
+
+**Current:** `handle_webhook_dispatch()` at `lib.rs:3714` → enqueues `DispatchTask::Push` → reconciled later in Step 17
+
+**New:** Match arm for `WebhookDispatch::Push` calls `self.handle_push(task).await` directly instead of enqueuing.
+
+**Why this is safe:**
+- `handle_push()` already takes `&mut self` (same as `handle_webhook_dispatch`)
+- `handle_push()` already checks if build-runner is active (line 2985) — prevents double-spawn
+- Other task types (ReviewPr, AutoMerge, PostMergeTestGate) still use dispatcher — no change to PR workflow
+
+### 3.2 Change 2: Set probed_at at start (break circuit breaker deadlock)
+
+**Current:** `probe_all()` at `provider_probe.rs:200` sets `self.probed_at = Some(Instant::now())` ONLY after ALL probes complete
+
+**New:** Move `self.probed_at = Some(Instant::now())` to line 98 (start of `probe_all()`)
+
+**Effect:** After a single probe attempt (even if cancelled), `is_stale()` returns false for TTL duration. Probes only re-run every TTL seconds, not every tick. The reconciler gets TTL seconds of clean ticks to drain dispatcher.
+
+**Additional:** Also update circuit breaker state for each probe result AS THEY ARRIVE, not only after all complete. This means even if `probe_all()` is cancelled partway through, the breakers for completed probes are already updated.
+
+### 3.3 Change 3: Process group kill
+
+**Current:** `probe_single()` at `provider_probe.rs:638`: `kill -9 <pid>` — kills bash, orphans opencode
+
+**New:** `kill -9 -<pgid>` — kills entire process group including opencode and its children
+
+**Mechanism:** After spawning the child, call `nix::unistd::getpgid(pid)` to get the process group ID. On timeout, use negative PID to signal the entire group. The `nix` crate is already in the dependency tree.
+
+### 3.4 Change 4: Runtime orphan reaping
+
+**Current:** `adf-cleanup.sh` runs at startup only
+
+**New:** Add a step 0.5 to `reconcile_tick()` that runs `pkill -9 -f '\.opencode run'` every N ticks (e.g., every 12 ticks = ~1 minute at 5 s interval)
+
+**Alternative:** Use `tokio::process::Command::new("pkill")` to run the cleanup from Rust.
+
+## 4. File/Module Change Plan
+
+| File | Action | What Changes | Risk |
+|------|--------|-------------|------|
+| `lib.rs:3714-3738` | Modify | `WebhookDispatch::Push` arm: call `handle_push(task)` instead of `dispatcher.enqueue()` | Low — push events no longer go through dispatcher |
+| `provider_probe.rs:98` | Modify | Add `self.probed_at = Some(Instant::now())` at start of `probe_all()` | Low — single line |
+| `provider_probe.rs:156-188` | Modify | Update circuit breakers inside the `for task in tasks` loop as results arrive, not after | Medium — changes control flow |
+| `provider_probe.rs:637-641` | Modify | Replace `kill -9 <pid>` with `kill -9 -<pgid>` | Low — requires `nix` crate |
+| `lib.rs:5621` | Add | New step 0.5: `pkill -9 -f '\.opencode run'` every 12 ticks | Low — new step in reconcile_tick |
+
+## 5. Step-by-Step Implementation Sequence
+
+### Step 1: Direct push handling (highest value, lowest risk)
+**File:** `lib.rs` lines 3714-3738
+
+Change the `WebhookDispatch::Push` match arm from:
+```rust
+self.dispatcher.enqueue(dispatcher::DispatchTask::Push { ... });
+```
+to:
+```rust
+let task = dispatcher::DispatchTask::Push { ... };
+if let Err(e) = self.handle_push(task).await {
+    warn!(error = %e, "handle_push failed in webhook handler");
+}
+```
+
+**Verification:** Push to Gitea, check if build-runner spawns within seconds.
+
+### Step 2: Set probed_at at start
+**File:** `provider_probe.rs` line 98
+
+Add after the function signature: `self.probed_at = Some(Instant::now());`
+
+Also move the circuit breaker update loop (`for result in &results`) to inside the `for task in tasks` loop, updating each breaker as results arrive.
+
+### Step 3: Process group kill
+**File:** `provider_probe.rs` lines 634-642
+
+1. After `let child = match tokio::process::Command::new("bash")...`, capture pgid:
+   ```rust
+   let pgid = child.id().and_then(|pid| {
+       nix::unistd::getpgid(Some(nix::unistd::Pid::from_raw(pid as i32))).ok()
+   });
+   ```
+2. On timeout, use: `kill -9 -<pgid>`
+
+### Step 4: Runtime orphan reaping
+**File:** `lib.rs` around line 5621
+
+Add as Step 0.5 (before rate limit cleaning):
+```rust
+if self.tick_count % 12 == 0 {
+    let _ = std::process::Command::new("pkill")
+        .arg("-9")
+        .arg("-f")
+        .arg(".opencode run")
+        .spawn();
+}
+```
+
+## 6. Testing Strategy
+
+| Criterion | Test Method | Location |
+|-----------|------------|----------|
+| AC1: Direct push handling | Manual: push to Gitea, check build-runner log | Bigbox |
+| AC2: probed_at at start | Unit: call probe_all(), cancel, check is_stale() | New test in provider_probe.rs |
+| AC3: Process group kill | Unit: spawn shell that spawns grandchild, kill pgid, verify grandchild dead | New test in provider_probe.rs |
+| AC5: Circuit breaker opens | Integration: let ADF run 5+ ticks with timeout providers, verify breaker state | Bigbox |
+| AC6: reconciler completes | System: monitor logs for absence of "exceeded timeout" | Bigbox |
+| AC7: Other tasks still dispatched | Existing: PR review and auto-merge tests still pass | Existing test suite |
+
+## 7. Risk Review
+
+| Risk | Mitigation | Residual |
+|------|------------|----------|
+| Push events double-dispatched (webhook + dispatcher) | Change 1 removes `dispatcher.enqueue()` for Push — no double-enqueue | Low |
+| probe_all() with probed_at at start prevents re-probing when provider recovers | TTL ensures re-probing after TTL expires — normal behaviour | None |
+| kill -9 -<pgid> may not work on all systems | `nix::unistd::getpgid` returns EPERM on some kernels — fall back to kill -9 <pid> if pgid unavailable | Low |
+| Runtime pkill may match legitimate opencode processes | `-f '\.opencode run'` pattern is specific to probe commands | Low |
+
+## 8. Questions for Reviewer
+
+1. **Should `handle_push()` call from webhook handler be conditional on `reconcile_tick` completion?** Currently proposed: always call directly. Alternative: only call directly if `provider_health.is_stale()` is true (indicating probes are blocking ticks).
+
+2. **Should the TTL for provider probes be increased from current value to reduce probe frequency?** Currently probes run every `self.ttl` (configurable in config). A longer TTL means less frequent probe re-attempts.
+
+3. **Should the `DispatchTask::Push` variant remain in the enum, or be removed?** Since push events are now handled directly, the variant is dead code. Should we clean it up?

.docs/research-adf-reconciler-timeout.md

@@ -0,0 +1,141 @@
+# Research Document: ADF Reconciler Tick Timeout and Process Leak
+
+**Date:** 2026-05-19
+**Status:** Phase 1 research (thorough code review)
+
+## 1. Problem Restatement and Scope
+
+### Problem
+
+The ADF orchestrator logs `reconcile_tick exceeded timeout, forcing continuation` every 90 s. Build-runner never spawns automatically on push events. Sixty-plus orphaned `opencode` processes accumulate during runtime.
+
+### IN scope
+
+- Why reconcile_tick always times out
+- Why build-runner never dispatches
+- Why opencode processes leak
+- The circuit breaker deadlock
+
+### OUT of scope
+
+- Changing tick interval configuration
+- Removing providers from KG routing rules
+- General memory optimisation
+
+## 2. System Elements — Trace of a Push Event
+
+Read from source at `crates/terraphim_orchestrator/src/`:
+
+### 2.1 Push webhook arrival
+
+1. **Gitea push** → webhook handler at `webhook.rs:498` (`handle_push_event`)
+2. Parses payload, creates `WebhookDispatch::Push`, sends via `dispatch_tx` (mpsc channel)
+3. Arrives in event loop at `lib.rs:1280` — tokio task forwards to main mpsc `loop_tx`
+4. Main loop `lib.rs:1301` receives `LoopEvent::Webhook(dispatch)`
+5. Calls `handle_webhook_dispatch()` at `lib.rs:3432`
+6. Match arm at `lib.rs:3714` — **enqueues `DispatchTask::Push` into `self.dispatcher`** — does NOT spawn build-runner here
+7. Sends `LoopEvent::Tick` to trigger reconciliation
+
+### 2.2 Tick event — where dispatch SHOULD happen
+
+8. Main loop receives `LoopEvent::Tick` at `lib.rs:1314`
+9. Drains queued events (webhooks, schedules, alerts), coalescing stale ticks
+10. Calls `reconcile_tick()` wrapped in `tokio::time::timeout(90s)` at `lib.rs:1336`
+11. `reconcile_tick()` has 18 numbered steps at `lib.rs:5618`
+
+### 2.3 reconcile_tick step ordering is the problem
+
+| Step | Line | What | Duration |
+|------|------|------|----------|
+| 0 | 5621 | Clean rate limits, poll timeouts | Fast |
+| 1 | 5629 | Poll agent exits | Fast |
+| 2 | 5632 | Restart safety agents | Fast |
+| 3 | 5635 | Check cron schedules | Fast |
+| 4 | 5638 | Drain output events | Fast |
+| 5-12 | 5644-5717 | Nightwatch, handoff, budget, flows, mentions, KG reload | Fast |
+| **13** | **5719** | **Provider probes** — `probe_all(kg_router).await` | **Blocks > 90 s** |
+| 14 | 5748 | Update `last_tick_time` | Fast |
+| 15-16 | 5752-5764 | Telemetry, budget persist | Fast |
+| **17** | **5795** | **Dispatch queue drain** — `handle_push()` for each Push task | **Never reached** |
+| 18 | 5859 | Poll pending reviews | Never reached |
+
+The tick is always cancelled at Step 13, so Steps 14-18 never execute.
+
+### 2.4 Why probes take > 90 s
+
+`probe_all()` at `provider_probe.rs:98`:
+
+1. Iterates KG routing rules, collects unique `(provider, model, action)` triples
+2. For each, spawns `tokio::spawn(async { probe_single(...) })`
+3. Awaits all results sequentially
+
+`probe_single()` at `provider_probe.rs:465`:
+
+1. Spawns `bash -c "opencode run -m <model> --format json echo hello"`
+2. Waits up to 120 s for completion (`Duration::from_secs(120)` at line 542)
+3. On timeout: `kill -9 <pid>` (line 638)
+4. Returns `ProbeResult { status: Timeout }`
+
+**Two connected problems:**
+
+**Problem A — Timing mismatch:** The probe's internal 120 s timeout exceeds reconcile_tick's 90 s timeout. Even a single slow probe causes the entire tick to timeout.
+
+**Problem B — Circuit breaker deadlock:** `probed_at` and circuit breaker state are only updated at lines 199-200 of `probe_all()`, AFTER all probes complete. When `probe_all()` is cancelled by the reconcile_tick timeout:
+
+- `probed_at` is never set → `is_stale()` always returns `true` → probes re-run on every tick
+- Circuit breakers are never updated → providers never transition to `Open` state → probes always include timeout providers
+- Each tick spawns a new set of probe tasks → tasks accumulate
+
+### 2.5 Process leak mechanism
+
+When `reconcile_tick()` is cancelled at 90 s, the `probe_all().await` is dropped. But `tokio::spawn` tasks are NOT cancelled — they continue running independently.
+
+Each probe task:
+1. `bash -c "opencode run ..."` is spawned
+2. The 120 s internal timeout fires
+3. `kill -9 <pid>` kills bash
+4. opencode (child of bash) becomes orphaned (PPID=1)
+5. orphan spawns child processes (`gtr`, `cached-context`, `sentrux`)
+6. All continue running indefinitely
+
+Over 50 minutes (~33 tick cycles × 3 timeout providers ≈ 99 leaked opencode instances), 60+ orphans observed.
+
+### 2.6 Why cleanup is insufficient
+
+`adf-cleanup.sh` already has `pkill -9 -f '\.opencode run'` — but it only runs at startup. Orphans that accumulate during runtime are never cleaned up.
+
+## 3. Constraints and Their Implications
+
+### C1: Probe timeout (120 s) exceeds reconcile timeout (90 s)
+- **Effect:** Multiple probes running in `tokio::spawn` will always exceed 90 s if any provider is slow/unavailable
+- **Implication:** reconcile_tick is ALWAYS cancelled; dispatch never happens
+
+### C2: Circuit breaker never opens for timeout providers
+- **Effect:** `probed_at` and breaker state only updated after `probe_all()` completes — which never happens
+- **Implication:** `is_stale()` always true → probes always run with full set of providers → self-perpetuating
+
+### C3: tokio::spawn handles not cancelled on parent drop
+- **Effect:** Probe tasks continue running after reconcile_tick is cancelled
+- **Implication:** Accumulating orphans drive memory/swapping, further slowing the reconciler (death spiral)
+
+### C4: Dispatch drain positioned after probes
+- **Effect:** Push events are enqueued but never dequeued because Step 17 is never reached
+- **Implication:** Build-runner never spawns automatically
+
+## 4. Risks
+
+| Risk | Severity | Cause |
+|------|----------|-------|
+| OOM killer terminates critical processes | Critical | 60+ orphaned opencode consuming 20+ GB |
+| Push events silently dropped | High | Dispatcher queue never drained |
+| System becomes unresponsive | Medium | Load 23+, swap 3.9/4 GB exhausted |
+
+## 5. Simplification Opportunities
+
+1. **Handle push events directly** — Don't use the dispatch queue for push events. Call `handle_push()` directly from `handle_webhook_dispatch()`. This completely decouples build-runner dispatch from probe health.
+
+2. **Set probed_at immediately** — Move `self.probed_at = Some(Instant::now())` to the START of `probe_all()`. This prevents `is_stale()` from returning true for the TTL duration, giving the reconciler breathing room.
+
+3. **Kill process groups** — Use `kill -9 -<pgid>` to kill opencode children, not just bash parent.
+
+4. **Add periodic orphan reaping** — Add a cleanup step to reconcile_tick that kills opencode processes older than 2 minutes.

crates/terraphim_orchestrator/src/lib.rs

@@ -1704,7 +1704,9 @@ impl AgentOrchestrator {
         let truncated: Vec<_> = learnings.into_iter().take(max_entries).collect();
         let ids: Vec<String> = truncated.iter().map(|l| l.id.clone()).collect();
 
-        let mut section = String::from("## Prior Lessons\n\nLessons learned from previous agent runs. Apply relevant insights:\n\n");
+        let mut section = String::from(
+            "## Prior Lessons\n\nLessons learned from previous agent runs. Apply relevant insights:\n\n",
+        );
         for l in &truncated {
             section.push_str(&format!(
                 "- [{}] {} (trust: {}, verified {}x)\n",
@@ -3725,16 +3727,19 @@ impl AgentOrchestrator {
                     after_sha = %after_sha,
                     pusher = %pusher_login,
                     files = files_changed.len(),
-                    "webhook: enqueuing Push dispatch task"
+                    "webhook: dispatching Push directly"
                 );
-                self.dispatcher.enqueue(dispatcher::DispatchTask::Push {
+                let task = dispatcher::DispatchTask::Push {
                     project,
                     ref_name,
                     before_sha,
                     after_sha,
                     pusher_login,
                     files_changed,
-                });
+                };
+                if let Err(e) = self.handle_push(task).await {
+                    warn!(error = %e, "handle_push failed in webhook handler");
+                }
             }
         }
     }
@@ -5618,6 +5623,18 @@ impl AgentOrchestrator {
     async fn reconcile_tick(&mut self) {
         let tick_start = Instant::now();
 
+        // 0. Reap orphaned opencode probe processes every 12 ticks (~1 min
+        // at 5 s interval). Processes that survive kill -9 -<pgid> in
+        // probe_single may become PPID=1 orphans; periodic cleanup
+        // prevents accumulation.
+        if self.tick_count % 12 == 0 {
+            let _ = std::process::Command::new("pkill")
+                .arg("-9")
+                .arg("-f")
+                .arg(".opencode run")
+                .spawn();
+        }
+
         self.provider_rate_limits.clean_expired();
         self.retry_counts
             .retain(|_, (_, ts)| ts.elapsed() < Duration::from_secs(3600));