Field report: a week of unattended overnight runs — what held, and three gaps

[Polloinfilzato]

What this is

I ran bmad-loop 0.8.1 unattended overnight, most nights, for about a week on a real project
(~30 stories across several epics, scm.isolation = "worktree", claude adapter, macOS). The goal
was the obvious one: start a run at night, wake up to it done.

To make that survivable I built a supervision layer around the engine — a shell poller plus a
triage protocol, never anything inside the control loop. This is a report on what that exercise
found. I'm not asking you to adopt any of it. Three of the findings are already filed as
separate issues; the rest is offered as operational data.

The headline, including the part that embarrasses me

bmad-loop is well defended inside a run. Nothing defends the layer above it.

I built guards for twelve failure modes. On finally reading the engine source properly, three
of mine turned out to duplicate defences you already had — and yours were better:

• Host suspend freezing the session timeout. I added a wall-clock stall backstop. You already
  co-bound spec.timeout_s with a wall clock (session-end journaled 2h19 after session_timeout_min fires when the session is wedged inside a tool call #157) — and your version is more careful than mine
  would have been, because it lets the wall clock only expire a deadline, never extend it,
  guarding a backward clock step I hadn't considered.
• A long dev turn looking like a stall. I hashed the tmux pane to re-arm my stall clock. Your
  grace window already re-arms on pane-log activity.
• A session dying without a Stop event. I polled liveness from outside. You already score a
  dead window as crashed.

I'm reporting these because they were wrong. A supervisor watching from outside sees symptoms
and infers the mechanism is missing — it usually isn't. That's worth knowing about any external
tooling built against this project, including mine.

What survived that filter is narrow and, I think, real.

The gap that survived: nothing acts above the run

Inside a run you handle crashes, stalls, suspends, timeouts, and you preserve failed work with
keep_failed. But once the engine pauses — or the engine process itself dies — nothing acts.
There's no watchdog and no auto-resume, by design as far as I can tell. gates.notify fires and
then it's on a human.

For an attended run that's exactly right. For the overnight case it means one mechanical hiccup
at 03:00 costs the entire window — which is the whole economic argument for an unattended
orchestrator. Every guard I built that turned out to be necessary lives at that level.

Two concrete consequences, both filed separately:

• notify.desktop = true silently does nothing on macOS and Windows #231 — notify.desktop is Linux-only, so on macOS/Windows the one channel that could reach
  a sleeping human is silently inert. I hit this on a mount-failure deferral: the engine did
  everything right, journalled and notified, and I found out the next morning.
• A worktree_seed directory entry is skipped entirely when any child is tracked #230 — a worktree_seed directory entry is skipped whole when any child is tracked, with
  no diagnostic.

And one I haven't filed yet because it's a design change rather than a defect:

There's no non-destructive way past an escalation under worktree isolation. Phase.ESCALATED
is terminal so resume skips the story; resolve re-drives from scratch; --restore-patch is
rejected under worktree isolation. The work is kept (keep_failed leaves the branch mounted)
— it just can't be continued.

That matters because in my experience most escalations were false positives. The recurring
one: the review session's VC-hygiene gate raises a CRITICAL escalation ("working tree is dirty…")
because the dev pass left artifact files uncommitted — while the same message self-reports
Status: done with a full delivery summary. Mechanical block, not a quality objection. Three
separate stories.

So every recovery I do is a hand-rolled version of one missing command — merge the kept branch,
mark the story done, start a fresh run. Used four times, zero work lost; one of them avoided a
~28M-token re-drive. #172/#173 established the same principle for the dev-retry path; this is its
sibling on the escalation path. Happy to open it as a proper feature request if it's welcome —
CONTRIBUTING.md points at Discord for changes this size, so I haven't just filed it.

On "no LLM in the control loop"

Worth pre-empting, since my layer involves a model. The control loop stays yours: deterministic
Python, no model. Mine sits outside it and only wakes once the deterministic loop has already
stopped and is waiting for a human. What decides when to wake it is a shell script that does no
reasoning — it polls read-only state and, on the first actionable condition, writes a JSON event
and exits. The exit is the wake signal; a process that exits can't fail silently, which
matters because the failure that started all this was an API error that killed a dev turn without
firing the Stop hook at all. It doesn't put a model in the loop; it replaces the human who
would otherwise have to be awake.

Two things worth stealing, neither of them code

Autonomy by class of problem, not by confidence. I ended up with four tiers: mechanical and
known → fix and resume; verified false positive → finalize and continue; real doubt → get a second
model's opinion; no safe path or irreversible risk → stop and wake me. Only the last one wakes a
human, and the boundary is what kind of problem it is, not how sure the supervisor feels.

Finalizing a story without its review creates review debt, and a log line is not a queue. One
story I finalized as a false positive was correct to finalize — but it skipped that story's
automated review, and it then sat marked done for two days like any reviewed story. When the
review finally ran it found a HIGH-severity security defect. The obligation has to land somewhere
that blocks. Worth noting the story most likely to be finalized that way is a security-sensitive
one, because those are the ones already flagged for follow-up.

What I'd offer, if any of it is useful

In order of what I think is worth your time: the escalation path (above); then the two filed
issues; then, only if someone asks, the supervisor itself as a reference implementation. It's
~1,700 lines of shell and a protocol document, three components are macOS-only with one-for-one
Linux equivalents, and two scripts assume my project's sprint layout. Nobody adopts another
project's bash — the transferable parts are the tier ladder and the exit-to-wake shape.

One honest caveat on scope: everything above is from sequential runs. I also built parallel
supervision by starting concurrent --story runs, and that half is reasoned from source and
unit-tested but has never met a real merge-back collision. I wouldn't want the sequential
evidence to lend it credibility it hasn't earned. (The one field datum from that half is already
in #229.)

Happy to expand on any of this, or to leave it as a data point.

[coderabbitai]

🔗 Related PRs

#77 - feat(stories): spec-to-loop engine, adapter, CLI + HITL checkpoints (Phase 2 + 3) [merged]
#86 - feat(resolve): intent-gap patch-restore re-drive + inline-review-layer preflight (re-cut) [merged]
#95 - feat(adapters): post-kill reconcile rescues finished stalled/timeout sessions [merged]
#182 - fix(worktree): degrade post-merge teardown failures instead of crashing the run (#139) [merged]
#219 - feat: graceful stop for bmad-loop runs (finish the in-flight item, then stop — resumable) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[Haven2026]

Additional field data (different project, same 0.8.1 shape)

We hit the same “well defended inside a run; nothing defends the layer above” headline on a multi-epic unattended campaign (worktree, claude dev + codex review, sprint-status).

Two concrete engine gaps we filed separately (not asking you to adopt our outer salvage layer):

1. feat: optional remote Integration Closeout after local unit-merged (done ≠ origin) #270 — optional remote Integration Closeout: unit-merged / story-done is local only; projects whose real gate is origin/main need either in-engine remote closeout or a permanent external salvage.
2. [BUG] Review session timeout always RETRY until max_review_cycles; no salvage/converge exit #271 — review session timeout always RETRY until max_review_cycles; no salvage-if-done / timeout-class exit, so endgame stories thrash review-2/3 while product sits only in a worktree.

Operationally we mirrored your “autonomy by class of problem” idea outside the loop (stop thrash → product-only PR → board cite → never redispatch false greens). Happy to expand if useful; otherwise these two issues stand alone.

[pbean]

Triage summary — keeping this open as a tracking issue for the work it produced, rather than
editing the report itself. Status of the three gaps as of 2026-07-24:

Gap	Status
#231 — notify.desktop inert on macOS/Windows	✅ Fixed (PR #282): native notifier seam — osascript / PowerShell toast — plus an explicit warning when the channel is inert instead of silently doing nothing
#230 — worktree_seed directory entry skipped whole when any child is tracked	✅ Fixed (#235 / PR #257): absent children are now seeded into an existing destination
Escalation continuation (unfiled)	📝 Now filed as #289, as a sub-issue of this report

On the unfiled one: yes, it was welcome — #289 carries over the mechanism (Phase.ESCALATED is
terminal, resume skips, resolve re-drives, --restore-patch rejected under worktree
isolation), the field data (4 uses, 0 work lost, one avoided a ~28M-token re-drive), the false-
positive VC-hygiene pattern, and the #172/#173 precedent. It is labelled needs-design — the open
questions there are what guards the "actually finished" claim and where the resulting review debt
lands, which is your own sharpest point: a log line is not a queue, and the story most likely to
be finalized that way is the security-sensitive one.

The remaining gap, the parallel half, is tracked at #229 (now the parent of #51).

Two things from the report worth stating on the record because they will otherwise be lost in a
long thread:

• The three redundant guards were correctly identified as redundant — the wall-clock stall
  backstop (session-end journaled 2h19 after session_timeout_min fires when the session is wedged inside a tool call #157 already co-binds and only lets the wall clock expire a deadline, never extend
  it), the pane-hash stall re-arm (the grace window already re-arms on pane-log activity), and
  external liveness polling (a dead window is already scored crashed). Reporting those because
  they were wrong is the most useful part of the report for anyone building tooling against this
  project.
• "bmad-loop is well defended inside a run; nothing defends the layer above it" is an accurate
  description of the current design boundary, not a defect — but it is the right sentence to have
  written down somewhere, and Non-destructive continuation past an ESCALATED story under worktree isolation #289 is the first piece of it moving in-engine.

Labelled documentation + area:engine; this stays open as long as it has open children.