fix(herdr): serialize workspace ensure so parallel first spawns share one workspace#733
Open
raphaelthiel wants to merge 3 commits into
Open
fix(herdr): serialize workspace ensure so parallel first spawns share one workspace#733raphaelthiel wants to merge 3 commits into
raphaelthiel wants to merge 3 commits into
Conversation
added 3 commits
July 19, 2026 11:39
… one workspace fm_backend_herdr_workspace_ensure ran find-before-create with no serialization between concurrent spawn processes, so N parallel FIRST spawns of a home each found no workspace yet and each minted its own identically-labeled one (live-fired 2026-07-19: three batch-spawned crewmates produced three 'firstmate' workspaces; herdr enforces no label uniqueness, so the fleet silently fragmented instead of failing). Serialize the find-or-create window with an atomic mkdir lock under TMPDIR, keyed by a cksum of FM_HOME plus session and home label - the same axes that scope the workspace itself. The holder records its pid; a waiter breaks the lock only for a provably dead holder. Fail-open by design: on acquire timeout the spawn proceeds unlocked after one stderr warning, so the worst case is exactly the pre-fix race, never a blocked spawn. Evidence and mechanism: docs/herdr-backend.md 'Incident (2026-07-19)'. Fixes kunchenguid#730
… on broken TMPDIR
…eak, fast fail-open
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix a live-fired race in the experimental herdr session backend: when firstmate spawns several crewmates in parallel as a home's FIRST spawns of the session, fm_backend_herdr_workspace_ensure ran find-before-create with no cross-process serialization, so every parallel spawn minted its own workspace with the identical home label (reproduced 2026-07-19 on real herdr 0.7.4/protocol 16, macOS aarch64: three batch-spawned crewmates produced three 'firstmate' workspaces; herdr enforces no label uniqueness so nothing failed, the fleet just fragmented across duplicate spaces). Upstream issue: #730. Deliberate decisions the captain made: serialize only the find-or-create window with an atomic mkdir lock (portable - macOS ships no flock) placed under TMPDIR rather than the home's state/ dir so unit tests and CI checkouts never write into the repo; key the lock by cksum of FM_HOME plus session plus home label, the same axes that scope the workspace itself, so different homes/sessions never contend; holder records its pid and a waiter breaks the lock only for a provably dead holder (kill -0 fails), a lock dir whose pid file has not appeared yet is waited out within the bounded 12s acquire budget; FAIL-OPEN on acquire timeout (warn on stderr, proceed unlocked) is intentional - a lock anomaly must never block dispatch, the worst case deliberately degrades to the pre-fix race, mirroring the repo's 'quota trouble never blocks dispatch' posture; no trap-based release inside the lib function (could clobber caller traps), explicit release on every return path instead. Three new unit tests against the stateful fake (parallel ensures mint exactly one shared workspace; deterministic hold-then-release proof that a waiter adopts the winner with zero creates; a provably dead holder's lock is broken and released), plus the mandated empirical incident section in docs/herdr-backend.md with the exact commands and output observed. bin/fm-lint.sh passes under the pinned shellcheck 0.11.0; the full tests/fm-backend-herdr.test.sh suite is green. Pre-fix duplicate workspaces need no migration - they self-heal through normal teardown.
What Changed
fm_backend_herdr_workspace_ensureserializes its find-or-create window with a newfm_backend_herdr_workspace_lock_acquire/_releasepair inbin/backends/herdr.sh: an atomicmkdirlock under${TMPDIR:-/tmp}(portable, no flock on macOS), keyed by cksum of$FM_HOME+ session + home label, so parallel first spawns of the same home mint exactly one shared workspace instead of duplicates. The holder records its pid; a waiter breaks a provably dead holder's lock atomically (rename-aside before removal), and the lock is fail-open: on acquire timeout (12s budget) or an uncreatable lock dir (e.g. brokenTMPDIR, detected after three consecutive misses) the spawn proceeds unlocked with a stderr warning — a lock anomaly never blocks dispatch. Release is explicit on every return path, no trap-based cleanup.tests/fm-backend-herdr.test.shagainst the stateful fake: concurrent first spawns mint one workspace, a waiter adopts a live holder's workspace with zero creates, and a dead holder's lock is broken and released.docs/herdr-backend.mdgains the empirical incident section (2026-07-19, herdr 0.7.4/protocol 16, macOS aarch64: three batch-spawned crewmates produced threefirstmateworkspaces) with the observed commands/output, root cause, fix mechanics, and the note that pre-fix duplicates self-heal through normal teardown.Risk Assessment
✅ Low: The round-2 commit correctly implements both prior findings (atomic mv-based dead-holder break and fast fail-open after three consecutive lock-dir misses) without introducing new races, keeps all three unit tests and the documented semantics consistent, and preserves every intent criterion including fail-open and break-only-provably-dead-holder.
Testing
Baseline-Suite und volle tests/fm-backend-herdr.test.sh grün (104 ok, inkl. der 3 neuen Lock-Tests); zusätzlich eine manuelle End-to-End-Demo, die den Duplicate-Workspace-Race am Base-Commit reproduziert (3 creates, fragmentierte Container), mit dem Fix genau einen geteilten Workspace zeigt und das fail-open-Verhalten bei kaputtem TMPDIR (Warnung, Spawn läuft weiter) belegt; Transcripts und Demo-Skript als Evidenz gesichert. Kein visuelles Artefakt, da die Änderung eine reine Shell-Backend-Bibliothek ohne UI-Oberfläche betrifft — CLI-Transcripts sind die reviewer-sichtbare Oberfläche.
Evidence: Vorher/Nachher-Race-Demo (Base vs. Fix vs. Fail-open)
=== VORHER (b2bf95f, ohne Lock): 3 parallele First-Spawns derselben Home === {"workspace_id":"w1","label":"firstmate"} {"workspace_id":"w3","label":"firstmate"} workspace-create-Aufrufe im CLI-Log: 3 spawn 1 -> default:w1 spawn 2 -> default:w3 spawn 3 -> default:w1 === NACHHER (Fix, mkdir-Lock) === {"workspace_id":"w1","label":"firstmate"} workspace-create-Aufrufe im CLI-Log: 1 spawn 1 -> default:w1 spawn 2 -> default:w1 spawn 3 -> default:w1 === FAIL-OPEN (Fix, kaputtes TMPDIR) === stderr: warning: herdr workspace-ensure lock not acquired; proceeding unlocked (concurrent first spawns may mint duplicate workspaces) Ergebnis-Container: default:w1Evidence: Voller Testlauf tests/fm-backend-herdr.test.sh (104 ok)
Evidence: Demo-Skript der Race-Reproduktion
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed ✅
bin/backends/herdr.sh:342- Dead-holder lock break is a non-atomic check-then-remove: two waiters that both read the same dead holder pid can interleave so that waiter A breaks the lock, re-acquires it via mkdir and writes its pid, and then waiter B's already-decidedrm -rf "$lock"deletes A's freshly acquired lock; B's next mkdir then succeeds and both processes enter the find-or-create window simultaneously, reproducing the duplicate-workspace race the fix targets (only after a crashed holder plus tight interleave). Fix mechanically by claiming the break atomically, e.g.mv "$lock" "$lock.breaking.$$" 2>/dev/null && rm -rf "$lock.breaking.$$"— directory rename is atomic, so only one waiter wins the break; failure of the mv just falls through to the sleep/retry path, preserving the fail-open posture.bin/backends/herdr.sh:335- If mkdir fails for a reason other than the lock already existing (e.g. TMPDIR missing or unwritable, or a path-hostile character in the derived label), the acquire loop still spins through all 120 iterations before failing open, so every spawn in such an environment pays the full 12s latency budget. An early exit after a failed mkdir when[ ! -d "$lock" ](no lock actually present) would fail open immediately in that broken-environment case without changing behavior in the contended case.🔧 Fix: atomic dead-holder lock break and fast fail-open on broken TMPDIR
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"Baseline (vorab grün):for t in tests/*.test.sh; do bash "$t"; donebash tests/fm-backend-herdr.test.sh— 104/104 ok, inkl.test_workspace_ensure_concurrent_first_spawns_mint_one_workspace,test_workspace_ensure_waits_for_live_lock_holder_then_adopts,test_workspace_ensure_breaks_provably_dead_lock_holderManuelle Vorher/Nachher-Demo (race-demo-script.sh): 3 parallelefm_backend_herdr_container_ensuregegen den Stateful-Fake mit geweitetem Find-Fenster — Base b2bf95f: 3 workspace creates + doppelte Workspaces; Fix: 1 create, alle Spawns teilendefault:w1Manuelle Fail-open-Prüfung:fm_backend_herdr_container_ensuremit nicht existentem TMPDIR — stderr-Warnung, Spawn läuft trotzdem durch (1 Workspace)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.