Skip to content

fix(herdr): serialize workspace ensure so parallel first spawns share one workspace#733

Open
raphaelthiel wants to merge 3 commits into
kunchenguid:mainfrom
raphaelthiel:fix/herdr-workspace-ensure-race
Open

fix(herdr): serialize workspace ensure so parallel first spawns share one workspace#733
raphaelthiel wants to merge 3 commits into
kunchenguid:mainfrom
raphaelthiel:fix/herdr-workspace-ensure-race

Conversation

@raphaelthiel

Copy link
Copy Markdown

Intent

Fix a live-fired race in the experimental herdr session backend: when firstmate spawns several crewmates in parallel as a home's FIRST spawns of the session, fm_backend_herdr_workspace_ensure ran find-before-create with no cross-process serialization, so every parallel spawn minted its own workspace with the identical home label (reproduced 2026-07-19 on real herdr 0.7.4/protocol 16, macOS aarch64: three batch-spawned crewmates produced three 'firstmate' workspaces; herdr enforces no label uniqueness so nothing failed, the fleet just fragmented across duplicate spaces). Upstream issue: #730. Deliberate decisions the captain made: serialize only the find-or-create window with an atomic mkdir lock (portable - macOS ships no flock) placed under TMPDIR rather than the home's state/ dir so unit tests and CI checkouts never write into the repo; key the lock by cksum of FM_HOME plus session plus home label, the same axes that scope the workspace itself, so different homes/sessions never contend; holder records its pid and a waiter breaks the lock only for a provably dead holder (kill -0 fails), a lock dir whose pid file has not appeared yet is waited out within the bounded 12s acquire budget; FAIL-OPEN on acquire timeout (warn on stderr, proceed unlocked) is intentional - a lock anomaly must never block dispatch, the worst case deliberately degrades to the pre-fix race, mirroring the repo's 'quota trouble never blocks dispatch' posture; no trap-based release inside the lib function (could clobber caller traps), explicit release on every return path instead. Three new unit tests against the stateful fake (parallel ensures mint exactly one shared workspace; deterministic hold-then-release proof that a waiter adopts the winner with zero creates; a provably dead holder's lock is broken and released), plus the mandated empirical incident section in docs/herdr-backend.md with the exact commands and output observed. bin/fm-lint.sh passes under the pinned shellcheck 0.11.0; the full tests/fm-backend-herdr.test.sh suite is green. Pre-fix duplicate workspaces need no migration - they self-heal through normal teardown.

What Changed

  • fm_backend_herdr_workspace_ensure serializes its find-or-create window with a new fm_backend_herdr_workspace_lock_acquire/_release pair in bin/backends/herdr.sh: an atomic mkdir lock under ${TMPDIR:-/tmp} (portable, no flock on macOS), keyed by cksum of $FM_HOME + session + home label, so parallel first spawns of the same home mint exactly one shared workspace instead of duplicates. The holder records its pid; a waiter breaks a provably dead holder's lock atomically (rename-aside before removal), and the lock is fail-open: on acquire timeout (12s budget) or an uncreatable lock dir (e.g. broken TMPDIR, detected after three consecutive misses) the spawn proceeds unlocked with a stderr warning — a lock anomaly never blocks dispatch. Release is explicit on every return path, no trap-based cleanup.
  • Three new tests in tests/fm-backend-herdr.test.sh against the stateful fake: concurrent first spawns mint one workspace, a waiter adopts a live holder's workspace with zero creates, and a dead holder's lock is broken and released.
  • docs/herdr-backend.md gains the empirical incident section (2026-07-19, herdr 0.7.4/protocol 16, macOS aarch64: three batch-spawned crewmates produced three firstmate workspaces) with the observed commands/output, root cause, fix mechanics, and the note that pre-fix duplicates self-heal through normal teardown.

Risk Assessment

✅ Low: The round-2 commit correctly implements both prior findings (atomic mv-based dead-holder break and fast fail-open after three consecutive lock-dir misses) without introducing new races, keeps all three unit tests and the documented semantics consistent, and preserves every intent criterion including fail-open and break-only-provably-dead-holder.

Testing

Baseline-Suite und volle tests/fm-backend-herdr.test.sh grün (104 ok, inkl. der 3 neuen Lock-Tests); zusätzlich eine manuelle End-to-End-Demo, die den Duplicate-Workspace-Race am Base-Commit reproduziert (3 creates, fragmentierte Container), mit dem Fix genau einen geteilten Workspace zeigt und das fail-open-Verhalten bei kaputtem TMPDIR (Warnung, Spawn läuft weiter) belegt; Transcripts und Demo-Skript als Evidenz gesichert. Kein visuelles Artefakt, da die Änderung eine reine Shell-Backend-Bibliothek ohne UI-Oberfläche betrifft — CLI-Transcripts sind die reviewer-sichtbare Oberfläche.

Evidence: Vorher/Nachher-Race-Demo (Base vs. Fix vs. Fail-open)

=== VORHER (b2bf95f, ohne Lock): 3 parallele First-Spawns derselben Home === {"workspace_id":"w1","label":"firstmate"} {"workspace_id":"w3","label":"firstmate"} workspace-create-Aufrufe im CLI-Log: 3 spawn 1 -> default:w1 spawn 2 -> default:w3 spawn 3 -> default:w1 === NACHHER (Fix, mkdir-Lock) === {"workspace_id":"w1","label":"firstmate"} workspace-create-Aufrufe im CLI-Log: 1 spawn 1 -> default:w1 spawn 2 -> default:w1 spawn 3 -> default:w1 === FAIL-OPEN (Fix, kaputtes TMPDIR) === stderr: warning: herdr workspace-ensure lock not acquired; proceeding unlocked (concurrent first spawns may mint duplicate workspaces) Ergebnis-Container: default:w1

=== VORHER (b2bf95f, ohne Lock): 3 parallele First-Spawns derselben Home ===
  herdr workspace list | jq -c '.workspaces[] | {workspace_id, label}':
    {"workspace_id":"w1","label":"firstmate"}
    {"workspace_id":"w3","label":"firstmate"}
  workspace-create-Aufrufe im CLI-Log: 3
  Container, die die 3 Spawns aufgeloest haben:
    spawn 1 -> default:w1
    spawn 2 -> default:w3
    spawn 3 -> default:w1

=== NACHHER (Fix, mkdir-Lock): 3 parallele First-Spawns derselben Home ===
  herdr workspace list | jq -c '.workspaces[] | {workspace_id, label}':
    {"workspace_id":"w1","label":"firstmate"}
  workspace-create-Aufrufe im CLI-Log: 1
  Container, die die 3 Spawns aufgeloest haben:
    spawn 1 -> default:w1
    spawn 2 -> default:w1
    spawn 3 -> default:w1

=== FAIL-OPEN (Fix, kaputtes TMPDIR): Lock unerreichbar => warnen + trotzdem spawnen ===
  stderr: warning: herdr workspace-ensure lock not acquired; proceeding unlocked (concurrent first spawns may mint duplicate workspaces)
  Ergebnis-Container: default:w1
  Workspaces danach: [{"workspace_id":"w1","label":"firstmate"}]
Evidence: Voller Testlauf tests/fm-backend-herdr.test.sh (104 ok)
ok - fm_backend_herdr_version_check: accepts the current protocol (14)
ok - fm_backend_herdr_version_check: refuses an old protocol loudly
ok - fm_backend_herdr_version_check: refuses loudly when herdr is not installed
ok - fm_backend_herdr_workspace_label: a primary home (no marker) resolves to 'firstmate'
ok - fm_backend_herdr_workspace_label: a secondmate home (.fm-secondmate-home) resolves to '2ndmate-<id>'
ok - fm_backend_herdr_workspace_label: trims whitespace around the marker's secondmate id
ok - fm_backend_herdr_workspace_label: an empty marker file falls back to the primary label 'firstmate'
ok - fm_backend_herdr_workspace_label: two different secondmate homes get two different, non-colliding labels
ok - fm_backend_herdr_cli: sets HERDR_SESSION AND appends a trailing --session flag on every call
ok - fm_backend_herdr_container_ensure: version-gates, starts the server, ensures the firstmate workspace, echoes session:workspace_id + the seeded default tab id
ok - fm_backend_herdr_container_ensure: reuses an existing firstmate workspace without recreating it, and reports no seeded default tab (adopted, not created)
ok - fm_backend_herdr_container_ensure: workspace create passes --no-focus
ok - fm_backend_herdr_container_ensure: creates the workspace under the SECONDMATE home's own label, not 'firstmate'
ok - fm_backend_herdr_create_task: prunes exactly the seeded default tab container_ensure identified, once the first real task tab exists
ok - herdr repeated spawn/teardown: one persistent firstmate workspace reused, zero orphans, default tab pruned, create ran once
ok - fm_backend_herdr_workspace_ensure: 3 concurrent first spawns mint exactly one workspace and share it
ok - fm_backend_herdr_workspace_ensure: waits on a live lock holder, then adopts the winner's workspace
ok - fm_backend_herdr_workspace_ensure: breaks a provably dead holder's lock and proceeds
ok - fm_backend_herdr_create_task: an ADOPTED workspace's pre-existing tab is never pruned (the created-vs-adopted gate)
ok - fm_backend_herdr_create_task: the label-collision startup-workspace scenario (2026-07-02 incident) leaves the captain's live tab untouched
ok - fm_backend_herdr_workspace_prune_seeded_default_tab: refuses to close the seeded default tab when its pane reports a working agent (defense in depth)
ok - no bin/ jq filter names a --arg/--argjson variable after a jq reserved keyword
ok - fm_backend_herdr_create_task: refuses a duplicate tab label (herdr's own tab create has no uniqueness check)
ok - fm_backend_herdr_create_task: a same-labeled tab with a live (even idle) registered agent still refuses exactly as before
ok - fm_backend_herdr_create_task: scans every same-labeled tab and refuses if any duplicate is live
ok - fm_backend_herdr_create_task: closes and replaces a same-labeled tab whose pane is dead (pane_not_found)
ok - fm_backend_herdr_create_task: closes and replaces a same-labeled tab whose pane is alive but hosts no registered agent (a restored plain shell)
ok - fm_backend_herdr_create_task: closes every confirmed same-labeled husk only after creating the replacement
ok - fm_backend_herdr_create_task: refuses success when a preexisting husk tab remains after replacement
ok - fm_backend_herdr_create_task: refuses (fail-safe) rather than guessing when the duplicate's agent state cannot be classified confidently
ok - fm_backend_herdr_create_task: creates the replacement tab BEFORE closing the husk tab, never the reverse
ok - fm_backend_herdr_create_task: creates a tab and parses tab_id/pane_id from the JSON response, prunes nothing when no seeded tab id is given
ok - fm_backend_herdr_create_task: tab create passes --no-focus
ok - fm_backend_herdr_workspace_find: matches only THIS home's own label among several coexisting workspaces
ok - fm_backend_herdr_list_live: scoped to this home's own workspace, never a sibling home's
ok - fm_backend_herdr_parse_target: splits '<session>:<pane_id>' on the FIRST colon (pane_id itself contains one)
ok - fm_backend_herdr_normalize_key: Enter/Escape/C-c map to herdr's verified enter/escape/ctrl+c
ok - fm_backend_herdr_capture: calls 'pane read <pane> --source recent --lines N' with the session set
ok - fm_backend_herdr_capture: works around the verified small-N '--lines' bug by over-fetching and trimming locally
ok - fm_backend_herdr_capture: ensures the session and preserves pane read failure
ok - fm_backend_herdr_send_key: normalizes the key and targets the right pane
ok - fm_backend_herdr_kill: calls pane close and stays best-effort on failure
ok - fm_backend_herdr_current_path: reads pane foreground_cwd (the live running process), not the frozen creation-time cwd
ok - fm_backend_herdr_busy_state: working -> busy
ok - fm_backend_herdr_busy_state: done -> idle, blocked -> idle (surfaced like a stale pane, not suppressed as busy)
ok - fm_backend_herdr_busy_state: unparseable/absent agent state reports unknown, the regex-fallback cue
ok - fm_backend_herdr_composer_state: a bare '❯' composer row reads empty
ok - fm_backend_herdr_composer_state: the ghost placeholder text reads empty, not pending
ok - fm_backend_herdr_composer_state: real composer text reads pending
ok - fm_backend_herdr_composer_state: a slash-command popup's argument-hint placeholder still reads pending (the incident fix)
ok - fm_backend_herdr_composer_state: reports unknown when the pane cannot be captured
ok - fm_backend_herdr_composer_state: reports unknown for bare shell prompts with no composer row
ok - fm_backend_herdr_composer_state: a native idle Pi separator composer reads empty
ok - fm_backend_herdr_composer_state: real Pi composer text remains pending
ok - fm_backend_herdr_composer_state: an incomplete lower Pi separator cannot inherit a stale empty row
ok - fm_backend_herdr_composer_state: Pi separators never authorize working, non-Pi, unreadable, or over-tall targets
ok - fm_backend_herdr_composer_state: a real-claude unbordered '❯' prompt row (no border box in view) reads empty
ok - fm_backend_herdr_composer_state: a real-claude unbordered '❯ <text>' prompt row reads pending
ok - fm_backend_herdr_composer_state: a live unbordered prompt row below a stale bordered decorative box still wins (not misread as the box's own row)
ok - fm_backend_herdr_composer_state: claude's dim prompt-suggestion ghost (the overnight wedge shape) reads empty
ok - fm_backend_herdr_composer_state: real typed text on the same claude prompt row still reads pending
ok - fm_backend_herdr_composer_state: grok's dark-truecolor placeholder (the TRUECOLOR gap) reads empty
ok - fm_backend_herdr_composer_state: grok's real bright typed input still reads pending
ok - fm_backend_herdr_composer_state: a real-codex unbordered '›' prompt row reads empty
ok - fm_backend_herdr_composer_state: a faint real-codex ghost suggestion reads empty
ok - fm_backend_herdr_composer_state: non-faint codex prompt text still reads pending
ok - fm_backend_herdr_wait_for_working: reports 'busy' immediately on the first poll, without spending the rest of the budget
ok - fm_backend_herdr_wait_for_working: a slow transition landing on a later sample within one window is still caught (robust against the 'slow transition' failure direction)
ok - fm_backend_herdr_wait_for_working: spreads six samples across the full budget endpoint without a final trailing sleep
ok - fm_backend_herdr_send_text_submit: applies the herdr minimum confirmation budget before polling agent-state
ok - fm_backend_herdr_wait_for_working: reports 'idle' (readable, genuinely not yet working) when 'busy' never appears
ok - fm_backend_herdr_wait_for_working: reports 'unknown' (a hard read failure, not a timing race) only when EVERY poll in the window fails
ok - fm_backend_herdr_wait_for_working: treats blocked as submit-active for confirmation without changing watcher busy-state semantics
ok - fm_backend_herdr_send_text_submit: reports 'empty' once agent_status reports working after one Enter, without ever reading the composer
ok - fm_backend_herdr_send_text_submit: reports 'pending' when agent_status never reports working after retried Enters (swallowed)
ok - fm_backend_herdr_send_text_submit: a slash-command popup's placeholder fill on Enter #1 never flips agent_status to working, so it does not short-circuit as submitted; Enter #2 is retried and lands it
ok - fm_backend_herdr_send_text_submit: a post-Enter blocked state confirms delivery without retrying into the prompt
ok - fm_backend_herdr_send_text_submit: preexisting working is not accepted as submit proof when the composer still holds the message
ok - fm_backend_herdr_send_text_submit: confirms submission via native agent-state alone, immune to a codex-style dynamic idle-tip composer that would have misread as 'pending' under the old composer-based confirmation
ok - fm_backend_herdr_composer_state: a faint real-codex dynamic idle-tip composer row reads empty
ok - fm_backend_composer_state (herdr): the pre-injection empty-box guard still refuses a genuinely non-empty composer, unaffected by the submit-confirmation change
ok - fm_backend_herdr_send_text_submit: a slow transition landing on a later sample within one Enter's budget is confirmed WITHOUT sending a needless extra Enter
ok - fm_backend_herdr_send_text_submit: reports 'send-failed' when the literal send-text call itself errors
ok - fm_backend_herdr_send_text_submit: reports 'unknown' when the post-Enter agent-get read fails (never retries past an unreadable target)
ok - fm_backend_validate: herdr is a known backend (P2)
ok - fm_backend_busy_state: tmux (no native primitive) always reports unknown, preserving the P1 regex-only path
error: unknown backend 'bogus' (known: tmux herdr zellij orca cmux)
ok - fm_backend_composer_state dispatches tmux/herdr/orca to their named classifiers, unknown for zellij/unrecognized backends
ok - fm-peek/fm-send: explicit stale targets matching metadata use the recorded backend
ok - fm_backend_herdr_normalize_event routes through the shared record with an empty from_status
ok - fm_backend_herdr_escalation_marker keys the dedupe marker exactly like the watcher's .stale-<key>
ok - fm_backend_herdr_apply_transition: blocked dedupe starts only after explicit commit
ok - fm_backend_herdr_apply_transition: a working edge clears the marker so the next ->blocked re-escalates
ok - fm_backend_herdr_clear_transition removes task-owned dedupe state
ok - fm_backend_herdr_apply_transition: idle/done (defer) and unknown/empty (fallback) take no fast action
ok - fm_backend_herdr_wait_transition: a home with no herdr panes falls back to polling (rc 2)
ok - fm_backend_herdr_wait_transition: below-capability protocol/schema falls back to polling (rc 2)
ok - fm_backend_herdr_wait_transition: reconnect level-reconcile returns an uncommitted blocked pane
ok - fm_backend_herdr_wait_transition: subscribes before reconnect level-reconcile
ok - fm_backend_herdr_wait_transition: a still-blocked, already-escalated pane is not re-delivered on reconnect
ok - fm_backend_herdr_wait_transition: a streamed ->blocked edge returns the record sub-poll
ok - fm_backend_herdr_wait_transition: streamed working clears the marker, idle/done are deferred (clean timeout)
ok - fm_backend_herdr_wait_transition: a reader/subscribe failure falls back to polling (rc 2)
ok - fm_backend_herdr_wait_transition: Bash 3.2-safe bad-ack path closes fd 9 and removes its FIFO
ok - fm_backend_herdr_wait_transition: stock macOS Bash clean timeout closes fd 9 and returns 1
Evidence: Demo-Skript der Race-Reproduktion
#!/usr/bin/env bash
# Vorher/Nachher-Demo: reproduziert den Duplicate-Workspace-Race aus Issue #730
# gegen den Stateful-Fake aus tests/fm-backend-herdr.test.sh.
# - "base"  = bin/backends/herdr.sh @ b2bf95f (vor dem Fix)
# - "fixed" = bin/backends/herdr.sh @ Arbeitskopie (mit mkdir-Lock)
# Der Race wird deterministisch gemacht, indem der Fake `workspace list`
# (das find-Fenster) um 0.5s verlangsamt - exakt das Fenster, das der Lock
# serialisiert.
set -euo pipefail
WORKTREE=$1
WORK=$(mktemp -d /tmp/fm-herdr-demo.XXXXXX)
trap 'rm -rf "$WORK"' EXIT

BASE_ROOT="$WORK/base-root"
mkdir -p "$BASE_ROOT/bin/backends"
cp "$WORKTREE/bin/fm-composer-lib.sh" "$WORKTREE/bin/fm-transition-lib.sh" "$BASE_ROOT/bin/"
git -C "$WORKTREE" show b2bf95f:bin/backends/herdr.sh > "$BASE_ROOT/bin/backends/herdr.sh"

# make_herdr_statefake aus der Test-Suite wiederverwenden (Zeilen 78-159)
sed -n '78,159p' "$WORKTREE/tests/fm-backend-herdr.test.sh" > "$WORK/fake-helper.sh"
# shellcheck disable=SC1091
. "$WORK/fake-helper.sh"

run_scenario() {  # <name> <root>
  local name=$1 root=$2 dir fb i
  dir="$WORK/$name"
  mkdir -p "$dir/locks"
  : > "$dir/log"
  fb=$(make_herdr_statefake "$dir")
  # find-Fenster kuenstlich weiten, damit der Race ohne Lock sicher feuert
  sed -i '' 's/"workspace list")/"workspace list") sleep 2;/' "$fb/herdr"
  for i in 1 2 3; do
    ( PATH="$fb:$PATH" FM_HERDR_LOG="$dir/log" FM_FAKE_HERDR_STATE="$dir/state.json" \
      HERDR_SESSION=default FM_HOME="$root" TMPDIR="$dir/locks" \
      bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_container_ensure /proj' "$root" \
      > "$dir/out.$i" 2>"$dir/err.$i" ) &
  done
  wait
  echo "  herdr workspace list | jq -c '.workspaces[] | {workspace_id, label}':"
  jq -c '.workspaces[] | {workspace_id, label}' "$dir/state.json" | sed 's/^/    /'
  echo "  workspace-create-Aufrufe im CLI-Log: $(grep -c $'\x1f''workspace'$'\x1f''create' "$dir/log" || true)"
  echo "  Container, die die 3 Spawns aufgeloest haben:"
  for i in 1 2 3; do
    printf '    spawn %s -> %s\n' "$i" "$(cut -d$'\t' -f1 "$dir/out.$i")"
  done
}

echo "=== VORHER (b2bf95f, ohne Lock): 3 parallele First-Spawns derselben Home ==="
run_scenario base "$BASE_ROOT"
echo
echo "=== NACHHER (Fix, mkdir-Lock): 3 parallele First-Spawns derselben Home ==="
run_scenario fixed "$WORKTREE"
echo
echo "=== FAIL-OPEN (Fix, kaputtes TMPDIR): Lock unerreichbar => warnen + trotzdem spawnen ==="
dir="$WORK/failopen"; mkdir -p "$dir"; : > "$dir/log"
fb=$(make_herdr_statefake "$dir")
out=$( PATH="$fb:$PATH" FM_HERDR_LOG="$dir/log" FM_FAKE_HERDR_STATE="$dir/state.json" \
  HERDR_SESSION=default FM_HOME="$WORKTREE" TMPDIR="$WORK/does/not/exist" \
  bash -c '. "$0/bin/backends/herdr.sh"; fm_backend_herdr_container_ensure /proj' "$WORKTREE" 2>"$dir/err" )
echo "  stderr: $(cat "$dir/err")"
echo "  Ergebnis-Container: $(printf '%s' "$out" | cut -d$'\t' -f1)"
echo "  Workspaces danach: $(jq -c '[.workspaces[]|{workspace_id,label}]' "$dir/state.json")"

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅
  • ⚠️ bin/backends/herdr.sh:342 - Dead-holder lock break is a non-atomic check-then-remove: two waiters that both read the same dead holder pid can interleave so that waiter A breaks the lock, re-acquires it via mkdir and writes its pid, and then waiter B's already-decided rm -rf &#34;$lock&#34; deletes A's freshly acquired lock; B's next mkdir then succeeds and both processes enter the find-or-create window simultaneously, reproducing the duplicate-workspace race the fix targets (only after a crashed holder plus tight interleave). Fix mechanically by claiming the break atomically, e.g. mv &#34;$lock&#34; &#34;$lock.breaking.$$&#34; 2&gt;/dev/null &amp;&amp; rm -rf &#34;$lock.breaking.$$&#34; — directory rename is atomic, so only one waiter wins the break; failure of the mv just falls through to the sleep/retry path, preserving the fail-open posture.
  • ℹ️ bin/backends/herdr.sh:335 - If mkdir fails for a reason other than the lock already existing (e.g. TMPDIR missing or unwritable, or a path-hostile character in the derived label), the acquire loop still spins through all 120 iterations before failing open, so every spawn in such an environment pays the full 12s latency budget. An early exit after a failed mkdir when [ ! -d &#34;$lock&#34; ] (no lock actually present) would fail open immediately in that broken-environment case without changing behavior in the contended case.

🔧 Fix: atomic dead-holder lock break and fast fail-open on broken TMPDIR
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • Baseline (vorab grün): for t in tests/*.test.sh; do bash &#34;$t&#34;; done
  • bash tests/fm-backend-herdr.test.sh — 104/104 ok, inkl. test_workspace_ensure_concurrent_first_spawns_mint_one_workspace, test_workspace_ensure_waits_for_live_lock_holder_then_adopts, test_workspace_ensure_breaks_provably_dead_lock_holder
  • Manuelle Vorher/Nachher-Demo (race-demo-script.sh): 3 parallele fm_backend_herdr_container_ensure gegen den Stateful-Fake mit geweitetem Find-Fenster — Base b2bf95f: 3 workspace creates + doppelte Workspaces; Fix: 1 create, alle Spawns teilen default:w1
  • Manuelle Fail-open-Prüfung: fm_backend_herdr_container_ensure mit nicht existentem TMPDIR — stderr-Warnung, Spawn läuft trotzdem durch (1 Workspace)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Raphael Thiel added 3 commits July 19, 2026 11:39
… one workspace

fm_backend_herdr_workspace_ensure ran find-before-create with no
serialization between concurrent spawn processes, so N parallel FIRST
spawns of a home each found no workspace yet and each minted its own
identically-labeled one (live-fired 2026-07-19: three batch-spawned
crewmates produced three 'firstmate' workspaces; herdr enforces no label
uniqueness, so the fleet silently fragmented instead of failing).

Serialize the find-or-create window with an atomic mkdir lock under
TMPDIR, keyed by a cksum of FM_HOME plus session and home label - the
same axes that scope the workspace itself. The holder records its pid; a
waiter breaks the lock only for a provably dead holder. Fail-open by
design: on acquire timeout the spawn proceeds unlocked after one stderr
warning, so the worst case is exactly the pre-fix race, never a blocked
spawn.

Evidence and mechanism: docs/herdr-backend.md 'Incident (2026-07-19)'.

Fixes kunchenguid#730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant