test: retry real-herdr smoke tests on treehouse contention

[mielyemitchell]

Intent

Make the real-herdr/treehouse smoke tests reliable under concurrent fleet load. The known flake was root-caused as environmental treehouse lease contention, not a production code bug. Keep production bin/ behavior unchanged and scope the change to tests. Characterize the contention surface first, then choose the least weakening fix: I chose a bounded whole-script retry because a test-only lock would serialize these two tests with each other but cannot be honored by live fleet treehouse operations. The retry only triggers on fm-spawn.sh's exact treehouse timeout signature ('error: treehouse get did not enter a worktree within 60s; inspect window ...'), so unrelated failures still fail immediately and deterministic regressions still fail after the retry budget. Prove with shellcheck, repeated target test runs, and a scratch held-lease run without touching existing fleet leases or herdr sessions.

What Changed

• Added tests/real-herdr-smoke-retry.sh, a bounded whole-script retry wrapper that re-runs a real-herdr/treehouse smoke test up to a fixed attempt budget, but only when the failure matches fm-spawn.sh's exact treehouse-timeout signature (error: treehouse get did not enter a worktree within 60s; inspect window ...) — any other failure, including a genuine deterministic regression, still fails immediately or after the retry budget is exhausted.
• Wired tests/fm-backend-autodetect-smoke.test.sh and tests/fm-backend-herdr-workspace-per-home-e2e.test.sh to run through the new retry wrapper.
• Updated docs/herdr-backend.md and CONTRIBUTING.md to document the retry behavior and why it exists (environmental treehouse lease contention under concurrent fleet load, not a production code bug).

No changes to production bin/ behavior — this is scoped entirely to test reliability.

Risk Assessment

✅ Low: Test-only change (no bin/ production code touched), the retry is tightly scoped to one exact, verified error string, bounded to 3 attempts, cleans up its own temp dirs on every exit path, preserves the existing no-herdr skip behavior on CI, and shellcheck is clean with no new warnings.

Testing

Ran the two real-herdr/treehouse smoke tests multiple times end-to-end against actual herdr 0.7.2 and treehouse v2.0.0 installs (all passed, no leaked sessions), shellchecked the new retry script and both modified test files at warning+ severity (clean, no regressions vs base), and independently reproduced the treehouse-contention retry logic in an isolated scratch harness that never touched real fleet leases or herdr sessions — confirming it retries only on the exact timeout signature, ignores unrelated failures, and fails after exhausting its bounded retry budget. All evidence supports the stated intent; no issues found.

Evidence: real-herdr autodetect smoke test (2 real runs)

ok - real herdr: fm-spawn.sh auto-detects herdr from HERDR_ENV=1 (no explicit config) and prints the loud notice
ok - real herdr: auto-detected spawn records backend=herdr and herdr_session/workspace/tab/pane fields in meta
ok - real herdr: the auto-detected spawn's launch command actually ran in the herdr pane
ok - real herdr: teardown completes the auto-detected spawn/teardown cycle (meta cleared, pane closed)
(repeated identically across 3 total runs, RC=0 each time)

Evidence: real-herdr workspace-per-home e2e smoke test (real run)

10/10 assertions ok, RC=0, including primary/secondmate workspace isolation and teardown scoping

Evidence: scratch held-lease retry proof (no real herdr/treehouse touched)

case 1 (contention then pass): stderr shows exact signature match -> 'retry: real-herdr smoke hit treehouse lease contention signature (attempt 1/3); retrying' -> passes on attempt 2, RC=0
case 2 (unrelated deterministic failure): fails immediately in 0s, no retry line emitted, RC=1
case 3 (persistent contention): retries attempts 1 and 2, exhausts budget on attempt 3, RC=1 with no further retry

Evidence: shellcheck on new/changed files

shellcheck -S warning tests/real-herdr-smoke-retry.sh tests/fm-backend-autodetect-smoke.test.sh tests/fm-backend-herdr-workspace-per-home-e2e.test.sh -> exit 0 (clean at warning+ severity). Only pre-existing SC1091 info-level notices remain, identical to base commit 22f6dab, so nothing new was introduced.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos

• ℹ️ tests/real-herdr-smoke-retry.sh:15 - The whole-script retry re-execs the entire test file from scratch on a treehouse-contention timeout. fm-backend-herdr-workspace-per-home-e2e.test.sh makes three sequential fm-spawn.sh calls; if contention hits the 2nd or 3rd call, the retry replays the 1st (and 2nd) call's already-passed assertions again, so a retried run's combined stdout contains duplicate 'ok' lines from the earlier, already-successful portion of the previous attempt.
• ℹ️ tests/real-herdr-smoke-retry.sh:24 - Matching is purely on the fm-spawn.sh error string, so a genuine deterministic worktree-mount regression that always produces this exact message (not just lease contention) would also be retried up to 3x (each waiting out the full 60s poll) before finally failing, adding ~2-3 minutes to the failure feedback loop. This is a bounded, intentional tradeoff per the stated design (deterministic regressions still fail after the retry budget), not a functional bug.

✅ **Test** - passed

✅ No issues found.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• shellcheck -S warning on tests/real-herdr-smoke-retry.sh, tests/fm-backend-autodetect-smoke.test.sh, tests/fm-backend-herdr-workspace-per-home-e2e.test.sh (clean, no new issues vs base commit)
• bash tests/fm-backend-autodetect-smoke.test.sh run 3x against real herdr/treehouse — all pass, no leaked herdr sessions afterward (verified via herdr server list)
• bash tests/fm-backend-herdr-workspace-per-home-e2e.test.sh run against real herdr/treehouse — all 10 assertions pass
• Scratch fake-script test sourcing tests/real-herdr-smoke-retry.sh directly to simulate a held-lease/contention failure on attempt 1 then success on attempt 2 — confirms retry fires only on the exact 'treehouse get did not enter a worktree within 60s' signature and recovers
• Scratch fake-script test simulating an unrelated deterministic failure — confirms no retry occurs, fails immediately (0s elapsed)
• Scratch fake-script test simulating persistent contention across all 3 attempts — confirms it retries exactly attempts-1 times then fails for good, matching the bounded-retry design

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.