feat: add deploy-sync automation and crew dispatch guardrails#673
Open
inivosdigital wants to merge 64 commits into
Open
feat: add deploy-sync automation and crew dispatch guardrails#673inivosdigital wants to merge 64 commits into
inivosdigital wants to merge 64 commits into
Conversation
Owner
|
Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch. When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again. Noted for firstmate#673 at |
added 29 commits
July 17, 2026 10:04
Two treehouse worktrees of the same project share a leaf directory basename, so Compose's default project-naming collides across them on container/volume/network names. fm-spawn.sh now derives a stable name from the worktree's own pool-slot path and drops it at .treehouse-compose-project (git-excluded, like the other turn-end pointer files), recorded in meta as compose_project=. fm-brief.sh tells ship/scout crewmates to consume it on every docker compose invocation and to avoid fixed host ports as a second collision vector.
…against backgrounded server call
…r-compose note to fm-brief.sh header comment
…compose isolation test
Claude crewmates and secondmates are separate claude processes launched by fm-spawn.sh; they never inherit the primary's .claude/settings.local.json, so they only compact at the hard context wall instead of proactively. Add CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000 to the claude launch template. The var is capped at the model's real context window, so a fixed 200000 lands auto-compaction around 200k regardless of whether the spawned model's window is 200k or (as verified live for --model sonnet on this account, which carries the 1M-context beta) ~1M. The primary's own two-var form (WINDOW=1000000 + PCT_OVERRIDE=20) assumes a 1M window and is not safe to copy blindly for a smaller-window model/account combination. grok, codex, opencode, and pi launches are left untouched: grok's binary is not verified to honor Claude Code env vars, and the other three ignore them harmlessly but should not carry a var meant for a harness they aren't.
Bump CLAUDE_CODE_AUTO_COMPACT_WINDOW from 200000 to 300000 for claude crewmates and secondmates, matching the primary session's own updated target. The override is capped at the model's real context window, so it is a no-op below 300000 and only takes effect above it; today's standard --model sonnet crewmate spawn resolves to the ~1M-context beta, well above the new threshold. Updated the fm-spawn.sh header comment, launch_template() comment, and the harness-adapters SKILL.md fact to describe this conditional behavior instead of the "regardless of window" framing that only held at the old 200000 value.
content_in_default() only ever diffed a branch against the fetched refs/remotes/origin/$name, so a no-mistakes-mode task whose branch was merged into the LOCAL default branch by hand (e.g. after its upstream PR was closed unmerged) was never recognized as landed and teardown refused it. It now also checks the local refs/heads/$name ref.
…tests content_in_default()'s diff-based comparison (from local main's own prior rewrite) no longer depends on git merge-tree --write-tree, so the two tests that used to skip themselves on older git now run and pass here.
Extends config/crew-dispatch.json's schema with model/effort/ultracode axes per the resource-tiering design: docs/examples/crew-dispatch.json gains rules for the firstmate-repo supervision-backbone carve-out, safety-critical/security/migration work (opus/xhigh, ultracode-flagged), and architectural/product tradeoffs (opus/xhigh). Adds the four hardened guardrails as mechanical checks rather than prose aspirations: - bin/fm-tier-guard.sh: firstmate-observable escalation triggers (diff size and elapsed time vs. the trivial tier's envelope, plus a general heavy-scale ceiling for any tier). - bin/fm-risk-tripwire.sh: a keyword/path trip-wire against a task's brief and diff, independent of the natural-language dispatch match. - bin/fm-ultracode-guard.sh: flag/reviewed/check subcommands that mechanically confirm a genuinely independent second pass ran before an ultracode-flagged task reaches PR-ready. - A documented model-verification gate in AGENTS.md section 4, extending the existing unverified-harness discipline to model names. Verification tiering (light/standard/heavy no-mistakes runs) is deliberately out of scope here and left for a follow-up change.
… distinct exit codes
…d distinct exit codes
… guardrail coverage
…herdr server-race
The read-tree portability rewrite restructured content_in_default() into an if/elif that checks exactly one candidate ref: the fetched origin ref when a remote exists, falling back to the local ref only when there is no remote at all. That silently dropped the case a no-mistakes-mode project with a closed, unmerged PR but work fast-forwarded into local main by hand - the fetched origin ref exists but does not contain the work, so the local ref is never even checked. Restore checking both candidates in sequence via a shared content_matches_ref helper, matching the pre-rewrite behavior tests/fm-teardown.test.sh's local-land regression test expects.
A merged PR does not, by itself, reach the live site: deployed apps run from a separate checkout under /mnt/nas/experiments/<name>/, managed by pm2, independent from the projects/<name> clone used for development. This gap required a manual pull+restart to notice and fix. bin/fm-nas-deploy-sync.sh fast-forwards a project's live NAS checkout and restarts its pm2 process(es), looked up from a captain-private data/nas-deployments.md mapping (best-effort, non-fatal, mirrors fm-fleet-sync.sh's dirty/diverged safety exactly). bin/fm-teardown.sh now calls it automatically after a landed ship task, including local-only merges.
added 27 commits
July 17, 2026 10:40
The turn-end guard blocked essentially every turn during a legitimate watcher re-arm, not just a genuine supervision gap. fm_watcher_healthy requires a live, identity-matched watcher lock with zero grace, but a backgrounded re-arm needs a brief moment, roughly 0.25-0.3s measured, more under load, to fork the new watcher and register its lock after the old one released it. The Stop hook fired inside that gap on essentially every re-arm, so it blocked with no real supervision lapse. The pull-based warning banner tolerates this same gap because it checks only beacon freshness with a 300s grace. The turn-end guard's sharper live-lock check had no equivalent tolerance. Replace the single fm_watcher_healthy check with a bounded poll: retry every 0.2s, mirroring the watcher-arm script's own confirm loop, and block only once a deadline passes still unhealthy. The deadline defaults to FM_ARM_CONFIRM_TIMEOUT (10s), the same budget the arm gives itself to confirm, and is independently overridable via FM_TURNEND_ARM_WAIT. A watcher that is genuinely missing, dead, or identity-mismatched for the whole window still blocks; it now takes up to that deadline instead of blocking on the first check. This intentionally slows the truly-blind path (no arm in flight at all), which is the documented tradeoff. Update docs/turnend-guard.md's Shared Predicate section, the contract's one authoritative statement, to describe the bounded poll instead of an immediate block. Testing: tests/fm-turnend-guard.test.sh's test_hook_runs_fast now pins the bounded-poll latency instead of asserting a near-instant return, and a new test_hook_allows_watcher_that_registers_mid_poll confirms a lock that registers partway through the window is let through rather than blocked. Other block-path tests default to a short override so the suite stays fast; ran the lint gate and the full behavior suite (41 tests, all passing).
…ndoff tasks-axi gate
…unch signal-race fix
…ard.md with turn-end-guard poll fix
Crew session for this task crashed (tmux/captain core dump). This commit preserves in-flight work from the orphaned worktree so it is not lost. A tar backup also exists under /home/orangepi/fm-recovery-backups/.
fm_test_tmproot installed its EXIT trap and appended to FM_TEST_CLEANUP_DIRS from inside a command-substitution subshell, so neither ever reached the calling script: the trap fired (and deleted the just-created dir) the instant the subshell exited, and the array mutation never propagated back. Every run of the secondmate suites therefore leaked its temp root, growing worse under SIGKILL (e.g. the OOM killer) since no trap can run for that signal regardless. Move registration to an on-disk manifest, which survives the subshell boundary, and install the EXIT trap once at source time in the real top-level shell. Add a startup sweep for leftover <prefix>.<pid>.XXXXXX roots whose owning PID is no longer alive, as the actual safety net for the SIGKILL case, and call it from both secondmate suites before they claim their own root.
Claude draws its cleared composer prompt as the glyph "❯" followed by a non-breaking space (U+00A0), not an ASCII space. Under a glibc UTF-8 locale, [[:space:]] does not classify U+00A0 as whitespace, so the composer classifier's ASCII-only trimming left the U+00A0 behind as phantom content and read an otherwise-empty composer as pending. fm-send's submit-verify loop then exhausted its retries and returned the false "Enter swallowed" / exit 1 the caller saw against genuinely submitted steers to a claude target. Normalize U+00A0 and its narrow sibling U+202F to an ASCII space before classifying, so a composer empty but for no-break padding reads empty on every backend while real typed text separated by a no-break space still classifies pending. Add tests/fm-composer-nbsp.test.sh pinning the fix at four layers: the shared classifier, the tmux composer reader on the real captured bytes, the submit loop, and the full fm-send exit code.
fm-merge-local.sh pushed to origin unconditionally whenever a local-only project happened to have one, contradicting the documented local-only contract of no remote, no PR. Auto-push now applies only when PROJ resolves to firstmate's own repo (the deliberate fork-sync case); every other project needs an explicit --push-origin. Also bundles three small pre-existing fixes touched during this review: fm-teardown.sh no longer runs the NAS deploy sync hook on a forced teardown, fm-ultracode-guard.sh validates the role argument before writing it into the line-based .ultracode marker file, and fm-bootstrap.sh's upstream fetch now falls back to gtimeout or a background-kill bound when timeout is unavailable, instead of fetching unbounded.
…kip and added missing FM_UPSTREAM_FETCH_TIMEOUT var
Add a config/autodeploy-logs sweep to fm-watch.sh's heartbeat fleet-scan. Each configured status log's last line is read on every heartbeat, and a failure (an ALERT rollup or a STUCK:/FAILED: line in fleet-sync's convention) surfaces once as a check wake carrying that line until it clears; a later healthy run re-arms the alarm. Mirrors bootstrap's critical_services_check but fires periodically, so a deploy failure between sessions is caught instead of waiting for the next session start. Absent config, a healthy "ok" last line, or a missing/unreadable log (a NAS hiccup) are all silent and non-fatal, so the check stays cheap enough to run every heartbeat without spurious alarms. Config is gitignored and absent by default, so it is a no-op for fleets that do not opt in.
Factor the failure predicate into the shared bin/fm-autodeploy-lib.sh (fm_autodeploy_line_failed), sourced by both the fm-watch.sh sweep and a new fm-bootstrap.sh autodeploy_logs_check, so an idle fleet catches a deploy failure at the next session start exactly as critical-services already does. One owner for the match set; docs and colocated tests updated to match.
On a hung NAS mount the tail read of an autodeploy status log could block the watcher heartbeat (and the bootstrap check). Wrap the read in the shared fm-autodeploy-lib.sh behind a bounded timeout and treat a timeout as unreadable, preserving the existing fail-quiet behavior for missing or unreadable logs. Both read sites go through the one shared helper.
…b and sweep When config/autodeploy-logs names a log but no read-timeout mechanism (timeout, gtimeout, or perl) is on PATH, fm_autodeploy_read_last_line fails every read and the autodeploy-alert feature silently never fires. Surface a one-time AUTODEPLOY_INERT bootstrap diagnostic in that case so the captain learns alerting is disabled rather than assuming a healthy deploy; the transient missing or unreadable log path stays fail-quiet. Document the FM_AUTODEPLOY_LOG_READ_TIMEOUT knob with the other timeout overrides and describe the heartbeat autodeploy sweep in the event-driven supervision architecture.
Independent review found the detected failure format attributed to firstmate's fleet-sync convention (bin/fm-fleet-sync.sh) in five places, including the primary ok/ALERT rollup example. fm-fleet-sync.sh emits the STUCK:/needs-attention alert line but never an ok or ALERT rollup; that rollup is an external per-project autodeploy tool's own convention (e.g. beamanalyzer's deploy job). Rephrase all five sites to name the external tool as the rollup's source, keep firstmate's own STUCK:/FAILED:/ needs-attention format, and point at the one-owner match set (bin/fm-autodeploy-lib.sh's fm_autodeploy_line_failed) instead of implying bin/fm-fleet-sync.sh produces the format.
…keys exist A scout task with no captain decision holds had no way to pass command_verify() without a tasks-axi build exposing the captain-hold contract, unlike every other tasks-axi touchpoint in this repo, which falls back to manual backlog handling when tasks-axi is incompatible. Move the decisions_reviewed/decision_keys read ahead of require_tasks_axi and only call it once decision_keys is non-empty.
…ard-timeout env vars
…oyments A second live-deployment topology alongside fm-nas-deploy-sync.sh's pm2 model: a project served from a NAS git checkout brought up by Docker Compose, optionally wrapped by a systemd unit, on a volume that must be mounted before it is safe to deploy. Same best-effort contract - one result line, always exit 0, bounded timeouts on every checkout touch, fast-forward-only, never force or touch a dirty/diverged checkout, shared packed-refs.lock race guard. Adds a mount gate (an unmounted required volume is an expected pre-unlock skip), runs the configured migration and rebuild/bring-up only when the fast-forward advanced HEAD, and verifies a configured health check (falling back to a lower-confidence containers-running check, said plainly). Reads a separate data/compose-deployments.md mapping kept apart from nas-deployments.md so a project is never matched by both scripts. Standalone: not auto-wired into teardown.
Once fetch retries are exhausted, the guard probed the lingering packed-refs.lock with an unbounded existence test, fm_lock_is_provably_stale (lsof/stat), and rm -f, plus an unbounded cd in packed_refs_lock_path. timeout/gtimeout only bound a real subprocess, not an in-process shell function, so a wedged NAS mount could hang the whole sync at that probe even though the fetch itself was already bounded. Run the staleness proof in a bounded child bash (export -f the fm_lock_* helpers) and bound the existence test, the removal, and the path-resolution cd like every other NAS touch. Add a regression test that stubs lsof to hang past the timeout and asserts the sync still returns promptly instead of blocking on the probe.
…grate/health phases
…m-lock-lib's fourth caller
A few merge-conflict resolutions during the rebase onto the new base kept the wrong side or an incomplete splice, silently dropping working test fixtures and doc rows: fm-bootstrap.test.sh's add_fake_systemctl helper, fm-backend.test.sh's symlink-spawn assertions, docs/scripts.md's tier-guard/risk-tripwire/ultracode-guard rows, and fm-bootstrap.sh's SERVICE_FAILED/UPSTREAM_DRIFT usage-line documentation.
inivosdigital
force-pushed
the
fm/add-fm-compose-deploy-sync
branch
from
July 17, 2026 15:29
009f1fd to
713a739
Compare
CI invokes each tests/*.test.sh directly, so a non-executable test file fails with Permission denied before it ever runs. The bit was lost when this file was restored during a rebase conflict resolution.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Add bin/fm-compose-deploy-sync.sh: a best-effort, mount-gated fast-forward sync + migrate + Docker Compose (optionally systemd-wrapped) redeploy for a project's live NAS git checkout - the second live-deployment topology alongside fm-nas-deploy-sync.sh's pm2 model. It reads data/compose-deployments.md, gates on a required mount, refuses to touch dirty/diverged checkouts (reports STUCK), always prints exactly one result line and exits 0 (best-effort, never fails its caller), and is deliberately standalone (NOT auto-wired into fm-teardown.sh yet - that is a separate explicit decision, asserted by a test). Every NAS filesystem/git touch is bounded with timeout/gtimeout so a wedged mount can never hang the caller.
This branch also fixes a hang found by an independent review: the exhausted-retry stale packed-refs-lock cleanup path was still unbounded (the existence test, fm_lock_is_provably_stale's lsof/stat probes, rm -f, and the cd in packed_refs_lock_path). Because timeout/gtimeout only bound a real subprocess and not an in-process shell function, a wedged NAS mount could hang the whole sync there despite the fetch itself already being bounded. The fix runs the staleness proof in a bounded child bash with the fm_lock_* helpers export -f'd (bounded_is_provably_stale), and wraps the existence test, the removal, and the path-resolution cd in the same bounded helpers. The single-quoted bash -c strings are intentional (SC2016 disabled inline) so the child bash expands its own positional args, not the parent's. A regression test stubs lsof to hang past the timeout and asserts the sync still returns promptly (verified: it hangs ~20s pre-fix, returns in ~1s post-fix). This mirrors the sibling fm-nas-deploy-sync.sh's fix shape.
What Changed
bin/fm-compose-deploy-sync.sh, a systemd + Docker Compose NAS deployment topology (mount-gated fast-forward, migrate, redeploy, health check, refuses dirty/diverged checkouts, always reports one result line and exits 0) alongside the existing pm2-basedbin/fm-nas-deploy-sync.sh, with every NAS git/filesystem touch bounded bytimeout/gtimeout, including a fix to bound the previously-unbounded stale packed-refs-lock cleanup and NAScdcalls in the deploy/migrate/health phases.bin/fm-tier-guard.sh,bin/fm-risk-tripwire.sh), and mechanical ultracode independent-review tracking (bin/fm-ultracode-guard.sh).Risk Assessment
✅ Low: The follow-up commit correctly fixes the previously-flagged unbounded cd by folding the cd into the same bounded child process (run_in_nas) at all four call sites (compose(), migrate, containers_running(), health_ok()); no unbounded cd "$NAS_PATH" remains in the file, argument/quoting propagation through the nested bash -c is correct, and stdout/stderr redirection semantics are preserved, so this is a small, well-contained, verified bug fix with no new risk introduced.
Testing
The pre-run baseline test command (all tests/*.test.sh) already passed; I additionally ran tests/fm-compose-deploy-sync.test.sh in isolation (21/21 pass) and, more importantly, drove bin/fm-compose-deploy-sync.sh directly as an end user would against hand-built throwaway NAS git fixtures — reproducing the exact pre-fix ~20s hang vs. post-fix ~1s return on the stale packed-refs-lock probe, and separately exercising a full real deploy (fast-forward, migrate, systemd-wrapped compose rebuild, health check) plus the mount-gate skip and dirty-checkout STUCK safety paths, all with checkout state verified byte-for-byte before/after. Everything behaved exactly as the user intent describes; no issues found.
Evidence: Pre-fix vs post-fix CLI transcript: stale packed-refs-lock probe hang bound (20s -> 1s)
Evidence: End-to-end CLI transcript: happy-path deploy, mount-gate skip, and dirty-checkout STUCK safety checks
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
⏭️ **Rebase** - skipped
.agents/skills/bootstrap-diagnostics/SKILL.md- branch carries 62 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (75 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
🔧 **Review** - 1 issue found → auto-fixed ✅
bin/fm-compose-deploy-sync.sh:427- Thecd "$NAS_PATH"calls incompose()(line 427), the migrate step (line 434),containers_running()(line 473), and the shell-command branch ofhealth_ok()(line 489) all run in a plain subshell( cd "$NAS_PATH" && ... ), so thecditself is never wrapped inbounded/bounded_deploy/bounded_health. This is exactly the class of bug the immediately-preceding commit (d768f56) just fixed forpacked_refs_lock_path()'scdand for the stale-lock probe:timeout/gtimeoutonly bounds a real subprocess, not an in-process shell builtin, so if the NAS mount degrades between the successful fetch/merge and the (up to 1200s-long) migrate/build/health phase,cdcan block indefinitely with no bound at all - reintroducing the exact hang risk this whole file (and its header comment, lines 84-89: "Every filesystem/git touch of the NAS checkout is bounded... an unreachable NAS is a hang risk") claims to have eliminated. Failure scenario: NAS mount becomes unresponsive mid-deploy (after fetch/merge succeeded) ->cd "$NAS_PATH"insidecompose()/migrate/health_ok blocks forever -> fm-compose-deploy-sync.sh never returns, violating its own "always exits 0, never blocks its caller" contract and hanging firstmate's invoking session. No existing test exercises this path (only the fetch hang and the exhausted-retry stale-lock probe hang are covered). Fix: apply the samebounded bash -c 'cd "$1" && ...' bash "$NAS_PATH"pattern already used forpacked_refs_lock_path()to these four call sites.🔧 Fix: fix(compose-sync): bound the NAS cd in deploy/migrate/health phases
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"bash tests/fm-compose-deploy-sync.test.sh (all 21 cases pass, ~10s)Manual CLI repro: prefix (pre-fix, commit cae9bec) fm-compose-deploy-sync.sh against a stubbed hunglsof+ persistent packed-refs.lock — hung for 20s before returning exit 0 with a correct skip messageManual CLI repro: current (post-fix, commit 94cc4c5) fm-compose-deploy-sync.sh against the identical fixture — returned in 1s with the identical result line and exit 0Manual CLI repro: full happy-path deploy (mounted volume, behind checkout, migrate_cmd, systemd-wrapped docker compose build + systemctl restart, curl health check) — confirmed fast-forward from f78ec82 to 24c277f, migrate log written, docker/systemctl invocation logs, single 'synced ... redeployed and health check passed' result line, exit 0Manual CLI repro: unmounted required volume — single skip line, checkout HEAD verified unchangedManual CLI repro: dirty checkout — single STUCK line, HEAD andgit status --porcelainverified byte-identical before/after✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.