Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bin/fm-brief.sh
Original file line number Diff line number Diff line change
Expand Up @@ -196,12 +196,12 @@ HERDR_SECTION=$(printf '%s\n' \
'# Herdr isolation - HARD SAFETY CONTRACT' \
'This brief was explicitly scaffolded with `--herdr-lab` because the task will drive Herdr lifecycle behavior.' \
'On Herdr 0.7.3 the API socket is not relocatable by `HERDR_CONFIG_PATH`, `XDG_CONFIG_HOME`, or `HOME`.' \
'A named non-`default` session plus a trailing `--session <name>` on every call is the only viable local isolation.' \
'A named non-`default` session plus an explicit Herdr-global `--session <name>` on every call is the only viable local isolation.' \
'' \
'1. Set `HERDR_LAB_HELPER='"$HERDR_LAB_HELPER"'` and generate the session name with `HERDR_LAB_SESSION=$("$HERDR_LAB_HELPER" name '"$ID"')`.' \
' Install `trap '\''"$HERDR_LAB_HELPER" teardown "$HERDR_LAB_SESSION"'\'' EXIT` before provisioning, then provision only with `"$HERDR_LAB_HELPER" provision "$HERDR_LAB_SESSION"`.' \
'2. Run every task-specific non-lifecycle Herdr command through `"$HERDR_LAB_HELPER" run "$HERDR_LAB_SESSION" <arguments...>`.' \
' The helper appends the required trailing `--session "$HERDR_LAB_SESSION"`; `HERDR_SESSION` alone is never accepted as isolation.' \
' The helper appends `--session "$HERDR_LAB_SESSION"` for ordinary calls and places it immediately before any `--` command-argv delimiter; `HERDR_SESSION` alone is never accepted as isolation.' \
'3. Teardown only through `"$HERDR_LAB_HELPER" teardown "$HERDR_LAB_SESSION"`.' \
' It re-checks refuse-default immediately before stop and again immediately before delete, and fails closed on ambiguity.' \
'4. If an experiment requires a deliberate mid-run session stop, use only `"$HERDR_LAB_HELPER" stop "$HERDR_LAB_SESSION"`; it performs the same immediate refuse-default check.' \
Expand Down
20 changes: 17 additions & 3 deletions bin/fm-herdr-lab.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
# Session names must begin with "fm-lab-" and can never be "default".
# The name command sanitizes the label, caps it at 16 characters, and appends
# process/random suffixes to keep generated socket paths short.
# Every Herdr call made here carries a trailing --session <session>.
# Every Herdr call made here carries --session <session> as a Herdr global
# option: trailing for ordinary calls, or immediately before a `--` command-argv
# delimiter so the child command cannot consume the isolation flag.
# The run command rejects caller-supplied --session flags, any leading option
# before the subcommand, all session lifecycle operations, and every server
# operation.
Expand Down Expand Up @@ -49,9 +51,21 @@ fm_herdr_lab_tripwire_path() { # <session>
}

fm_herdr_lab_raw() { # <session> <herdr arguments...>
local name=$1
local name=$1 arg found_delimiter=0
local herdr_args=()
shift
HERDR_SESSION="$name" herdr "$@" --session "$name"
for arg in "$@"; do
if [ "$found_delimiter" -eq 0 ] && [ "$arg" = -- ]; then
herdr_args+=(--session "$name" --)
found_delimiter=1
else
herdr_args+=("$arg")
fi
done
if [ "$found_delimiter" -eq 0 ]; then
herdr_args+=(--session "$name")
fi
HERDR_SESSION="$name" herdr "${herdr_args[@]}"
}

fm_herdr_lab_session_list() { # <session>
Expand Down
144 changes: 144 additions & 0 deletions bin/fm-watch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,146 @@ _event_cap_key=""
_event_cap_ok=0
_event_cap_fails=0

# Herdr's agent lifecycle state must stay truthful: an idle primary remains idle
# while this watcher blocks. Display-only pane metadata provides the distinct
# captain-visible state instead. A watcher publishes `idle · supervised` only
# after it owns the singleton, refreshes a bounded TTL while it stays live, and
# clears its own unique source on exit. The TTL lets a SIGKILL or host crash age
# out rather than leaving a false permanent indicator. Other runtime backends
# have no equivalent shared display surface and remain unchanged.
FM_HERDR_SUPERVISION_STATUS_TTL_MS=${FM_HERDR_SUPERVISION_STATUS_TTL_MS:-360000}
FM_SUPERVISION_VISIBILITY_PUBLISHED=0
FM_SUPERVISION_VISIBILITY_SOURCE=
FM_SUPERVISION_VISIBILITY_SESSION=
FM_SUPERVISION_VISIBILITY_PANE=
FM_SUPERVISION_VISIBILITY_REFRESH_PID=
FM_SUPERVISION_VISIBILITY_REFRESH_IDENTITY=

fm_supervision_visibility_resolve_herdr() {
local target backend=${FM_SUPERVISOR_BACKEND:-}
FM_SUPERVISION_VISIBILITY_SESSION=
FM_SUPERVISION_VISIBILITY_PANE=
case "$backend" in
herdr)
target=${FM_SUPERVISOR_TARGET:-}
[ -n "$target" ] || return 1
FM_SUPERVISION_VISIBILITY_SESSION=${target%%:*}
FM_SUPERVISION_VISIBILITY_PANE=${target#*:}
[ "$FM_SUPERVISION_VISIBILITY_PANE" != "$target" ] || return 1
;;
'')
[ "${HERDR_ENV:-}" = 1 ] || return 1
[ -z "${TMUX:-}" ] || return 1
[ -n "${HERDR_PANE_ID:-}" ] || return 1
FM_SUPERVISION_VISIBILITY_SESSION=${HERDR_SESSION:-default}
FM_SUPERVISION_VISIBILITY_PANE=$HERDR_PANE_ID
;;
*) return 1 ;;
esac
[ -n "$FM_SUPERVISION_VISIBILITY_SESSION" ] && [ -n "$FM_SUPERVISION_VISIBILITY_PANE" ]
}

# Display metadata is optional observability and must never wedge the watcher.
# Use the same dual session targeting as fm_backend_herdr_cli, but put a hard
# three-second process bound around this cosmetic call.
fm_supervision_visibility_herdr_call() { # <session> <herdr args...>
local session=$1
shift
command -v herdr >/dev/null 2>&1 || return 1
if command -v timeout >/dev/null 2>&1; then
HERDR_SESSION="$session" timeout --kill-after=1 2 herdr "$@" --session "$session"
elif command -v gtimeout >/dev/null 2>&1; then
HERDR_SESSION="$session" gtimeout --kill-after=1 2 herdr "$@" --session "$session"
elif command -v perl >/dev/null 2>&1; then
HERDR_SESSION="$session" perl -e 'my $seconds = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV } my $stop = sub { $SIG{TERM} = $SIG{INT} = $SIG{HUP} = "IGNORE"; kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; waitpid $pid, 0; exit 124 }; local $SIG{ALRM} = $stop; local $SIG{TERM} = $stop; local $SIG{INT} = $stop; local $SIG{HUP} = $stop; alarm $seconds; waitpid $pid, 0; alarm 0; exit($? >> 8)' \
2 herdr "$@" --session "$session"
else
return 1
fi
}

fm_supervision_visibility_refresh() {
local ttl=$FM_HERDR_SUPERVISION_STATUS_TTL_MS
[ -n "$FM_SUPERVISION_VISIBILITY_SOURCE" ] || return 0
fm_supervision_visibility_resolve_herdr || return 0
case "$ttl" in ''|*[!0-9]*|0) ttl=360000 ;; esac
if fm_supervision_visibility_herdr_call "$FM_SUPERVISION_VISIBILITY_SESSION" \
pane report-metadata "$FM_SUPERVISION_VISIBILITY_PANE" \
--source "$FM_SUPERVISION_VISIBILITY_SOURCE" \
--custom-status supervised \
--state-label 'idle=idle · supervised' \
--ttl-ms "$ttl" >/dev/null 2>&1; then
FM_SUPERVISION_VISIBILITY_PUBLISHED=1
fi
return 0
}

fm_supervision_visibility_clear() {
[ -n "$FM_SUPERVISION_VISIBILITY_SOURCE" ] || return 0
fm_supervision_visibility_resolve_herdr || return 0
fm_supervision_visibility_herdr_call "$FM_SUPERVISION_VISIBILITY_SESSION" \
pane report-metadata "$FM_SUPERVISION_VISIBILITY_PANE" \
--source "$FM_SUPERVISION_VISIBILITY_SOURCE" \
--clear-custom-status --clear-state-labels >/dev/null 2>&1 || true
# shellcheck disable=SC2034 # Read by deterministic tests after sourcing this script.
FM_SUPERVISION_VISIBILITY_PUBLISHED=0
}

fm_supervision_visibility_refresh_interval() {
local ttl=$FM_HERDR_SUPERVISION_STATUS_TTL_MS half seconds millis
case "$ttl" in ''|*[!0-9]*|0) ttl=360000 ;; esac
half=$((ttl / 2))
[ "$half" -gt 0 ] || half=1
seconds=$((half / 1000))
millis=$((half % 1000))
printf '%d.%03d\n' "$seconds" "$millis"
}

fm_supervision_visibility_refresh_loop() { # <watcher pid>
local owner_pid=$1 interval sleeper='' expected_identity='' current_identity=''
interval=$(fm_supervision_visibility_refresh_interval)
trap '[ -z "$sleeper" ] || kill -TERM "$sleeper" 2>/dev/null || true; exit 0' HUP INT TERM
while :; do
sleep "$interval" &
sleeper=$!
wait "$sleeper" 2>/dev/null || exit 0
sleeper=
[ "$(cat "$WATCH_LOCK/pid" 2>/dev/null || true)" = "$owner_pid" ] || exit 0
expected_identity=$(cat "$WATCH_LOCK/pid-identity" 2>/dev/null || true)
current_identity=$(fm_pid_identity "$owner_pid" 2>/dev/null || true)
[ -n "$expected_identity" ] && [ "$current_identity" = "$expected_identity" ] || exit 0
fm_supervision_visibility_refresh
done
}

fm_supervision_visibility_start() { # <watcher pid>
local owner_pid=$1 pid identity
fm_supervision_visibility_resolve_herdr || return 0
fm_supervision_visibility_refresh
fm_supervision_visibility_refresh_loop "$owner_pid" &
pid=$!
identity=$(fm_pid_identity "$pid" 2>/dev/null || true)
if [ -z "$identity" ]; then
kill -TERM "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || true
return 0
fi
FM_SUPERVISION_VISIBILITY_REFRESH_PID=$pid
FM_SUPERVISION_VISIBILITY_REFRESH_IDENTITY=$identity
}

fm_supervision_visibility_stop() {
local pid=${FM_SUPERVISION_VISIBILITY_REFRESH_PID:-} identity=${FM_SUPERVISION_VISIBILITY_REFRESH_IDENTITY:-} current=
[ -n "$pid" ] || return 0
current=$(fm_pid_identity "$pid" 2>/dev/null || true)
if [ -n "$identity" ] && [ "$current" = "$identity" ]; then
kill -TERM "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || true
fi
FM_SUPERVISION_VISIBILITY_REFRESH_PID=
FM_SUPERVISION_VISIBILITY_REFRESH_IDENTITY=
}

# afk_present: 0 while the away-mode flag exists. When set, the daemon wraps this
# watcher and owns triage, so the watcher must behave one-shot (enqueue + exit on
# every wake) and let the daemon classify - never absorb here, or the daemon's
Expand Down Expand Up @@ -686,6 +826,8 @@ if ! fm_lock_try_acquire "$WATCH_LOCK"; then
exit 0
fi
watcher_cleanup() {
fm_supervision_visibility_stop
fm_supervision_visibility_clear
fm_active_check_stop || return 1
fm_check_output_cleanup
fm_custom_check_snapshot_cleanup
Expand All @@ -697,9 +839,11 @@ trap 'exit 1' HUP INT TERM
# ${BASHPID:-$$} from this same main shell). Read directly, never via a command
# substitution, so it matches the stored holder pid for the self-eviction check.
WATCHER_PID=${BASHPID:-$$}
FM_SUPERVISION_VISIBILITY_SOURCE="firstmate-supervision:$WATCHER_PID"
printf '%s\n' "$FM_HOME" > "$WATCH_LOCK/fm-home" || true
printf '%s\n' "$WATCH_PATH" > "$WATCH_LOCK/watcher-path" || true
fm_pid_identity "$WATCHER_PID" > "$WATCH_LOCK/pid-identity" 2>/dev/null || true
fm_supervision_visibility_start "$WATCHER_PID"

[ -e "$STATE/.last-heartbeat" ] || touch "$STATE/.last-heartbeat"

Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ firstmate's always-loaded operating contract and routing index for conditional p
## Event-driven supervision

A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable.
On Herdr, that live singleton also marks the idle primary as `idle · supervised` through display-only pane metadata without changing its real agent lifecycle state; [`herdr-backend.md`](herdr-backend.md#captain-visible-supervision-state-2026-07-19) owns the mechanism and verification evidence.
Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, stale panes whose crew is not provably working whether their status log looks terminal or non-terminal, provably-working stale panes that persist past `FM_STALE_ESCALATE_SECS`, declared external waits that remain paused past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits.
Repeated provably-working stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker.
Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue.
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,7 @@ FM_WATCH_REARM_RETRY_LIMIT=5 # Pi/OpenCode adapter launch-failure retries befo
FM_WATCH_CYCLE_LOG_MAX_BYTES=262144 # size cap for the arm-owned watcher lifecycle ledger
FM_WATCH_CYCLE_LOG_KEEP_LINES=1000 # newest complete lifecycle rows considered when the ledger is capped
FM_WATCHER_STALE_GRACE=300 # defaults to FM_GUARD_GRACE; seconds a live watcher lock may have a stale beacon before re-arm errors
FM_HERDR_SUPERVISION_STATUS_TTL_MS=360000 # Herdr-only: display-metadata TTL for the live watcher's idle · supervised label; unset, blank, zero, or nonnumeric values use 360000
FM_SIGNAL_GRACE=30 # seconds to coalesce nearby status and turn-end signals into one wake
FM_CAPTAIN_RE='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' # status regex that makes watcher and daemon signal/stale/scan output captain-relevant
FM_CLASSIFY_PAUSED_VERB=paused # leading status verb for a declared external wait; excluded from FM_CAPTAIN_RE and distinct from blocked
Expand Down
Loading