Skip to content

fix(watcher): surface stopped stales once per situation, not per poll or hash tick #735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anxsec wants to merge 8 commits into
base: main
Choose a base branch
from
Open

fix(watcher): surface stopped stales once per situation, not per poll or hash tick #735

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
19688e0
fix(watcher): surface stopped stales once per situation, not per poll…
anxsec Jul 11, 2026
d4717c6
no-mistakes(test): fix tmux 3.7 marker drop and locale/race test fail…
anxsec Jul 11, 2026
0be11c3
no-mistakes(document): sync herdr regex and tmux marker-send docs
anxsec Jul 11, 2026
54ae6c6
no-mistakes(review): remove dead 0x1f hex-send; re-arm stale surfacin…
anxsec Jul 19, 2026
87411d1
no-mistakes(document): sync stale-surfacing cadence note in configura…
anxsec Jul 19, 2026
15874e4
no-mistakes(review): never cache a failed crew-state read as supersed…
anxsec Jul 19, 2026
5fc1964
no-mistakes(document): sync crew_absorb_class wrapper comment with un…
anxsec Jul 19, 2026
b16ac6d
fix(watcher): keep an undisproven pause on the bounded cadence
anxsec Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions bin/backends/herdr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1297,7 +1297,10 @@ FM_BACKEND_HERDR_IDLE_RE=${FM_BACKEND_HERDR_IDLE_RE:-'^Type a message\.\.\.$'}
# Known bare (unbordered) prompt glyphs a composer row may start with: ❯
# (claude) and › (codex) only. Generic shell-style glyphs > $ % # are still
# recognized after a bordered composer row has already been structurally found.
FM_BACKEND_HERDR_BARE_PROMPT_RE=${FM_BACKEND_HERDR_BARE_PROMPT_RE:-'^[❯›]'}
# Written as an anchored alternation, not a bracket class: under a C/byte
# locale a bracket class matches these glyphs' individual UTF-8 bytes, so box
# corners like ╰ (shared 0xE2 lead byte) would misread as a bare prompt row.
FM_BACKEND_HERDR_BARE_PROMPT_RE=${FM_BACKEND_HERDR_BARE_PROMPT_RE:-'^(❯|›)'}
# Pi allows a multi-line composer between its horizontal separators. Bound the
# structural candidate so two unrelated transcript rules with an arbitrarily
# large region between them can never be promoted into a composer.
Expand Down Expand Up @@ -1461,7 +1464,7 @@ EOF
fi
# Delegate the empty/pending/unknown decision to the shared owner. The bare
# shape only ever starts with an AGENT glyph (FM_BACKEND_HERDR_BARE_PROMPT_RE
# is '^[❯›]'), so a bare shell prompt never reaches here - it stays 'unknown'
# is '^(❯|›)'), so a bare shell prompt never reaches here - it stays 'unknown'
# via the no-composer-row path above, exactly as before.
fm_composer_classify_content "$bordered" "$stripped" "$FM_BACKEND_HERDR_IDLE_RE"
}
Expand Down
25 changes: 15 additions & 10 deletions bin/fm-classify-lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,13 +318,17 @@ signal_reason_is_actionable() { # <file> ...
# Classify WHY an idle/stale crew MIGHT be safely absorbed instead of surfaced,
# from bin/fm-crew-state.sh's one authoritative current-state line
# ("state: <s> · source: <src> · <detail>"). Prints exactly one token:
# working - an actively-running no-mistakes step (running/fixing/ci) or a busy
# pane; the crew is legitimately mid-work on a static-looking pane
# (e.g. waiting on CI);
# paused - the crew's authoritative current state is a declared external-wait
# pause (paused:), which is EXPECTED to idle;
# none - neither, so the wake must surface (a stopped/finished/parked/failed/
# torn-down/unknown crew, or an unreadable verdict).
# working - an actively-running no-mistakes step (running/fixing/ci) or a busy
# pane; the crew is legitimately mid-work on a static-looking pane
# (e.g. waiting on CI);
# paused - the crew's authoritative current state is a declared external-wait
# pause (paused:), which is EXPECTED to idle;
# none - neither, so the wake must surface (a stopped/finished/parked/
# failed/torn-down/unknown crew);
# unreadable - the fm-crew-state.sh read itself failed or produced no parseable
# state line: NOT a verdict about the crew. Consumers must treat it
# like none for surfacing (conservative) but never durably cache it
# as a negative verdict, so the next poll re-reads.
# One fm-crew-state.sh read serves BOTH absorb reasons at once. Reading the state
# authoritatively (not the status log) is what keeps run-step precedence: a crew
# that appended paused: but then STARTED a run reports working, never paused.
Expand All @@ -334,8 +338,8 @@ signal_reason_is_actionable() { # <file> ...
crew_absorb_class() { # <id>
local id=$1 line state src
[ -n "$id" ] || { printf 'none'; return; }
line=$("$FM_CREW_STATE_BIN" "$id" 2>/dev/null) || true
case "$line" in state:*) ;; *) printf 'none'; return ;; esac
line=$("$FM_CREW_STATE_BIN" "$id" 2>/dev/null) || { printf 'unreadable'; return; }
case "$line" in state:*) ;; *) printf 'unreadable'; return ;; esac
state=${line#state: }; state=${state%% *}
if [ "$state" = paused ]; then printf 'paused'; return; fi
if [ "$state" = working ]; then
Expand All @@ -351,7 +355,8 @@ crew_absorb_class() { # <id>
# ONLY when this returns 0, and SURFACED otherwise (the crew may be done, waiting
# on a decision, or wedged). For stale panes it is checked before trusting the
# status log so a pre-validation captain-relevant line does not override an active
# run. See crew_absorb_class for the exact working/paused/none decision.
# run. See crew_absorb_class for the exact working/paused/none/unreadable
# decision (unreadable counts as not provably working here).
crew_is_provably_working() { # <id>
[ "$(crew_absorb_class "$1")" = working ]
}
Expand Down
130 changes: 116 additions & 14 deletions bin/fm-watch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,20 @@
# re-surface cadence, never as a wedge. Only when neither
# absorb class applies does the log's last line decide:
# terminal (captain-relevant) or non-terminal (no verb),
# both surfaced at once. A provably-working stale past the
# both surfaced at once - and both surfaced ONCE PER
# SITUATION (.stale-surfaced-*, keyed on the last status
# line), not once per pane hash: a live idle agent's
# ticking status bar churns the hash every minute, so a
# repeat of an already-surfaced situation is absorbed
# until positive working evidence (a busy pane, an
# active run) or a new status line re-arms surfacing.
# A paused: log line found superseded by the
# authoritative crew state is cached against the status
# file's signature (.paused-none-*) so it is neither
# re-read nor re-surfaced every poll; a FAILED crew-state
# read is never cached, so a transient misread cannot
# disarm a genuine pause's recheck cadence.
# A provably-working stale past the
# wedge threshold also surfaces, with an "escalation N"
# count in the reason; at FM_WEDGE_DEMAND_INSPECT_COUNT
# consecutive escalations on the SAME pane, the reason
Expand Down Expand Up @@ -350,12 +363,22 @@ clear_pause_tracking() { # <window>
key=${key//\//_}
key=${key//./_}
clear_pause_state "$win"
rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key"
rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" "$STATE/.paused-none-$key"
}

# Reconcile a declared pause or captain-held status with authoritative crew state.
# Only a confidently dead ordinary crew may recover paused classification after
# fm-crew-state has fallen back to stopped or unknown.
# Prints working/paused/none/unreadable as crew_absorb_class does, plus one token
# of its own:
# hold - the declared pause or captain hold is NOT disproven (either the read
# was skipped inside the bounded recheck window, or it still reported
# paused) while the agent process is not confidently dead. Not a verdict
# that the crew stopped: a fresh stale surfaces it once, but a repeat on
# an unchanged pane keeps the bounded pause cadence instead.
# Keeping hold distinct from none/unreadable is what lets a live external-decision
# gate retain that cadence while a pause the authoritative state SUPERSEDES (none)
# is surfaced once and cached, and an unreadable read is surfaced without caching.
pause_state_class() { # <window> <task>
local win=$1 task=$2 key last recheck_file class agent_alive
key=${win//:/_}
Expand All @@ -372,8 +395,10 @@ pause_state_class() { # <window> <task>
if [ "$(window_kind "$win")" != secondmate ]; then
agent_alive=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || agent_alive=unknown
if [ "$agent_alive" != dead ]; then
rm -f "$recheck_file"
printf 'none'
# A hold keeps the recheck stamp: like paused, it is still on the bounded
# cadence, so dropping the stamp would re-run the costly crew-state read
# on every poll of an unchanged pane.
printf 'hold'
return
fi
fi
Expand All @@ -389,8 +414,15 @@ pause_state_class() { # <window> <task>
if [ "$(window_kind "$win")" != secondmate ]; then
agent_alive=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || agent_alive=unknown
if [ "$agent_alive" != dead ]; then
rm -f "$recheck_file"
printf 'none'
# A still-paused authoritative read on a live agent is a hold, not a
# verdict that the crew stopped; none/unreadable stay themselves so the
# superseded-pause cache and the no-verdict path can tell them apart.
# A hold stamps the recheck like paused does, so the bounded cadence -
# not every poll - owns the next costly read.
case "$class" in
paused) date +%s > "$recheck_file"; printf 'hold' ;;
*) rm -f "$recheck_file"; printf '%s' "$class" ;;
esac
return
fi
fi
Expand All @@ -402,13 +434,34 @@ pause_state_class() { # <window> <task>
printf '%s' "$class"
}

# Surface a stopped crew's stale pane - once per distinct SITUATION, not once
# per pane hash. A surfaced stale records "stopped|<last status line>" in
# .stale-surfaced-<key>; a later stale whose situation matches that record is
# absorbed instead of re-surfaced. The pane hash alone is not a stable dedup
# key: a live idle agent's ticking status bar changes the hash every minute,
# and a watcher restart can re-see a drifted hash, so hash-keyed dedup alone
# re-surfaced the same finished merge-wait crew on effectively every poll
# (the 2026-07-11 fix-sync-ssh-m4 incident). The record is cleared only on
# POSITIVE working evidence (a busy pane, an active run), so a crew that
# visibly resumes and stops again - or writes a new status line - still
# surfaces; the first stop of any situation always surfaces, preserving
# absorb-only-when-provably-working.
surface_nonterminal_stale() { # <window> <hash>
local win=$1 h=$2 key task last
local win=$1 h=$2 key task last sit sitf
key=$(printf '%s' "$win" | tr ':/.' '___')
task=$(window_to_task "$win" "$STATE")
sit="stopped|$(last_status_line "$STATE/$task.status")"
sitf="$STATE/.stale-surfaced-$key"
if [ "$sit" = "$(cat "$sitf" 2>/dev/null)" ]; then
printf '%s' "$h" > "$STATE/.stale-$key"
rm -f "$STATE/.stale-since-$key" "$STATE/.paused-$key" "$STATE/.paused-rechecked-$key" "$STATE/.paused-resurfaced-$key"
triage_log "absorbed stale (stopped situation already surfaced): $win"
return 0
fi
fm_wake_append stale "$win" "stale: $win" || exit 1
printf '%s' "$h" > "$STATE/.stale-$key"
printf '%s' "$sit" > "$sitf"
rm -f "$STATE/.stale-since-$key"
task=$(window_to_task "$win" "$STATE")
last=$(last_status_line "$STATE/$task.status")
if status_is_paused_or_captain_held "$last"; then
: > "$STATE/.paused-$key"
Expand Down Expand Up @@ -893,6 +946,8 @@ EOF
ssf="$STATE/.stale-since-$key"
ewf="$STATE/.wedge-escalations-$key"
pf="$STATE/.paused-$key" # flag: this key's stale is using the bounded pause cadence
sitf="$STATE/.stale-surfaced-$key" # situation record of the last SURFACED stale (see surface_nonterminal_stale)
pnf="$STATE/.paused-none-$key" # status-file signature at which a paused: log line was last found superseded
prev=$(cat "$hf" 2>/dev/null || true)
if [ "$h" = "$prev" ]; then
n=$(( $(cat "$cf" 2>/dev/null || echo 0) + 1 ))
Expand Down Expand Up @@ -933,12 +988,22 @@ EOF
# over the log) a chance to override before trusting the log.
if [ "$(cat "$sf" 2>/dev/null || true)" != "$h" ]; then
if crew_is_provably_working "$(window_to_task "$w" "$STATE")"; then
rm -f "$sitf"
printf '%s' "$h" > "$sf"
date +%s > "$ssf"
triage_log "absorbed stale (provably working, overriding a stale captain-relevant status): $w"
elif [ "terminal|$(last_status_line "$STATE/$task.status")" = "$(cat "$sitf" 2>/dev/null)" ]; then
# Already surfaced this exact terminal situation: only the pane
# hash moved (a live idle agent's ticking status bar, or a
# watcher restart re-seeing a drifted pane). Advance the
# suppressor quietly instead of waking once per tick.
printf '%s' "$h" > "$sf"
rm -f "$ssf"
triage_log "absorbed stale (terminal situation already surfaced): $w"
else
fm_wake_append stale "$w" "stale: $w" || exit 1
printf '%s' "$h" > "$sf"
printf 'terminal|%s' "$(last_status_line "$STATE/$task.status")" > "$sitf"
rm -f "$ssf"
mark_surfaced "$STATE/$(window_to_task "$w" "$STATE").status"
wake "stale: $w"
Expand Down Expand Up @@ -967,12 +1032,16 @@ EOF
# pause - the crew has STOPPED. Surface immediately so firstmate peeks
# (it may be done via an interactive menu that wrote no done: status,
# waiting on a decision, or wedged) instead of leaving the finish to
# wait out the timer.
# wait out the timer;
# - hold (or an unreadable read): no verdict either way, so a FRESH
# stale surfaces here exactly as none does; only the unchanged-hash
# re-entry below distinguishes them.
if [ "$(cat "$sf" 2>/dev/null || true)" != "$h" ]; then
task=$(window_to_task "$w" "$STATE")
case "$(pause_state_class "$w" "$task")" in
working)
clear_pause_tracking "$w"
rm -f "$sitf"
printf '%s' "$h" > "$sf"
date +%s > "$ssf"
triage_log "absorbed non-terminal stale (provably working): $w"
Expand All @@ -986,23 +1055,55 @@ EOF
esac
else
task=$(window_to_task "$w" "$STATE")
if [ -e "$pf" ] || status_is_paused_or_captain_held "$(last_status_line "$STATE/$task.status")"; then
statusf="$STATE/$task.status"
# The paused-looking re-entry is gated by a negative-verdict cache:
# once pause_state_class found the paused: log line superseded
# (class none, e.g. run-step precedence reporting done) at this
# exact status-file signature, re-running the costly read every
# poll would only re-decide the same thing - and, before this
# cache, re-SURFACE the same stopped crew on every poll through
# this branch (the 2026-07-11 merge-wait incident). A new status
# write changes the signature and re-evaluates. Only a DEFINITIVE
# none verdict is cached: an unreadable crew-state read is no
# verdict at all, so it surfaces conservatively but leaves the
# recheck cadence armed for the next poll instead of freezing a
# possibly-genuine pause behind a transient read failure.
# A hold - the pause still undisproven on a not-confidently-dead
# agent - is likewise no verdict, but here it keeps the bounded
# PAUSE_RESURFACE_SECS cadence rather than re-surfacing, so a live
# external-decision gate that already surfaced once stays parked.
if [ -e "$pf" ] || { status_is_paused_or_captain_held "$(last_status_line "$statusf")" \
&& [ "$(stat_sig "$statusf")" != "$(cat "$pnf" 2>/dev/null)" ]; }; then
case "$(pause_state_class "$w" "$task")" in
paused) handle_paused_stale "$w" "$task" "$h" ;;
working) clear_pause_state "$w"
paused) rm -f "$pnf"
handle_paused_stale "$w" "$task" "$h" ;;
working) rm -f "$pnf" "$sitf"
clear_pause_state "$w"
printf '%s' "$h" > "$sf"
wedge_timer_check "$w" "$ssf" "non-terminal stale (provably working after a declared pause)" "$ewf"
triage_log "absorbed non-terminal stale (provably working): $w" ;;
*) handle_paused_stale "$w" "$task" "$h" ;;
none) stat_sig "$statusf" > "$pnf" || : > "$pnf"
surface_nonterminal_stale "$w" "$h" ;;
hold) rm -f "$pnf"
handle_paused_stale "$w" "$task" "$h" ;;
*) surface_nonterminal_stale "$w" "$h" ;;
esac
else
elif [ -e "$ssf" ] || [ ! -e "$sitf" ]; then
# Wedge timing belongs to provably-working absorbs (and heals a
# timer lost to a crash between the hash and timer writes); an
# already-SURFACED stopped stale stays quiet instead of being
# re-nagged as a possible wedge every STALE_ESCALATE_SECS.
wedge_timer_check "$w" "$ssf" "non-terminal stale" "$ewf"
fi
fi
fi
else
# Pane busy or not yet stably stale: reset pending escalation bookkeeping.
rm -f "$ssf" "$ewf"
# Reaching here with n >= 2 means window_is_busy returned busy: positive
# working evidence, so clear the surfaced-situation record - the next
# stop is a genuinely new event and must surface again.
[ "$n" -ge 2 ] && rm -f "$sitf"
if [ -e "$pf" ] && { [ "$n" -ge 2 ] || ! status_is_paused_or_captain_held "$(last_status_line "$STATE/$(window_to_task "$w" "$STATE").status")"; }; then
clear_pause_tracking "$w"
fi
Expand All @@ -1012,6 +1113,7 @@ EOF
echo 0 > "$cf"
rm -f "$ssf" "$ewf"
task=$(window_to_task "$w" "$STATE")
[ -e "$sitf" ] && window_is_busy "$w" "$tail40" && rm -f "$sitf"
if ! afk_present && status_is_paused_or_captain_held "$(last_status_line "$STATE/$task.status")" && ! window_is_busy "$w" "$tail40"; then
case "$(pause_state_class "$w" "$task")" in
paused) handle_paused_stale "$w" "$task" "$h" ;;
Expand Down
Loading
Loading