Skip to content

feat: --settle returns the settled diff in the interaction response (#1101)#1106

Merged
thymikee merged 14 commits into
mainfrom
feat/interaction-settle
Jul 6, 2026
Merged

feat: --settle returns the settled diff in the interaction response (#1101)#1106
thymikee merged 14 commits into
mainfrom
feat/interaction-settle

Conversation

@thymikee

@thymikee thymikee commented Jul 4, 2026

Copy link
Copy Markdown
Member

Summary

Implements #1101: press|click|fill|longpress <target> --settle executes the action, waits for the UI to go quiet (the wait stable quiet-window loop, extracted to a shared stable-capture.ts and reused verbatim), and returns the settled observation in the same response — collapsing the dominant interact → observe agent-loop pair into one round trip.

Before / after:

press @e2            → Tapped @e2 (200, 322)
diff snapshot -i     → +@e4 [text] "Welcome!" …        (second call, second inference)

press @e2 --settle   → Tapped @e2 (200, 322)
                       settled after 812ms: +1 -1 (~14 unchanged)
                       + @e4 [text] "Welcome!"          (one call, one inference)

Design (as agreed on the issue):

  • Payload = settled DIFF vs the pre-action tree. Baseline is the --verify machinery's preActionNodes (ref/selector targets reuse the resolution capture; point targets opt into the evidence-baseline capture). The diff reuses the snapshot-diff machinery (buildSnapshotDiff, flattened like diff -i) and ships changed lines only (added/removed, bounded at 80 with a truncated marker); the unchanged bulk rides as diff.summary.unchanged. A full tree per interaction would invert the snapshot token-budget principle.
  • Best-effort, never an action failure. Quiet reached → settled: true + diff. Never-settling content (carousel/ticker) → the last capture's diff + settled: false + a hint (the tiny-tree hint style generalized). A broken or stalled settle capture reports itself in settle.hint; the action result is untouched.
  • Fresh refs ride along (Stale @refs silently resolve to the wrong node after the session tree changes #1076/feat: versioned snapshot refs with MCP auto-pinning #1096 integration). The final settled capture becomes the stored session snapshot; added diff lines carry structured ref bodies minted from it, and settle.refsGeneration rides the payload. A diff-carrying settle response is therefore ref-issuing: it clears snapshotRefsStale at the existing markSessionSnapshotRefsIssued choke point (the same accepted coarse blessing as find's single re-issued ref — documented at the choke point), and the MCP layer merge-only re-pins the added-line refs at the settle generation (SETTLE_REF_ISSUING_TOOLS beside REF_ISSUING_TOOLS; deliberately not IN REF_ISSUING_TOOLS — a plain press carries no generation and must never clear the pin scope). Diff-less settle payloads (stalled loop, sparse capture not stored) issue nothing and leave staleness untouched.
  • --settle --verify costs zero extra captures: the settle loop's final capture doubles as the verify evidence source.
  • Grammar: --settle (boolean opt-in) + --settle-quiet <ms> (quiet window, default 500ms) + the existing --timeout <ms> as the settle deadline (default 10s) — mirroring wait stable [quietMs] [timeoutMs]. Tuning flags without --settle are rejected (INVALID_ARGS). The four interaction descriptors declare a flag-sourced timeout budget with new envelope: 'widen' semantics: like wait's positional budget, --timeout extends the request envelope past the settle budget and never shrinks it (replay/prepare/snapshot keep their verbatim bound semantics).
  • Fast paths delegate: --settle disables the direct-iOS-selector and native-ref fast paths exactly like --verify (settling needs the tree-based baseline and captures).
  • Response construction stays single-site: the settle payload rides interactionResultExtrabuildInteractionResponseData (which injects refsGeneration); the construction guard passes unchanged.
  • Response levels: a conservative interactionSettleView digest keeps the verdict + diff.summary + refsGeneration and drops the line texts; non-settle responses pass through by reference at every level. full returns the default shape ("nothing richer is computed yet", like every existing view) — the issue's "full = whole interactive tree" is deferred because leveled views are pure functions of the default data, and the default payload must not carry the tree.

Guarantee matrix (ADR 0011): --settle adds no dispatch path, but it is a new cross-path response guarantee — added as the settleObservation row and classified for every path: runtime via settleAfterInteraction on runtime-selector / runtime-ref / coordinate, delegated (the flag disables the fast path) on direct-ios-selector / native-ref, inapplicable on the maestro replay path. No appliesTo scoping anywhere: the flag covers every command each path dispatches, and the gate rejects redundant full-coverage lists. One contract scenario per enforced/delegated cell (5 new); the registry-driven coverage gate enforces them. The gap/pin list is unchanged.

Default responses stay byte-identical without --settle (opt-in flag; the provider suite is the oracle and passes untouched). The snapshot-diff withRefs option is additive and off for the diff command, so its wire shape is unchanged too.

Live dogfood evidence is now included below: the Bluesky gpt-5.4-mini experiments cover token efficiency, command count, wall time, and behavior for --settle vs the classic interaction -> snapshot loop. Per-call cost when opted in remains the quiet window (default ~500ms) + captures.
Closes #1101.

Touched files: 48 — interaction command family plus its command-surface/MCP/descriptor/response-level projections; scope intentionally crosses those layers because the flag is a full command-surface addition (checklist steps 1–10).

Validation

Automated gates

  • Formatting / static: pnpm format, pnpm check:quick, git diff --check.
  • Focused unit/help/schema checks: pnpm exec vitest run src/commands/interaction/output.test.ts src/cli/parser/__tests__ src/utils/__tests__/command-schema-guards.test.ts (150 passed).
  • Build: pnpm build.
  • Existing branch coverage from the original implementation remains: settle runtime/daemon/MCP/response-view/timeout-policy tests, five settleObservation contract scenarios, provider transcript coverage, integration tests, and fallow/progress-model classification.
  • SkillGym planning cases were added for the observed agent mistakes: network-backed settle + wait text, raw output before shell projection, valid selector role filters, and treating the settled diff as the observation. Local focused SkillGym execution stayed silent for ~90s and was interrupted, so live gpt-mini workers below are the model-behavior evidence.

Live dogfood methodology

  • Target app/devices: logged-in Bluesky (xyz.blueskyweb.app) on iOS simulators iPhone 17 and iPhone 17 Pro.
  • Model/harness: codex exec --json -m gpt-5.4-mini workers. Agent-device commands were plain text only: no --json, no shell pipes, no jq/grep/stderr hiding.
  • Arms: five paired scenarios, each with one settle worker and one classic worker. Settle workers were instructed to use --settle on mutating interactions when the next decision depended on resulting UI. Classic workers were instructed not to use --settle and to observe with normal snapshots/find/get/is/wait after mutations.
  • Scenarios: notifications empty state, search for callstack, profile tab readback, home feed readback, and feed directory.
  • Device balancing: settle/classic arms were rotated across the two simulators. The valid run used one shared state dir per simulator to avoid cross-daemon runner lease conflicts; all sessions were closed and both simulator daemons were shut down after the run.
  • Metrics source: Codex JSONL turn.completed.usage for input, cached input, reasoning output, and output tokens; command-execution events for agent-device command count and tool output bytes; harness wall time for elapsed time. “Total tokens” here means input + reasoning output + output.
  • Run hygiene: an initial attempt was discarded after per-run state dirs created stale runner lease conflicts. In the valid run, one feed-directory classic worker hit transient model capacity before doing useful work; only that arm was retried and replaced in the adjusted summary.
  • Artifacts:
    • Adjusted summary: /private/tmp/ad-bluesky-dogfood-20260706-adjusted-summary.md
    • Main valid run logs: /private/tmp/ad-bluesky-dogfood-20260706-rerun-2026-07-06T10-43-39.528Z
    • Feed-directory classic retry: /private/tmp/ad-bluesky-dogfood-20260706-feed-directory-classic-retry

Aggregate adjusted result:

Arm Runs Success Total tokens Input Cached input Reasoning Output Agent-device commands Tool output bytes Wall time
settle 5 5 1,835,538 1,805,486 1,718,784 11,999 18,053 33 59,976 474s
classic 5 5 3,269,153 3,205,527 3,008,896 26,715 36,911 63 137,951 800s

Settle vs classic deltas: -43.9% total tokens, -43.7% input tokens, -55.1% reasoning tokens, -51.1% output tokens, -47.6% commands, -56.5% tool output bytes, -40.8% wall time.

Scenario deltas (settle minus classic):

Scenario Success Token delta Command delta Tool bytes delta Time delta
notifications-empty both true -295,074 -3 -30,918 -96s
search-callstack both true -198,486 -3 -27,531 -30s
profile-tab both true -268,162 -7 -2,351 -61s
home-feed both true -722,638 -20 -35,377 -145s
feed-directory both true +50,745 +3 +18,202 +5s

Feed-directory follow-up

The only losing scenario was feed-directory. Trace inspection showed the loss was dominated by planning mistakes, not settle semantics: an invalid selector (button="Search"), unnecessary help reads, and an extra snapshot after the settled diff already showed the feed-search UI.

Focused isolated gpt-5.4-mini workers on iPhone 17 Pro tested guidance variants:

Variant Success Tokens Commands Tool bytes Time
baseline true 1,060,367 17 21,585 247s
selector grammar guidance true 397,960 8 11,716 94s
settled diff as observation true 444,973 7 23,066 119s
task shortcut upper bound true 353,702 7 8,972 69s

Conclusion: improve agent guidance first. The branch now documents that selector role filters are role=button label="Search", not button="Search", and that a --settle diff is already the post-action observation when it contains the next target or final evidence. Focused artifacts: /private/tmp/ad-feeddir-variants-2026-07-06T13-02-45.404Z/summary.md.

Manual smoke checks

  • Reproduced Bluesky search on iPhone 17 Pro and confirmed plain fill --settle now prints the settled diff in CLI text output.
  • Reproduced selector-based tap output with press 'label="Notifications"' --settle; the CLI now prints the settled diff for selector/coordinate taps too, not just ref taps.
  • Verified cleanup after dogfood: both requested simulators were shut down and no ad-dogfood / feed-directory worker processes remained.

Known gaps / follow-ups:

  • A repo-owned model-token dogfood harness is still follow-up work; the experiment harnesses above were local /private/tmp scripts for controlled measurement.
  • full-level whole-tree settle view remains deferred.
  • Ref-target baselines reuse the stored session snapshot, so a non--i stored tree can over-report removals in the settled diff — the same baseline caveat --verify's changedFromBefore already accepts; noted in code.
  • The shared stable loop now derives its poll cadence from the quiet window (min(300ms, max(25ms, quietMs))); wait stable at the default 500ms quiet window is byte-identical, sub-300ms quiet windows poll faster.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.5 MB 1.6 MB +11.9 kB
JS gzip 495.9 kB 499.4 kB +3.5 kB
npm tarball 591.7 kB 596.3 kB +4.6 kB
npm unpacked 2.1 MB 2.1 MB +14.1 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.4 ms 29.1 ms +0.7 ms
CLI --help 53.0 ms 55.6 ms +2.6 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/registry.js +126.2 kB +36.9 kB
dist/src/device-rotation.js +72.3 kB +23.3 kB
dist/src/runtime.js -4.6 kB -1.6 kB
dist/src/cli-help.js +1.9 kB +638 B
dist/src/sdk-batch.js +831 B +249 B

@thymikee

thymikee commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Reviewed blessing-scenario-first, then dogfooded on Bluesky (real sim). Verdict: the core semantics are right, and two content-policy fixes landed from dogfooding.

Review (primary-justification scenarios, all verified in code + tests):

  • Generation integrity: storeSettledSnapshot routes through createDaemonRuntimeSessionStore.setRecordsetSessionSnapshot, so the generation bumps before settleRefsGenerationIssue reads it. Settle refs pin at G+1, pre-press pins stay at G and warn precisely — no re-blessing (the feat: versioned snapshot refs with MCP auto-pinning #1096 failure mode is structurally absent).
  • MCP merge path is conditional on settle.refsGeneration + diff presence; a plain press never touches the pin scope (tested). Removed lines never carry ref (enforced in the diff builder, documented on SnapshotDiffLine), so dead baseline refs can't be pinned.
  • diff ⇔ refs-issued holds at the source: the diff is only attached when the settled tree became the stored snapshot (non-sparse), and the e2e proves the follow-up press @e2 resolves on the stored settled tree with zero extra runner requests.

Fixes pushed to this branch:

  1. SettleParams moved to contracts/interaction.ts — the daemon import of commands/…/settle.ts violated the layering DAG (the local gate chain didn't include scripts/layering/check.ts; worth adding to the AGENTS.md gate list).
  2. Content-first diff policy, from Bluesky dogfood: a fill that summons the keyboard spent 49/80 capped lines on QWERTY [key] nodes, and a screen transition with 269 removals could starve out added lines entirely. Key-type nodes are now filtered from both diff sides ([keyboard] container survives as the signal) and added lines (the fresh-ref half) win cap slots over removals. Unit-tested.

Dogfood evidence (iPhone 17 Pro sim, Bluesky dev build): the flagship loop worked with zero intermediate snapshots — press @e69 --settle (search tab) returned a diff whose added lines included @e23 [text-field] "Search", and fill @e23 alpenglow --settle acted on it directly. Settle waits observed: 1.0–2.3s, 3–4 captures, payload ~11KB vs ~28KB for a full snapshot -i.

Known trade-off confirmed on-device: network-backed content (search typeahead) can arrive after the quiet window — the diff honestly reflects the settled-but-still-loading screen and the agent needs a follow-up (wait text). This is the designed behavior; the end-to-end benchmark now running will show whether it matters in practice.

@thymikee

thymikee commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

End-to-end agent benchmark: acceptance evidence

Headless claude -p agents (Bash tool only, max 40 turns) drove the real CLI against Bluesky on an iPhone 17 Pro simulator. Task: search "alpenglow" → report first account handle → Notifications tab → first label → print DONE: line. Hard environment reset between runs. 27 runs total across arms; success = correct DONE line.

Steered prompt: classic act→observe loop vs --settle loop

arm success wall (med) turns (med) cost (med)
classic × haiku 0/3 174s (all max-turns) 41 $0.31
settle × haiku 3/3 111s 19 $0.25
classic × sonnet 3/3 269s 35 $1.23
settle × sonnet 3/3 358s 36 $1.50

--settle takes Haiku from 0% to 100%. Classic-Haiku drowns in observation management (25 snapshots + 18 file re-reads in 40 turns; 1.4–2.1M cache-read tokens). With --settle it finishes in ~18–19 turns at a quarter the wall time of Sonnet. For Sonnet, --settle is reliability-neutral (3/3 both) and mildly wall/cost-negative in this sample — Sonnet already runs an efficient diff-snapshot loop, and settle's per-action quiet-window waits (~1–3s each) don't buy it turns. Matches the opt-in design: it's a small-model/reliability feature, not a universal accelerator.

Instruction minimalism: does the CLI teach itself?

Bare prompt (no tips), or bare prompt + "run --help first":

  • Sonnet: 5/5 across minimal + help-discovery. Best run overall: 12 turns / 137s / $0.54 — it read --help, discovered --settle on its own, and used it on every mutating action. The help-workflow line ships the whole loop to strong models with zero steering.
  • Haiku, original help: 1/4 — falls into plain-snapshot 38KB-payload spirals.
  • Haiku, after adding one core-loop line to the top-level help (open → snapshot -i → press/fill --settle → repeat, commit e619c1c): 2/3, both successes at 21–24 turns following the taught loop exactly (snapshot -i first, --settle everywhere, one-retry recovery from the @-prefix hint). The single failure shows five 90-second inter-turn gaps — API rate limiting, not agent-device.

Environment reliability

The profile-screen capture wedge (#1105, deterministic repro posted there) was routed around in this benchmark and is being fixed on fix/ios-capture-stall-recovery. Two further findings filed: iOS fill types at ~700ms/char (task chip), and small models blind themselves with 2>/dev/null | jq '.data.x' projections of error JSON — the existing stdout-JSON error contract is what saves them when they drop the projection.

Verdict: the PR does what it set out to do. The settled-diff loop is the difference between a small model failing 100% and succeeding 100% on a real multi-screen task, and the loop is discoverable from --help alone by both model classes once the core-loop line states it. Recommending un-drafting after the #1105 fix lands.

@thymikee thymikee force-pushed the feat/interaction-settle branch from e619c1c to cafb16f Compare July 5, 2026 08:48
@thymikee thymikee marked this pull request as ready for review July 5, 2026 08:54
@thymikee

thymikee commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Un-drafting: all acceptance criteria are met.

Deferred (stated in PR body, unchanged): README/website/SkillGym doc updates; #1076 enforcement decision remains dated + measurement-gated.

@thymikee

thymikee commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Review pass after the latest rebase/un-draft: no actionable blockers found.

Checked against #1101 and the changed command path: CLI/typed input exposes --settle and its budget flags; interaction fast paths are disabled when settling needs a tree baseline; runtime stores the final settled capture before issuing added-line refs; daemon response construction only attaches settle.refsGeneration when a diff exists; MCP merge-pins only those added refs; and the timeout policy widens the request envelope while preserving daemon sessions on capture stalls. The contract/provider tests cover the shipped route, including follow-up action on a diff-issued ref.

CI is green, and the PR now includes the live Bluesky benchmark/replay evidence requested for the device-facing behavior. Known deferred README/website/SkillGym updates are explicitly called out in the PR body/comment, so this is ready for human maintainer judgment rather than more automation review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 5, 2026
@thymikee

thymikee commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Follow-up independent review/testing pass:

  • gpt-5.5 review from clean worktree /private/tmp/agent-device-pr-1106-review on head cafb16fd8c84e3b84e2a1ffb42ba822fe770c4ae: no blockers. Traced CLI/client flags, command descriptors, timeout policy, daemon routing, runtime settle capture/diff storage, response views, MCP ref pinning, and ADR 0011 interaction guarantees. Validation passed: pnpm check:quick, focused settle/interaction Vitest suites, pnpm format:check, pnpm test:integration:provider, and pnpm test:coverage.
  • codex-mini-style sample-app pass from clean worktree /private/tmp/agent-device-pr1106 on the same head: pnpm build, pnpm clean:daemon, iOS runner prep, Settings simulator flow, and repo Agent Device Tester flow. press --settle returned useful diff/fresh-ref data on both apps (settled: true, 1488ms/3 captures for Settings; 860ms/2 captures for the demo app). wait stable stayed bounded at ~850-870ms.

Residual gap: Bluesky was installed on another simulator, but that device was leased by another daemon, so it was not exercised in this pass. The blocker was external lease/device availability rather than a PR failure. Based on the two clean-worker passes, I still consider this ready for maintainer judgment.

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Follow-up from an independent review: this should not be considered ready yet.

  • responseLevel: digest drops settle.diff.lines, but MCP ref auto-pinning currently collects refs only from those lines. That means digest settle responses can carry refsGeneration without recording pins for added-line refs. Issue fix(mcp): settle digest responses must retain refs for auto-pinning #1124 tracks the fix: keep a capped digest-level refs array and teach MCP merge to read it when lines are absent.
  • settleFlagGuardResponse now rejects bare timeoutMs without --settle; main silently ignored orphan tuning flags on click/press/fill. Please either document the breaking requirement or preserve compatibility.
  • A material settle.diff should clear the non-hittable "may have had no visible effect" hint the same way --verify evidence does.
  • Daemon envelope widening should account for selector resolution/action overhead in addition to the post-action settle timeout.
  • Please add contract coverage for a never-quiet UI that still stores a diff, proving added-line refs stay actionable.

I am removing ready-for-human until the digest ref-pinning blocker is resolved or explicitly deferred by a maintainer.

@thymikee thymikee removed the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 6, 2026
@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Implemented the review-blocker follow-up for settle digest/MCP pinning and the adjacent compatibility/runtime concerns.

Summary

  • responseLevel: digest now mirrors snapshot digest behavior by keeping a capped settle.refs array from added diff lines, while still omitting verbose settle.diff.lines.
  • MCP auto-pinning now merges settle-issued refs from either settle.diff.lines or digest-level settle.refs, so digest responses can still feed follow-up @ref actions.
  • Added a never-settled/still-diffed provider scenario proving added settle refs remain actionable even when settled: false and diff.lines are absent from the compact response.
  • Preserved bare timeoutMs compatibility on touch commands without --settle; only orphan settle tuning flags remain rejected.
  • Cleared the non-hittable/no-visible-effect hint when settle diff proves a material post-action change.
  • Widened the daemon request envelope for --settle --timeout as selector/action envelope + settle budget + margin, rather than only post-action settle budget + margin.

Validation

Fresh validation added for these fixes:

  • pnpm exec vitest run src/daemon/__tests__/response-views.test.ts src/mcp/__tests__/command-tools.test.ts src/utils/__tests__/daemon-client.test.ts src/daemon/handlers/__tests__/interaction-settle.test.ts src/commands/interaction/runtime/interactions.test.ts test/integration/provider-scenarios/settle-observation.test.ts
  • pnpm format
  • pnpm check:quick
  • pnpm build
  • pnpm check:unit after build: 351 unit files / 3282 tests passed, built-CLI smoke 14 passed / 1 env-skipped
  • pnpm test:integration:provider
  • pnpm test:coverage
  • git diff --check

Existing evidence still applies to the core settle behavior because this patch does not change the action/settle capture loop itself: the prior Bluesky dogfood/benchmark, the independent clean-worktree review pass, and the codex-mini sample-app pass all remain relevant for simulator/device behavior. The new evidence above is specifically for digest-level MCP pinning, timeout compatibility/envelope behavior, hint reconciliation, and the never-settled actionable-ref case.

Benchmark note

I did not run a fresh 3x classic vs 3x --settle live-model benchmark from this worker. The practical blocker is that this environment does not have a confirmed same-state Bluesky/login setup or a configured mini/small-model e2e runner with token/cost telemetry available to execute six comparable agent trials without adding new external setup variance. The older benchmark evidence on the PR remains the safe performance comparison to cite; the safe conclusion from this follow-up is narrower: digest-level settle refs are now preserved and merge-pinned, including the still-diffed/never-settled case.

Residual risk: no new live model/device benchmark was added in this pass; the newly changed paths are covered by focused unit/contract/provider tests and the aggregate local gates above.

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Updated PR #1106 with a focused Fallow fix.\n\nChanged files:\n- src/daemon/response-views.ts\n- src/daemon/handlers/tests/interaction-settle.test.ts\n\nValidation:\n- pnpm exec vitest run src/daemon/tests/response-views.test.ts src/daemon/handlers/tests/interaction-settle.test.ts\n- pnpm check:fallow --base origin/main\n- pnpm format\n- pnpm check:quick

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Ready for human review.

I rechecked #1106 after the final fix commit 7c8a7cbe: all 21 PR checks are passing. The previous blocker around digest-level settle refs was addressed by retaining/merging refs for auto-pinning, and the later Fallow-only cleanup is green.

Residual risk is limited to the already-discussed live-device coverage: the contract and MCP paths are covered, but broader real-app settle benchmarking remains a follow-up evaluation rather than a merge blocker for this PR.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Jul 6, 2026
@thymikee thymikee force-pushed the feat/interaction-settle branch from 7c8a7cb to 3db2325 Compare July 6, 2026 13:31
@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Rechecked after latest head 3db2325c (fix: surface settle output in CLI text). The new changes are targeted: Fallow complexity cleanup in response-views.ts, plus CLI text formatting so press/fill --settle surface the settle verdict and changed lines. Focused tests were added in src/commands/interaction/output.test.ts and src/daemon/handlers/__tests__/interaction-settle.test.ts; all 21 PR checks are passing. I do not see a new blocker from the latest commits; ready-for-human still looks appropriate.

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

CI regression after latest head b98b7b52 (fix: complete settle handling for longpress): Fallow Code Quality is failing.

Actionable findings from the job:

  • duplicate block dup:bd84ce85 between src/daemon/request-router.ts:259-263 and src/daemon/server/daemon-runtime.ts:325-329 (5 duplicated lines)
  • changed-file complexity now gates on src/daemon/client/daemon-client.ts: sendToDaemon and resolveDaemonRequestTimeoutMs; the latest settle envelope/timeout changes likely need a small helper extraction rather than more inline branching

Run locally with the CI base shown in the job:
pnpm check:fallow --base be4bd092b6b0b6b770ed288936bac970be335da0

Please clear Fallow before this can stay ready-for-human; the label is now stale until the check is green again.

thymikee added 10 commits July 6, 2026 17:22
…1101)

press/click/fill/longpress --settle executes the action, waits for the UI
to go quiet (wait stable's loop, shared via stable-capture.ts), and returns
the settled diff vs the pre-action tree in the same response — one round
trip instead of the interact -> observe pair.

- payload: changed lines only (bounded), summary counts, added-line refs,
  refsGeneration; best-effort (settled:false + hint on never-quiet content,
  never an action failure); --verify shares the settle captures
- ref issuance: the settled tree becomes the session snapshot; a
  diff-carrying settle response clears snapshotRefsStale and the MCP layer
  merge-only re-pins added-line refs at the settle generation
- grammar: --settle + --settle-quiet <ms> + --timeout <ms> (flag-sourced
  descriptor budget with new envelope:'widen' semantics mirroring wait)
- ADR 0011: new settleObservation guarantee classified on every path with
  contract scenarios per enforced/delegated cell
The doctor provider scenarios sit at ~5s of real daemon-harness work on a
loaded host and flake at vitest's 5s default during full-suite runs (the
known contention flake AGENTS.md documents). Same in-file precedent as the
Metro-probe scenario's 10s budget.
daemon/handlers/interaction-flags.ts imported the type across the
daemon -> commands boundary (R2 commands-floor). The tuning params are
part of the interaction contract like SettleObservation, so they live
in contracts/interaction.ts and both layers import from there.
…in the cap

Bluesky dogfood: a fill that summons the iOS keyboard spent 49 of the 80
capped diff lines spelling out QWERTY keys, and a screen transition with
269 removals could starve out the added lines entirely. Key-type nodes
are now filtered from both diff sides (the [keyboard] container line
still signals presence), and under truncation added lines — the ones
carrying fresh refs — win slots over removals.
Benchmarked with headless haiku/sonnet agents given only --help: both
models skipped the help-workflow pointer and started with plain
snapshot (38KB payloads they then had to re-read from files). One
core-loop line at the starting point is what teaches snapshot -i and
--settle to models that never read a second help page.
@thymikee thymikee force-pushed the feat/interaction-settle branch from b98b7b5 to cd69255 Compare July 6, 2026 15:36
@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Rechecked #1106 at latest head cd692559 after the Fallow timeout-envelope refactor. All 21 checks are passing, including Fallow, Integration Tests, Coverage, Smoke, Typecheck, and Swift runner compile.

Review focus:

  • The latest commit only extracts resolveDaemonRequestTimeoutMs into daemon-client-timeout.ts; the settle envelope behavior remains covered: --settle --timeout 120000 widens to normal touch envelope + settle budget + margin, bare timeoutMs without --settle stays ignored for compatibility, and small settle budgets do not shrink the base request envelope.
  • The previous MCP digest blocker is covered: digest-level settle responses keep capped settle.refs, and MCP auto-pinning merges those refs even for never-settled diffs.
  • Route-level provider coverage exercises open -> snapshot -> press --settle -> press @settledRef, including never-settled content, so the shipped daemon path is covered rather than only helper mocks.

No new blockers found. The existing ready-for-human label still looks correct. Residual risk remains the previously noted live-app/agent-loop benchmark work, not a merge blocker for this contract implementation.

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Rechecked #1106 at latest head e291d4f after the post-action observation refactor. All 21 checks are now green: Fallow, Integration Tests, Coverage, Smoke, Typecheck, Swift runner compile, and the rest.

Targeted review of the new delta:

  • post-action-observation.ts centralizes the existing --verify / --settle composition without changing the route contract: settle still uses the final capture for verify evidence, and verify-only still takes one best-effort post-action capture.
  • press/click/fill still disable native-ref fast paths when a pre-action baseline is needed, preserving the settle/verify behavior reviewed earlier.
  • The non-hittable hint reconciliation was moved, not relaxed: hints are still dropped only when verify evidence or a material settle diff proves a post-action change.
  • longpress --settle now goes through the same shared observation helper, matching the intended cleanup from the previous Fallow fix.

No new blockers found. ready-for-human remains appropriate.

@thymikee

thymikee commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Rechecked #1106 at latest head 553f023 after the descriptor helper trim. All 21 checks are green.

The latest delta only removes unused exported settle helper wrappers and updates tests to derive/anchor the same settle-capable command set (click, fill, longpress, press). Timeout policy remains tied to resolvePostActionObservationSupport, and the interaction guarantee text now points at the descriptor trait rather than local flag checks. No new blockers found; ready-for-human still looks correct.

@thymikee thymikee merged commit 5c5fa01 into main Jul 6, 2026
21 checks passed
@thymikee thymikee deleted the feat/interaction-settle branch July 6, 2026 18:18
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-06 18:19 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

press/click/fill --settle: settled observation in the interaction response

1 participant