feat: --settle returns the settled diff in the interaction response (#1101)#1106
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
|
Reviewed blessing-scenario-first, then dogfooded on Bluesky (real sim). Verdict: the core semantics are right, and two content-policy fixes landed from dogfooding. Review (primary-justification scenarios, all verified in code + tests):
Fixes pushed to this branch:
Dogfood evidence (iPhone 17 Pro sim, Bluesky dev build): the flagship loop worked with zero intermediate snapshots — Known trade-off confirmed on-device: network-backed content (search typeahead) can arrive after the quiet window — the diff honestly reflects the settled-but-still-loading screen and the agent needs a follow-up ( |
End-to-end agent benchmark: acceptance evidenceHeadless Steered prompt: classic act→observe loop vs --settle loop
--settle takes Haiku from 0% to 100%. Classic-Haiku drowns in observation management (25 snapshots + 18 file re-reads in 40 turns; 1.4–2.1M cache-read tokens). With --settle it finishes in ~18–19 turns at a quarter the wall time of Sonnet. For Sonnet, --settle is reliability-neutral (3/3 both) and mildly wall/cost-negative in this sample — Sonnet already runs an efficient diff-snapshot loop, and settle's per-action quiet-window waits (~1–3s each) don't buy it turns. Matches the opt-in design: it's a small-model/reliability feature, not a universal accelerator. Instruction minimalism: does the CLI teach itself?Bare prompt (no tips), or bare prompt + "run --help first":
Environment reliabilityThe profile-screen capture wedge (#1105, deterministic repro posted there) was routed around in this benchmark and is being fixed on Verdict: the PR does what it set out to do. The settled-diff loop is the difference between a small model failing 100% and succeeding 100% on a real multi-screen task, and the loop is discoverable from |
e619c1c to
cafb16f
Compare
|
Un-drafting: all acceptance criteria are met.
Deferred (stated in PR body, unchanged): README/website/SkillGym doc updates; #1076 enforcement decision remains dated + measurement-gated. |
|
Review pass after the latest rebase/un-draft: no actionable blockers found. Checked against #1101 and the changed command path: CLI/typed input exposes CI is green, and the PR now includes the live Bluesky benchmark/replay evidence requested for the device-facing behavior. Known deferred README/website/SkillGym updates are explicitly called out in the PR body/comment, so this is ready for human maintainer judgment rather than more automation review. |
|
Follow-up independent review/testing pass:
Residual gap: Bluesky was installed on another simulator, but that device was leased by another daemon, so it was not exercised in this pass. The blocker was external lease/device availability rather than a PR failure. Based on the two clean-worker passes, I still consider this ready for maintainer judgment. |
|
Follow-up from an independent review: this should not be considered ready yet.
I am removing |
|
Implemented the review-blocker follow-up for settle digest/MCP pinning and the adjacent compatibility/runtime concerns. Summary
ValidationFresh validation added for these fixes:
Existing evidence still applies to the core settle behavior because this patch does not change the action/settle capture loop itself: the prior Bluesky dogfood/benchmark, the independent clean-worktree review pass, and the codex-mini sample-app pass all remain relevant for simulator/device behavior. The new evidence above is specifically for digest-level MCP pinning, timeout compatibility/envelope behavior, hint reconciliation, and the never-settled actionable-ref case. Benchmark noteI did not run a fresh 3x classic vs 3x Residual risk: no new live model/device benchmark was added in this pass; the newly changed paths are covered by focused unit/contract/provider tests and the aggregate local gates above. |
|
Updated PR #1106 with a focused Fallow fix.\n\nChanged files:\n- src/daemon/response-views.ts\n- src/daemon/handlers/tests/interaction-settle.test.ts\n\nValidation:\n- pnpm exec vitest run src/daemon/tests/response-views.test.ts src/daemon/handlers/tests/interaction-settle.test.ts\n- pnpm check:fallow --base origin/main\n- pnpm format\n- pnpm check:quick |
|
Ready for human review. I rechecked #1106 after the final fix commit Residual risk is limited to the already-discussed live-device coverage: the contract and MCP paths are covered, but broader real-app settle benchmarking remains a follow-up evaluation rather than a merge blocker for this PR. |
7c8a7cb to
3db2325
Compare
|
Rechecked after latest head |
|
CI regression after latest head Actionable findings from the job:
Run locally with the CI base shown in the job: Please clear Fallow before this can stay |
…1101) press/click/fill/longpress --settle executes the action, waits for the UI to go quiet (wait stable's loop, shared via stable-capture.ts), and returns the settled diff vs the pre-action tree in the same response — one round trip instead of the interact -> observe pair. - payload: changed lines only (bounded), summary counts, added-line refs, refsGeneration; best-effort (settled:false + hint on never-quiet content, never an action failure); --verify shares the settle captures - ref issuance: the settled tree becomes the session snapshot; a diff-carrying settle response clears snapshotRefsStale and the MCP layer merge-only re-pins added-line refs at the settle generation - grammar: --settle + --settle-quiet <ms> + --timeout <ms> (flag-sourced descriptor budget with new envelope:'widen' semantics mirroring wait) - ADR 0011: new settleObservation guarantee classified on every path with contract scenarios per enforced/delegated cell
The doctor provider scenarios sit at ~5s of real daemon-harness work on a loaded host and flake at vitest's 5s default during full-suite runs (the known contention flake AGENTS.md documents). Same in-file precedent as the Metro-probe scenario's 10s budget.
daemon/handlers/interaction-flags.ts imported the type across the daemon -> commands boundary (R2 commands-floor). The tuning params are part of the interaction contract like SettleObservation, so they live in contracts/interaction.ts and both layers import from there.
…in the cap Bluesky dogfood: a fill that summons the iOS keyboard spent 49 of the 80 capped diff lines spelling out QWERTY keys, and a screen transition with 269 removals could starve out the added lines entirely. Key-type nodes are now filtered from both diff sides (the [keyboard] container line still signals presence), and under truncation added lines — the ones carrying fresh refs — win slots over removals.
Benchmarked with headless haiku/sonnet agents given only --help: both models skipped the help-workflow pointer and started with plain snapshot (38KB payloads they then had to re-read from files). One core-loop line at the starting point is what teaches snapshot -i and --settle to models that never read a second help page.
b98b7b5 to
cd69255
Compare
|
Rechecked #1106 at latest head Review focus:
No new blockers found. The existing |
|
Rechecked #1106 at latest head Targeted review of the new delta:
No new blockers found. |
|
Rechecked #1106 at latest head The latest delta only removes unused exported settle helper wrappers and updates tests to derive/anchor the same settle-capable command set ( |
|
Summary
Implements #1101:
press|click|fill|longpress <target> --settleexecutes the action, waits for the UI to go quiet (thewait stablequiet-window loop, extracted to a sharedstable-capture.tsand reused verbatim), and returns the settled observation in the same response — collapsing the dominant interact → observe agent-loop pair into one round trip.Before / after:
Design (as agreed on the issue):
--verifymachinery'spreActionNodes(ref/selector targets reuse the resolution capture; point targets opt into the evidence-baseline capture). The diff reuses the snapshot-diff machinery (buildSnapshotDiff, flattened likediff -i) and ships changed lines only (added/removed, bounded at 80 with atruncatedmarker); the unchanged bulk rides asdiff.summary.unchanged. A full tree per interaction would invert the snapshot token-budget principle.settled: true+ diff. Never-settling content (carousel/ticker) → the last capture's diff +settled: false+ a hint (the tiny-tree hint style generalized). A broken or stalled settle capture reports itself insettle.hint; the action result is untouched.refbodies minted from it, andsettle.refsGenerationrides the payload. A diff-carrying settle response is therefore ref-issuing: it clearssnapshotRefsStaleat the existingmarkSessionSnapshotRefsIssuedchoke point (the same accepted coarse blessing asfind's single re-issued ref — documented at the choke point), and the MCP layer merge-only re-pins the added-line refs at the settle generation (SETTLE_REF_ISSUING_TOOLSbesideREF_ISSUING_TOOLS; deliberately not INREF_ISSUING_TOOLS— a plain press carries no generation and must never clear the pin scope). Diff-less settle payloads (stalled loop, sparse capture not stored) issue nothing and leave staleness untouched.--settle --verifycosts zero extra captures: the settle loop's final capture doubles as the verify evidence source.--settle(boolean opt-in) +--settle-quiet <ms>(quiet window, default 500ms) + the existing--timeout <ms>as the settle deadline (default 10s) — mirroringwait stable [quietMs] [timeoutMs]. Tuning flags without--settleare rejected (INVALID_ARGS). The four interaction descriptors declare a flag-sourced timeout budget with newenvelope: 'widen'semantics: likewait's positional budget,--timeoutextends the request envelope past the settle budget and never shrinks it (replay/prepare/snapshot keep their verbatimboundsemantics).--settledisables the direct-iOS-selector and native-ref fast paths exactly like--verify(settling needs the tree-based baseline and captures).interactionResultExtra→buildInteractionResponseData(which injectsrefsGeneration); the construction guard passes unchanged.interactionSettleViewdigest keeps the verdict +diff.summary+refsGenerationand drops the line texts; non-settle responses pass through by reference at every level.fullreturns the default shape ("nothing richer is computed yet", like every existing view) — the issue's "full = whole interactive tree" is deferred because leveled views are pure functions of the default data, and the default payload must not carry the tree.Guarantee matrix (ADR 0011):
--settleadds no dispatch path, but it is a new cross-path response guarantee — added as thesettleObservationrow and classified for every path:runtimeviasettleAfterInteractionon runtime-selector / runtime-ref / coordinate,delegated(the flag disables the fast path) on direct-ios-selector / native-ref,inapplicableon the maestro replay path. NoappliesToscoping anywhere: the flag covers every command each path dispatches, and the gate rejects redundant full-coverage lists. One contract scenario per enforced/delegated cell (5 new); the registry-driven coverage gate enforces them. The gap/pin list is unchanged.Default responses stay byte-identical without
--settle(opt-in flag; the provider suite is the oracle and passes untouched). The snapshot-diffwithRefsoption is additive and off for thediffcommand, so its wire shape is unchanged too.Live dogfood evidence is now included below: the Bluesky gpt-5.4-mini experiments cover token efficiency, command count, wall time, and behavior for
--settlevs the classic interaction -> snapshot loop. Per-call cost when opted in remains the quiet window (default ~500ms) + captures.Closes #1101.
Touched files: 48 — interaction command family plus its command-surface/MCP/descriptor/response-level projections; scope intentionally crosses those layers because the flag is a full command-surface addition (checklist steps 1–10).
Validation
Automated gates
pnpm format,pnpm check:quick,git diff --check.pnpm exec vitest run src/commands/interaction/output.test.ts src/cli/parser/__tests__ src/utils/__tests__/command-schema-guards.test.ts(150 passed).pnpm build.settleObservationcontract scenarios, provider transcript coverage, integration tests, and fallow/progress-model classification.wait text, raw output before shell projection, valid selector role filters, and treating the settled diff as the observation. Local focused SkillGym execution stayed silent for ~90s and was interrupted, so live gpt-mini workers below are the model-behavior evidence.Live dogfood methodology
xyz.blueskyweb.app) on iOS simulatorsiPhone 17andiPhone 17 Pro.codex exec --json -m gpt-5.4-miniworkers. Agent-device commands were plain text only: no--json, no shell pipes, no jq/grep/stderr hiding.settleworker and oneclassicworker. Settle workers were instructed to use--settleon mutating interactions when the next decision depended on resulting UI. Classic workers were instructed not to use--settleand to observe with normal snapshots/find/get/is/wait after mutations.callstack, profile tab readback, home feed readback, and feed directory.turn.completed.usagefor input, cached input, reasoning output, and output tokens; command-execution events for agent-device command count and tool output bytes; harness wall time for elapsed time. “Total tokens” here meansinput + reasoning output + output./private/tmp/ad-bluesky-dogfood-20260706-adjusted-summary.md/private/tmp/ad-bluesky-dogfood-20260706-rerun-2026-07-06T10-43-39.528Z/private/tmp/ad-bluesky-dogfood-20260706-feed-directory-classic-retryAggregate adjusted result:
Settle vs classic deltas: -43.9% total tokens, -43.7% input tokens, -55.1% reasoning tokens, -51.1% output tokens, -47.6% commands, -56.5% tool output bytes, -40.8% wall time.
Scenario deltas (settle minus classic):
Feed-directory follow-up
The only losing scenario was feed-directory. Trace inspection showed the loss was dominated by planning mistakes, not settle semantics: an invalid selector (
button="Search"), unnecessary help reads, and an extra snapshot after the settled diff already showed the feed-search UI.Focused isolated gpt-5.4-mini workers on iPhone 17 Pro tested guidance variants:
Conclusion: improve agent guidance first. The branch now documents that selector role filters are
role=button label="Search", notbutton="Search", and that a--settlediff is already the post-action observation when it contains the next target or final evidence. Focused artifacts:/private/tmp/ad-feeddir-variants-2026-07-06T13-02-45.404Z/summary.md.Manual smoke checks
fill --settlenow prints the settled diff in CLI text output.press 'label="Notifications"' --settle; the CLI now prints the settled diff for selector/coordinate taps too, not just ref taps.ad-dogfood/ feed-directory worker processes remained.Known gaps / follow-ups:
/private/tmpscripts for controlled measurement.full-level whole-tree settle view remains deferred.-istored tree can over-report removals in the settled diff — the same baseline caveat--verify'schangedFromBeforealready accepts; noted in code.min(300ms, max(25ms, quietMs)));wait stableat the default 500ms quiet window is byte-identical, sub-300ms quiet windows poll faster.