Dogfooding the pew_w154_diff1 example end-to-end (ingest → plan → approve → export → run 200 gpt-5.5 twins → import → twin-validate) surfaced a cluster of golden-path guidance bugs: zwill next and inline next_steps route users to non-runnable or incomplete commands, and never surface twin-validate — the decisive leakage + conditional-baseline + bootstrap-CI validation the guide insists on. A user following the printed guidance verbatim stops at a bare twin score, exactly the "positive result from a bare twin run" the guide warns against.
Fixes prepared on branch fix/golden-path-next-steps (full suite green, 261 passed).
F2/F3 — zwill next at run_twins suggests a non-runnable command
next_command was:
zwill edsl-export --survey <s> --target twin-probability-job --heldout-questions <q1,q2> --context-question-count 8 --complete-cases --model openai:gpt-5.5 --path twin.edsl.json
which (a) errors with approval_required (it needs an approved twin-experiment plan or --allow-unapproved), and (b) hardcodes --context-question-count 8 even for surveys with fewer questions (pew_w154_diff1 has 5). Fix: route to the runnable, gated plan flow (twin-experiment init-plan → validate → approve → export-plan → run → import).
F7 — inline next_steps never reach twin-validate
twin-results import next_steps ended at twin-results report (twin-only).
twin-experiment export-plan next_steps went import → twin-experiment compare (twin-only).
Neither surfaces twin-validate. Fix: lead/insert twin-validate in both. (Note: zwill next itself does route to twin-validate at the validate stage — the bug is that the inline next_steps disagree with it.)
F1 — status re-suggests commit after commit
zwill status always emitted next_steps: ["zwill commit --survey <survey>"] even when every survey was already committed. Fix: suggest commit only for uncommitted surveys; otherwise zwill next.
Minor / not changed
- F5:
twin-experiment compare defaults to a rich table while most commands default to JSON (it does support --format json/csv). Left as-is; flagging for consistency.
Verified NOT bugs (checked and retracted)
- Output-root relocation of
--path/--out is documented + intentional ("contained under an output root … override with ZWILL_OUT"); the command reports the real path.
twin-results report / twin-experiment compare "no output" was a rich table (not JSON), not a failure.
Happy to open a PR (branch is ready).
Dogfooding the
pew_w154_diff1example end-to-end (ingest → plan → approve → export → run 200 gpt-5.5 twins → import →twin-validate) surfaced a cluster of golden-path guidance bugs:zwill nextand inlinenext_stepsroute users to non-runnable or incomplete commands, and never surfacetwin-validate— the decisive leakage + conditional-baseline + bootstrap-CI validation the guide insists on. A user following the printed guidance verbatim stops at a bare twin score, exactly the "positive result from a bare twin run" the guide warns against.Fixes prepared on branch
fix/golden-path-next-steps(full suite green, 261 passed).F2/F3 —
zwill nextatrun_twinssuggests a non-runnable commandnext_commandwas:zwill edsl-export --survey <s> --target twin-probability-job --heldout-questions <q1,q2> --context-question-count 8 --complete-cases --model openai:gpt-5.5 --path twin.edsl.jsonwhich (a) errors with
approval_required(it needs an approved twin-experiment plan or--allow-unapproved), and (b) hardcodes--context-question-count 8even for surveys with fewer questions (pew_w154_diff1 has 5). Fix: route to the runnable, gated plan flow (twin-experiment init-plan → validate → approve → export-plan → run → import).F7 — inline
next_stepsnever reachtwin-validatetwin-results importnext_steps ended attwin-results report(twin-only).twin-experiment export-plannext_steps wentimport → twin-experiment compare(twin-only).Neither surfaces
twin-validate. Fix: lead/inserttwin-validatein both. (Note:zwill nextitself does route totwin-validateat thevalidatestage — the bug is that the inline next_steps disagree with it.)F1 —
statusre-suggestscommitafter commitzwill statusalways emittednext_steps: ["zwill commit --survey <survey>"]even when every survey was already committed. Fix: suggestcommitonly for uncommitted surveys; otherwisezwill next.Minor / not changed
twin-experiment comparedefaults to a rich table while most commands default to JSON (it does support--format json/csv). Left as-is; flagging for consistency.Verified NOT bugs (checked and retracted)
--path/--outis documented + intentional ("contained under an output root … override withZWILL_OUT"); the command reports the real path.twin-results report/twin-experiment compare"no output" was a rich table (not JSON), not a failure.Happy to open a PR (branch is ready).