Skip to content

Golden-path guidance routes users away from the decisive twin-validate flow (F1/F2/F3/F7) #80

Description

@johnjosephhorton

Dogfooding the pew_w154_diff1 example end-to-end (ingest → plan → approve → export → run 200 gpt-5.5 twins → import → twin-validate) surfaced a cluster of golden-path guidance bugs: zwill next and inline next_steps route users to non-runnable or incomplete commands, and never surface twin-validate — the decisive leakage + conditional-baseline + bootstrap-CI validation the guide insists on. A user following the printed guidance verbatim stops at a bare twin score, exactly the "positive result from a bare twin run" the guide warns against.

Fixes prepared on branch fix/golden-path-next-steps (full suite green, 261 passed).

F2/F3 — zwill next at run_twins suggests a non-runnable command

next_command was:
zwill edsl-export --survey <s> --target twin-probability-job --heldout-questions <q1,q2> --context-question-count 8 --complete-cases --model openai:gpt-5.5 --path twin.edsl.json
which (a) errors with approval_required (it needs an approved twin-experiment plan or --allow-unapproved), and (b) hardcodes --context-question-count 8 even for surveys with fewer questions (pew_w154_diff1 has 5). Fix: route to the runnable, gated plan flow (twin-experiment init-plan → validate → approve → export-plan → run → import).

F7 — inline next_steps never reach twin-validate

  • twin-results import next_steps ended at twin-results report (twin-only).
  • twin-experiment export-plan next_steps went import → twin-experiment compare (twin-only).
    Neither surfaces twin-validate. Fix: lead/insert twin-validate in both. (Note: zwill next itself does route to twin-validate at the validate stage — the bug is that the inline next_steps disagree with it.)

F1 — status re-suggests commit after commit

zwill status always emitted next_steps: ["zwill commit --survey <survey>"] even when every survey was already committed. Fix: suggest commit only for uncommitted surveys; otherwise zwill next.

Minor / not changed

  • F5: twin-experiment compare defaults to a rich table while most commands default to JSON (it does support --format json/csv). Left as-is; flagging for consistency.

Verified NOT bugs (checked and retracted)

  • Output-root relocation of --path/--out is documented + intentional ("contained under an output root … override with ZWILL_OUT"); the command reports the real path.
  • twin-results report / twin-experiment compare "no output" was a rich table (not JSON), not a failure.

Happy to open a PR (branch is ready).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions