Skip to content

Add guards for stale help examples and JSON output purity#197

Merged
alexkroman merged 1 commit into
mainfrom
claude/blissful-clarke-18vei0
Jun 16, 2026
Merged

Add guards for stale help examples and JSON output purity#197
alexkroman merged 1 commit into
mainfrom
claude/blissful-clarke-18vei0

Conversation

@alexkroman

Copy link
Copy Markdown
Collaborator

Summary

Adds two new test suites that enforce repo-wide invariants across all leaf commands:

  1. Help example freshness — validates that --help examples reference only flags and subcommands that actually exist, catching silent rot from renames (e.g., --api-key--with-api-key) or removed subcommands.
  2. JSON output purity — verifies the "Errors → stderr, data → stdout" contract holds for every leaf command, ensuring pipeline safety when chaining assembly … --json | next-tool.

Key Changes

  • tests/test_help_example_flags.py — New test module that:

    • Parses the Examples epilog from every leaf command's --help output
    • Extracts $ assembly … command lines and splits pipelines into per-assembly segments
    • Walks the live Typer tree to validate that each flag and subcommand path exists
    • Reports stale examples with the exact unknown flags they reference
    • Includes unit tests proving the detection logic catches renamed/removed flags and correctly handles shell syntax (pipes, redirects, command substitution)
  • tests/test_json_stdout_purity.py — New test module that:

    • Parametrizes over every leaf command path
    • Invokes each with an unknown flag (deterministic parse error before command body)
    • Asserts exit_code == 2 (standard Click usage error)
    • Verifies stdout == "" (no data leaked during error)
    • Validates that stderr contains only valid JSON (no human-prose leaks)
    • Tests both --json mode (machine-readable error envelope) and human mode (error on stderr, not stdout)

Implementation Details

  • Conservative parsing — The help-example validator skips segments where the assembly token is glued to shell syntax (e.g., $(assembly …) rather than mis-parsing, since those same commands appear unglued elsewhere in the examples.
  • Uniform error path — JSON purity uses an unknown flag as the trigger because it fails during Click's argument parsing, before credential resolution or network calls, making it deterministic and uniform across all commands.
  • Parametrized coverage — Both suites use leaf_command_items() and leaf_command_argvs() helpers (from tests/_cli_tree) to automatically cover every leaf command without manual enumeration.

These guards close gaps left by existing checks: test_help_examples_coverage proves examples exist but not that they're valid; docs_consistency_gate.py keeps REFERENCE.md/README in sync but doesn't check in---help examples.

https://claude.ai/code/session_01A8VXBitwFzqz5VsLCyepiY

…commands

Add two overarching meta-tests that close gaps the per-command suite left:

- test_json_stdout_purity: walks the live Typer tree and asserts every leaf
  command keeps the "errors → stderr, data → stdout" contract under --json.
  Triggered by an unknown flag (a uniform, pre-network parse error, safe even
  for interactive commands), it pins stdout empty + a pure-JSON error envelope
  on stderr, plus the human-mode split. Nothing previously swept this.

- test_help_example_flags: parses each command's --help "Examples" epilog and
  validates every flag/subcommand against the live tree, catching stale copy
  after a rename (e.g. login --api-key → --with-api-key). docs_consistency_gate
  covers REFERENCE/README refs but not the in-help examples users copy-paste.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A8VXBitwFzqz5VsLCyepiY
@alexkroman alexkroman enabled auto-merge June 16, 2026 22:40
@alexkroman alexkroman added this pull request to the merge queue Jun 16, 2026
Merged via the queue into main with commit 410b650 Jun 16, 2026
19 checks passed
@alexkroman alexkroman deleted the claude/blissful-clarke-18vei0 branch June 16, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants