test: testing infrastructure overhaul — shared framework + dogfood scanner#397
Merged
test: testing infrastructure overhaul — shared framework + dogfood scanner#397
Conversation
Option C from testing research — improve existing framework with assertion helpers, mock registry, subshell isolation, and behavioral test conversions across 3 parallel batches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enhance test-framework.zsh with mock registry (create_mock, assert_mock_called, assert_mock_args, reset_mocks), subshell isolation (run_isolated), and convenience aliases - Convert test-work.zsh from existence-only checks to behavioral assertions (24 tests now verify exit codes, output, and mocks) - Convert test-dispatcher-enhancements.zsh to use shared framework with assert_contains instead of inline grep checks - Add dogfood-test-quality.zsh scanner that detects 4 anti-patterns: permissive exit codes, existence-only tests, unused output captures, and inline frameworks Full suite: 45/45 passing, no regressions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace inline test frameworks with shared test-framework.zsh. Convert existence-only checks to behavioral assertions, fix permissive exit codes, and eliminate unused output captures. Dogfood scanner delta: existence-only 94→65, unused captures 51→31, permissive exit codes 14→11, inline frameworks 115→103. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three-layer testing taxonomy (unit/E2E/dogfood), framework comparison (ShellSpec vs BATS vs ZTAP vs native), and anti-pattern catalog. Complements ORCHESTRATE file with strategic direction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert 4 more test files to use test-framework.zsh with proper behavioral assertions. Add test_skip() and TESTS_SKIPPED counter to framework. Anti-pattern progress: permissive 11→2, existence 65→45, unused captures 31→21, inline frameworks 103→91. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert 4 high-anti-pattern test files to use test-framework.zsh: - test-atlas-e2e.zsh: 19/19 passing (was inline framework + 11 existence-only) - test-token-automation-unit.zsh: 25/25 passing (fixed jq newline capture) - test-token-automation.zsh: 18/18+2skip+1skip (doc files skip in worktree) - test-doctor-cache.zsh: 19/20 passing (fixed readonly scoping bug, 10→19) Anti-pattern progress: existence 45→11, inline 91→80, permissive 2→1 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test-work.zsh: replace permissive exit code (0||1) with strict assert - test-adhd.zsh: add output assertions to 6 help/flag tests - test-capture.zsh: add output assertions to 6 unused captures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add output assertions to 3 timer tests that captured output but only checked exit codes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ework.zsh Eliminate inline pass/fail/log_test functions across 30 test files, replacing them with shared test-framework.zsh assertions. This reduces the dogfood scanner's inline framework violations from 80 in 30 files to 2 in 1 file (97.5% reduction). Files converted: test-ai-features, test-atlas-integration, test-cc-*, test-course-planning-*, test-doctor-email-*, test-em-dispatcher, test-flow, test-g-feature-*, test-keychain-default-automated, test-mcp-dispatcher, test-obs-dispatcher, test-pick-*, test-production-*, test-qu-dispatcher, test-quick-wins, test-r-dispatcher, test-status-fields, test-sync, test-tm-dispatcher, test-token-automation-e2e, test-wt-*. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All acceptance criteria met: - 14 assertion helpers + 5 mock registry functions in test-framework.zsh - 30+ test files converted from inline to shared framework - Dogfood scanner: permissive exit codes eliminated (0), inline frameworks 80→2 - Full suite: 45/45 passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert test-course-planning-docs-unit.zsh and test-doctor-email-interactive.zsh from inline pass/fail/log_test to shared test-framework.zsh. Reduces dogfood inline framework violations to 0. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ptures) - Add behavioral follow-ups to existence-only tests in 6 files - Fix unused output captures in 4 files - Dogfood scanner now passes 4/4 categories (all clean) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `trap cleanup EXIT` to 31 test files that create temp dirs (prevents /tmp leaks when tests fail early) - Standardize on `test_suite_end` instead of mixed `print_summary` (10 files updated for consistency) - Update ORCHESTRATE results table to match final scanner output Test suite: 45/45 passing, dogfood scanner: 4/4 clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Guard test_pass against double-counting when test_fail already fired - Remove unused mock_function/restore_function (superseded by mock registry) - Move ORCHESTRATE files from repo root to docs/specs/ - Add scalar-only limitation note to with_env helper - Deduplicate .STATUS wins entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…framework Resolves merge conflicts from work -e editor flag (PR #396) added on dev after the testing overhaul branched. Converts 40+ new editor/help/arg-parser tests from inline framework to shared test-framework.zsh assertions. 71/71 test-work.zsh passing, dogfood 4/4 clean, full suite 45/45. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-framework.zshwith 14 assertion helpers, mock registry (5 functions), and subshell isolationdogfood-test-quality.zshscanner that catches 4 anti-pattern categoriesAnti-pattern elimination
0 || 1)Key changes
tests/test-framework.zsh— assertion helpers, mock registry, subshell isolationtests/dogfood-test-quality.zsh— meta-test scanner for anti-patternsdocs/specs/SPEC-testing-framework-2026-02-16.md— testing strategy specORCHESTRATE-testing-overhaul.md— implementation planTest plan
source flow.plugin.zshexit 0)🤖 Generated with Claude Code