fix(scenarios): holistic-review follow-ups — tighten assertions + docs + helpers#8369
Merged
HydraOps-T-rav merged 18 commits intomainfrom Apr 20, 2026
Merged
fix(scenarios): holistic-review follow-ups — tighten assertions + docs + helpers#8369HydraOps-T-rav merged 18 commits intomainfrom
HydraOps-T-rav merged 18 commits intomainfrom
Conversation
…st PR numbering drift)
…t_for_ci signature; A20/L18 assertion corrections
…iles Replaces the fabricated A1–A23 and L9–L23 descriptions with entries derived directly from test names and docstrings in test_agent_realistic.py, test_bead_workflow.py, test_caretaker_loops.py, and test_caretaker_loops_part2.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the stale 'FakeGitHub.add_alerts(pr_number=...)' reference with the actual API: add_alerts(branch=...) keyed by branch string, matching PRPort.fetch_code_scanning_alerts(branch: str). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'or outcome.worker_result is None' branch was always true when workspace creation raised (worker never ran → None), making the entire compound assertion vacuous. Production wraps PermissionError in a WorkerResult(success=False) via run_with_fatal_guard, so the real-signal arm already fires. Dropping the tautological arm means the test fails loudly if the signal disappears. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up PR addressing the deep-review findings across Tier 1/2/3 scenario work (PRs #8356, #8360, #8361). Tightens weak assertions, aligns fakes with real ports, consolidates shared helpers, extends coverage, and documents conventions.
Zero production-code changes. 15 commits total (13 test-side + 2 docs).
Must-fix items (7)
FakeGitHub.add_alertssignature aligned with port — waspr_number: int, nowbranch: strmatchingPRPort.fetch_code_scanning_alerts(branch). A19 updated. (commit384d0619)init_test_worktreeacceptsorigin=...— A14's 30-line inline workaround collapsed to 3 helper calls. Multi-worktree-under-same-parent now supported. (commit9167f155)importorskipfrom fuzz bead DAG test —fake_beadsis in main now; skip was masking future deletions. (commit284f3faa)_pr_counterstarting value; would silently pass ifwait_for_cimissed the PR. Follow-up also fixedmax_ci_fix_attemptsconfig to actually exercise the fix_ci path. (commits6a6ca5f3,4b40266b)assert not merged(tautology for empty pipeline); now checks WorkerResult failure or escalation state. (commite19079cf)assert result is not None; now verifies StateTracker loaded and pipeline progressed. (commitddc93d88)680e932f)Should-fix items (2)
_seed_portshelper — was duplicated with divergent signatures acrosstest_caretaker_loops.pyandtest_caretaker_loops_part2.py. Now lives attests/scenarios/helpers/loop_port_seeding.pywith a single kwargs API. (commit9db403b6)FakeLLM.script_fix_ciAPI —fix_ciwas hardcodedfixes_made=True. New scripting API lets scenarios exercise the "fix_ci gives up" branch. (commit694ef7f7)Nice-fix items (3)
13843174)permissionwas scenario-tested. (commit799a0edb)vitest.config.mjscomment — explains why e2e specs are excluded. (commit57b76ee7)Documentation
docs/scenarios/README.mdupdated with Tier 1/2/3 conventions:init_test_worktree,seed_ports,use_real_agent_runner,wiki_store/beads_manager, fault modes, Pattern A vs. Pattern B for loops, and extended scenario catalog covering A10–A22, B1, L9–L23. (commit832bfaaa)Follow-up issues filed
Follow-up corrections during the batch
FakeGitHub.wait_for_cisignature — updated to accept positional*_argsmatching production's call withci_check_timeout,ci_poll_interval,stop_eventpassed positionally.RepoWikiLoopstats now includequeue_drainedkey; test updated to assert required keys rather than full dict equality.init_test_worktree(origin=...)tests.Test plan
tests/scenarios/scenario + scenario_loops: 195 passed, 1 xfailed (pre-existing)tests/scenarios/fuzz/: 7 passedsrc/ui/e2e/interactions.spec.js: 4 passed (Playwright)pyright tests/scenarios/: 0 errorsmake quality: 10939 passed + 1 pre-existing failure tracked in test: test_build_prompt_truncates_long_body fails on main (plugin skill registry bloat) #8364🤖 Generated with Claude Code