feat(testset): add-tests/promotion for inherited generatorScript (#601)#605
Open
rsalesc wants to merge 10 commits into
Open
feat(testset): add-tests/promotion for inherited generatorScript (#601)#605rsalesc wants to merge 10 commits into
rsalesc wants to merge 10 commits into
Conversation
Design for making add-tests and promotion aware of the inherited problem-level generatorScript (#599): a shared effective-script resolver, @testgroup block enumeration/insertion, a run-key-granular add-tests picker, and a promotion safety gate that only allows removals affecting a single run-key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#601) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single source of truth for own-else-inherited generatorScript, plus iter_effective_scripts over every leaf run-key, reused by add-tests and promotion. No behavior change in the visitor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
testgroup_blocks() returns each block's full /-joined path and line span, used to target add-tests insertions and reason about removal scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
append_in_block inserts calls inside an existing @testgroup block; append_new_block appends a fresh scoped block. Both keep an added test confined to a single run-key in a shared script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
script_format_by_path / remove_script_entries now resolve effective (inherited) scripts. is_isolated_removal allows promoting an inherited-script test only when removing its line affects a single run-key -- a @testgroup block scoped to that group, or a script used by only that group. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ted (#601) The promote command (interactive + selectors) now offers/accepts tests generated by an inherited problem-level generatorScript, gating on is_isolated_removal so a removal that would change another test group is rejected with a clear reason. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stress "add found tests" picker now lists, per run-key, each matching @testgroup block (<run-key> @ <script>:<line>) and a create/append target, resolving inherited problem-level scripts. add_calls_to_target inserts into the chosen block, appends top-level when the script is exclusive to the run-key, or creates a scoped @testgroup block otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A problem-level @testgroup-partitioned script inherited by grpa/grpb; promoting grpa/0 into a manual group removes the scoped line from the shared script (grpa drops 2->1) and writes a static test, leaving grpb untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #601.
Makes
rbx stress's "add found tests" picker andrbx testcases promoteaware of the inherited problem-levelgeneratorScript(#599), with@testgroup-scoped edits and a promotion safety gate. Builds on #600/#604 (@testgrouppath filtering).The two scenarios (issue body + maintainer's comment)
1. add-tests — the picker now lists, per run-key (group or subgroup), each matching
@testgroupblock as<run-key> @ <script>:<line>, plus a create/append target. A finding is appended scoped to that run-key: into the chosen block, into a fresh@testgroup <run-key>block, or top-level when the script is exclusive to that run-key. Inherited problem-level scripts are resolved, not just explicit per-group ones.2. promotion — a test generated by an inherited script is promotable iff removing its line affects only its own run-key: the line sits in a
@testgroupblock scoped to that run-key, or the script is used by only that run-key. Otherwise it is rejected with a clear reason. This matches the comment's rule exactly.Design decision
Chose Option A (operate on the shared script,
@testgroup-scoped) over the issue's tentative Option B (materialize per-group copies). Given@testgroupscoping (#600), per-run-key edits are already isolated within the shared file, so materialization would only duplicate other groups' blocks and mutateproblem.rbx.ymlfor no benefit. (Confirmed with @rsalesc.) Design doc:docs/plans/2026-06-21-inherited-generator-script-edits-design.md.Key pieces
testcase_extractors.effective_generator_script/iter_effective_scripts— one own-else-inherited resolver, reused by the visitor, add-tests, and promotion.generator_script_parser.testgroup_blocks— enumerate@testgroupblocks (full path + span).RbxGeneratorScriptHandler.append_in_block/append_new_block— scoped insertion.promotion.is_isolated_removal/removal_affects_only_run_key— the safety gate, using the same_group_matchespredicate as generation, so it cannot drift from the filtering semantics.Tests
@testgroupblock.inherited-script-editsfixture — promoting a@testgroup-scoped test removes its line from the shared script (grpa 2→1) and writes a static test, leaving the sibling group untouched.Touched-module suites all green; the 24 pre-existing failures on this machine (validator/checker/generator C++-compilation tests) reproduce identically on
origin/mainand are unrelated.🤖 Generated with Claude Code