Skip to content

Skip subproject tests during PRB if the code is unaffected#4328

Open
alecgrieser wants to merge 7 commits into
FoundationDB:mainfrom
alecgrieser:selective-prbs
Open

Skip subproject tests during PRB if the code is unaffected#4328
alecgrieser wants to merge 7 commits into
FoundationDB:mainfrom
alecgrieser:selective-prbs

Conversation

@alecgrieser

@alecgrieser alecgrieser commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

This adds an additional step to our PRB config. It looks at the change set, and from that, it computes a list of subprojects that have been directly affected. It combines that with a list of dependent subprojects to produce a full set of tests that need to be run. If a subproject is not affected by a change (directly or indirectly), then it gets skipped.

This has a few safe-guards:

  1. If a build file (as determined by the file being in the build or gradle directory or one of the top-level gradle files) is modified, then all of the tests get run
  2. If we have errors parsing anything or determinging the changes, then we fallback to running all tests
  3. This isn't perfect, but teamscale enforces that we have coverage of a function, so if there's a bug that somehow results in modified code being skipped, we should still get teamscale warnings about it
  4. We continue to run all of the tests in the nightly and release jobs, so if this skips an affected module, we will still know within a day or a release, whichever is sooner
  5. The plan job logs which subprojects it is running, and the coverage job notes if any subprojects were skipped

The benefit, though, is that we can skip a lot of work for certain kinds of changes. For example, fdb-extensions has grown quite a bit with recent vector work, and this protects other PRs from needing to run that test suite. Documentation only changes can avoid testing all together (unless and until we add a documentation build, which may be a good idea), and changes to subprojects like fdb-record-layer-lucene or yaml-tests which are not dependend on by anything only need to run their own tests.

Note that the list of subprojects and their downstream dependencies is now computed in printDependentSubprojects. This means that if a gradle change results in the dependency graph changing, those changes will be reflected automatically in picking better tests.

To show how this works, I've created a few test PRs against my fork, with PRBs running only selective builds:

  1. Modification to readme to test selective PRB alecgrieser/fdb-record-layer#4: Modifies just the README. PRB run https://github.com/alecgrieser/fdb-record-layer/actions/runs/29259820923?pr=4 validates that no tests are run (though style currently is)
  2. Modify a file in fdb-record-layer-core alecgrieser/fdb-record-layer#5: Modifies a file in fdb-record-layer-core. PRB run https://github.com/alecgrieser/fdb-record-layer/actions/runs/29259845608?pr=5 validates that the fdb-extensions tests are skipped
  3. Modify a file in fdb-record-layer-lucene alecgrieser/fdb-record-layer#6: Modifies a file in fdb-record-layer-lucene. PRB run https://github.com/alecgrieser/fdb-record-layer/actions/runs/29259868619?pr=6 validates that only the fdb-record-layer-lucene tests are run in this case

Those are the end-to-end tests. There are also unit tests of the various scenarios in test_affected_subprojects.py, which was run during the development process.

This adds an additional step to our PRB code. It looks at the change set, and from that, it computes a list of subprojects that have been directly affected. It combines that with a list of dependent subprojects to produce a full set of tests that need to be run. If a subproject is not affected by a change (diretly or indirectly), then it gets skipped.

This has a few safe-guards:

1. If a build file (as determined by the file being in the `build` or `gradle` directory or one of the top-level `gradle` files), then all of the tests get run
1. If we have errors parsing anything or determinging the changes, then we fallback to running all tests
1. This isn't perfect, but teamscale enforces that we have coverage of a function, so if there's a bug that somehow results in modified code being skipped, we should still get teamscale warnings about it
1. We continue to run all of the tests in the nightly and release jobs, so if this skips an affected module, we will still know within a day or a release, whichever is sooner

The benefit, though, is that we can skip a lot of work for certain kinds of changes. For example, `fdb-extensions` has grown quite a bit with recent vector work, and this protects other PRs from needing to run that test suite. Documentation only changes can avoid testing all together (unless and until we add a documentation build, which may be a good idea), and changes to subprojects like `fdb-record-layer-lucene` or `yaml-tests` which are not dependend on by anything only need to run their own tests.

Note that the list of subprojects and their downstream dependencies is now computed in `printDependentSubprojects`. This means that if a gradle change results in the dependency graph changing, those changes will be reflected automatically in picking better tests.
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 8, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
1. We don't have all of the subproject jars if we run only a subset of them available during `coverage`
1. The report xml might be entirely missing if all of the tests are skipped
1. The step summary was missing some information due to how bash escaping works
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser added a commit to alecgrieser/fdb-record-layer that referenced this pull request Jul 13, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
@alecgrieser alecgrieser marked this pull request as ready for review July 13, 2026 15:24
@alecgrieser alecgrieser requested a review from ScottDugas July 13, 2026 15:28
@alecgrieser

Copy link
Copy Markdown
Collaborator Author

A couple of additional thoughts: this currently treats main and test code the same, but I actually think that a test only change should only need to run the tests for that project (now that we've switched to testFixtures). So, this could theoretically do a better job by taking that into account, but I didn't try that.

Also, once we get this in, we probably need to lift the other-tests requirement. Requiring that coverage passes will still ensure that all tests that were supposed to actually ran

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build improvement Improvement to the build system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant