Skip subproject tests during PRB if the code is unaffected#4328
Open
alecgrieser wants to merge 7 commits into
Open
Skip subproject tests during PRB if the code is unaffected#4328alecgrieser wants to merge 7 commits into
alecgrieser wants to merge 7 commits into
Conversation
This was referenced Jul 8, 2026
This adds an additional step to our PRB code. It looks at the change set, and from that, it computes a list of subprojects that have been directly affected. It combines that with a list of dependent subprojects to produce a full set of tests that need to be run. If a subproject is not affected by a change (diretly or indirectly), then it gets skipped. This has a few safe-guards: 1. If a build file (as determined by the file being in the `build` or `gradle` directory or one of the top-level `gradle` files), then all of the tests get run 1. If we have errors parsing anything or determinging the changes, then we fallback to running all tests 1. This isn't perfect, but teamscale enforces that we have coverage of a function, so if there's a bug that somehow results in modified code being skipped, we should still get teamscale warnings about it 1. We continue to run all of the tests in the nightly and release jobs, so if this skips an affected module, we will still know within a day or a release, whichever is sooner The benefit, though, is that we can skip a lot of work for certain kinds of changes. For example, `fdb-extensions` has grown quite a bit with recent vector work, and this protects other PRs from needing to run that test suite. Documentation only changes can avoid testing all together (unless and until we add a documentation build, which may be a good idea), and changes to subprojects like `fdb-record-layer-lucene` or `yaml-tests` which are not dependend on by anything only need to run their own tests. Note that the list of subprojects and their downstream dependencies is now computed in `printDependentSubprojects`. This means that if a gradle change results in the dependency graph changing, those changes will be reflected automatically in picking better tests.
d46f72d to
5ffa34e
Compare
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 8, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
1. We don't have all of the subproject jars if we run only a subset of them available during `coverage` 1. The report xml might be entirely missing if all of the tests are skipped 1. The step summary was missing some information due to how bash escaping works
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
07f38b5 to
fa8e219
Compare
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 9, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Jul 13, 2026
Change to `fdb-record-layer-core` to test: FoundationDB#4328 This should skip `fdb-extensions`, but it should run essentially all of the other tests.
Collaborator
Author
|
A couple of additional thoughts: this currently treats main and test code the same, but I actually think that a test only change should only need to run the tests for that project (now that we've switched to Also, once we get this in, we probably need to lift the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds an additional step to our PRB config. It looks at the change set, and from that, it computes a list of subprojects that have been directly affected. It combines that with a list of dependent subprojects to produce a full set of tests that need to be run. If a subproject is not affected by a change (directly or indirectly), then it gets skipped.
This has a few safe-guards:
buildorgradledirectory or one of the top-levelgradlefiles) is modified, then all of the tests get runThe benefit, though, is that we can skip a lot of work for certain kinds of changes. For example,
fdb-extensionshas grown quite a bit with recent vector work, and this protects other PRs from needing to run that test suite. Documentation only changes can avoid testing all together (unless and until we add a documentation build, which may be a good idea), and changes to subprojects likefdb-record-layer-luceneoryaml-testswhich are not dependend on by anything only need to run their own tests.Note that the list of subprojects and their downstream dependencies is now computed in
printDependentSubprojects. This means that if a gradle change results in the dependency graph changing, those changes will be reflected automatically in picking better tests.To show how this works, I've created a few test PRs against my fork, with PRBs running only selective builds:
stylecurrently is)fdb-record-layer-corealecgrieser/fdb-record-layer#5: Modifies a file infdb-record-layer-core. PRB run https://github.com/alecgrieser/fdb-record-layer/actions/runs/29259845608?pr=5 validates that thefdb-extensionstests are skippedfdb-record-layer-lucenealecgrieser/fdb-record-layer#6: Modifies a file infdb-record-layer-lucene. PRB run https://github.com/alecgrieser/fdb-record-layer/actions/runs/29259868619?pr=6 validates that only thefdb-record-layer-lucenetests are run in this caseThose are the end-to-end tests. There are also unit tests of the various scenarios in
test_affected_subprojects.py, which was run during the development process.