refactor: replace deprecated local_resources flag with local_cpu_reso…#21
Merged
Conversation
…urces and local_ram_resources in CI workflows
Add a deliberately buggy BuggyMetricsCollector that mirrors DCodeX's
real DynamicWorkerCoordinator::Metrics pattern. It contains intentional
data races (unprotected read-modify-write on shared counters and a
TOCTOU race in GetCacheHitRate) that are invisible to normal testing
but caught deterministically by ThreadSanitizer.
The test is tagged 'manual' so it never runs in //..., and a new CI
step ('TSan Proof-of-Detection') runs it under --config=tsan and
EXPECTS failure. If TSan fails to detect the race, the CI itself
fails — proving the sanitizer pipeline is broken.
Verified locally:
- Without TSan: PASSED (race is invisible)
- With TSan: FAILED with 'ThreadSanitizer: data race' (race caught)
* Refactor the TSan proof step in CI to avoid bash pipefail conflicts when capturing Bazel exit codes. * Read the test log directly from bazel-testlogs to reliably detect the ThreadSanitizer data race footprint. * Update ci.bazelrc to use the non-deprecated local_resources=cpu=N syntax for Bazel 7.x.
sandbox_test forks clang++ and compiled binaries as child processes. Under TSan (8x memory overhead), these forked processes exhaust the 7GB CI runner memory, causing 'out of memory: failed to allocate TracePart' errors and spurious exit code 66 failures. Tagged sandbox_test with 'no-tsan' and added --test_tag_filters=-no-tsan to the CI TSan step. sandbox_test still runs under Standard, ASan, and MSan configurations.
Instead of excluding sandbox_test from TSan entirely, split the TSan
step into two:
1. Main TSan step: runs all tests except sandbox_test (20 iterations)
2. Constrained TSan step: runs sandbox_test with --runs_per_test=1
and --local_test_jobs=1 to prevent OOM from forked clang++
Also reduce CI log noise:
- test_output=errors (only print output for failing tests)
- noannounce_rc (suppress the full bazelrc option dump)
- show_progress_rate_limit=5 (throttle progress updates)
- curses=no (disable terminal control sequences)
The test log symlink lives at .bazel/testlogs/ (due to --symlink_prefix=.bazel/) not bazel-testlogs/. Instead of guessing the symlink path, capture bazel's stdout+stderr to /tmp/tsan_proof_output.log and grep that directly.
MSan fundamentally requires ALL linked libraries to be compiled with MSan instrumentation. The system libstdc++ on GitHub Actions runners is not instrumented, causing every test to fail in googletest's static initialization (testing::Message::Message) before any DCodeX code even executes. TSan and ASan remain — they work correctly with non-instrumented system libraries. The MSan config is preserved in .bazelrc for local use with a custom toolchain.
Sam-Si
force-pushed
the
feat/final-ci-hardening
branch
from
f656e91 to
c344ec8
Compare
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…urces and local_ram_resources in CI workflows