fix: remove --test_tmpdir that conflicts with --sandbox_tmpfs_path#23
Merged
Conversation
The --test_tmpdir=/tmp/bazel-test-logs setting conflicts with --sandbox_tmpfs_path=/tmp because the tmpfs overlay mounts a fresh empty filesystem over /tmp inside the sandbox, causing the bind-mount of TEST_TMPDIR to fail with 'No such file or directory'. This was preventing ALL tests from running — they were killed by the sandbox infrastructure before the test binary could even start. Removing --test_tmpdir lets Bazel use its default TEST_TMPDIR handling, which is sandbox-compatible.
When a previous build/test is interrupted (Ctrl+C, OOM kill, crash), Bazel's sandbox directories aren't cleaned up. The next run fails with 'Could not copy inputs into sandbox: ... (File exists)'. Using --sandbox_base=/dev/shm places sandbox directories on tmpfs, which is auto-cleaned on container restart and eliminates stale state accumulation. This also improves sandbox I/O performance since /dev/shm is memory-backed.
The previous approach (--sandbox_base=/dev/shm) failed because Docker containers default to 64MB /dev/shm, which is too small for linking large C++ binaries (Bus error + No space left on device). Instead, handle stale sandbox state by: 1. Purging .bazel/output_base/sandbox in dcodex-setup.sh before builds (instant, preserves disk cache for fast incremental builds) 2. Documenting in .bazelrc why --sandbox_base=/dev/shm must NOT be used This combined with the --test_tmpdir removal (previous commit) fixes all known sandbox infrastructure failures.
Now that sandbox issues are resolved, the per-action debug traces (thousands of lines per build) are no longer needed and drown the actual test output. Can still be passed manually when needed: bazel test --sandbox_debug ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The --test_tmpdir=/tmp/bazel-test-logs setting conflicts with --sandbox_tmpfs_path=/tmp because the tmpfs overlay mounts a fresh empty filesystem over /tmp inside the sandbox, causing the bind-mount of TEST_TMPDIR to fail with 'No such file or directory'.
This was preventing ALL tests from running — they were killed by the sandbox infrastructure before the test binary could even start.
Removing --test_tmpdir lets Bazel use its default TEST_TMPDIR handling, which is sandbox-compatible.