Skip to content

rust: add incremental builds and persistent worker pipelining#11

Closed
walter-zeromatter wants to merge 2 commits intohermeticbuild:mainfrom
walter-zeromatter:user/wgray/persistent-worker
Closed

rust: add incremental builds and persistent worker pipelining#11
walter-zeromatter wants to merge 2 commits intohermeticbuild:mainfrom
walter-zeromatter:user/wgray/persistent-worker

Conversation

@walter-zeromatter
Copy link
Copy Markdown

@walter-zeromatter walter-zeromatter commented Apr 4, 2026

Summary

  • Persistent worker pipelining: Adds a multiplex persistent worker mode to process_wrapper that enables Cargo-style pipelining — downstream crates can start compiling as soon as .rmeta metadata is emitted, without waiting for full codegen. This significantly reduces critical path build times. See DESIGN.md for details on how this works.
  • Incremental compilation: Adds optional incremental compilation support, allowing rustc to reuse prior compilation artifacts for faster rebuilds at the cost of higher up-front build cost

Key changes

  • rust/private/rustc.bzl: Wires up worker pipelining flags, metadata actions, and incremental build support
  • rust/private/incremental.bzl: New file for incremental compilation logic
  • rust/settings/settings.bzl: New settings (experimental_worker_pipelining, experimental_incremental)
  • util/process_wrapper/worker*.rs: Full multiplex worker implementation with pipelining, sandboxing, and cancellation support
  • test/unit/pipelined_compilation/: New tests for worker pipelining and strace-based sandbox compliance verification

Performance

On a 5-crate benchmark (Bazel 8.4.2, no multiplex sandboxing):

  • No pipelining: 20.7s (143 actions)
  • Hollow rlib (existing): 11.4s (251 actions)
  • Worker pipelining: 8.4s (115 actions) — 1.62× faster than no-pipeline, comparable to Cargo (8.2s)

Test plan

  • 10/10 pipelined_compilation tests pass
  • 51 process_wrapper tests pass

@walter-zeromatter walter-zeromatter force-pushed the user/wgray/persistent-worker branch 2 times, most recently from cabbe75 to 553565d Compare April 8, 2026 00:48
Introduce fast metadata vs full metadata terminology (Cargo-style vs
Buck2-style) across DESIGN.md, settings.bzl, and test comments.
Recommend hollow_rlib as the portable pipelining mode for sandboxed,
remote, and dynamic execution. Expand nondeterministic proc macro
regression test to cover all four failure-boundary scenarios (worker,
no-pipeline, hollow_rlib, worker+sandboxed). Add artifact hash
instrumentation script for SVH debugging.
@walter-zeromatter walter-zeromatter force-pushed the user/wgray/persistent-worker branch from 553565d to 0ec51fb Compare April 9, 2026 01:07
@walter-zeromatter
Copy link
Copy Markdown
Author

This whole effort seems to have been based on benchmarks identifying a problem which was effectively an AI hallucination I didn't validate hard enough. There's no measured performance benefit for workers vs actual pipelining with -Zno-codegen.

Hypothetically this could enable pipelined compilation for crates with non-deterministic compilation, however it would never be possible for that to be stable since the differing execution strategies would produce differing results, violating the bazel contract

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant