Read incremental files on-demand

[cjgillot]

This should avoid a short-lived memory spike at the beginning of the compilation.

[rust-highfive]

r? @davidtwco

(rust-highfive has picked a reviewer for you, use r? to override)

[jyn514]

cc @tgnottingham

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 3c25ddf with merge 833254a567948c75829d14b281da174002a6cfd5...

[bors]

☀️ Try build successful - checks-actions
Build commit: 833254a567948c75829d14b281da174002a6cfd5 (833254a567948c75829d14b281da174002a6cfd5)

[rust-timer]

Queued 833254a567948c75829d14b281da174002a6cfd5 with parent b3e19a2, future comparison URL.

[rust-timer]

Finished benchmarking try commit (833254a567948c75829d14b281da174002a6cfd5): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 296ca4c with merge 4937463fe51189b570f25fbd1d948a163a1e89c9...

[bors]

☀️ Try build successful - checks-actions
Build commit: 4937463fe51189b570f25fbd1d948a163a1e89c9 (4937463fe51189b570f25fbd1d948a163a1e89c9)

[rust-timer]

Queued 4937463fe51189b570f25fbd1d948a163a1e89c9 with parent f293f70, future comparison URL.

[rust-timer]

Finished benchmarking try commit (4937463fe51189b570f25fbd1d948a163a1e89c9): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[joshtriplett]

This does substantially reduce memory usage for incremental builds, but increases compile times.

This doesn't seem like it inherently needs to be slower. I'm wondering if using the ReadBuf-based interfaces might help here, to avoid repeatedly having to initialize buffers and use resize.

[joshtriplett]

I'm also wondering if it'd be possible to use BufRead for this, which does already have some of these optimizations (including seek_relative).

[bjorn3]

Could mmap be used here? Maybe with madvise(MADV_SEQUENTIAL) though after #82183 that would be better avoided.

[cjgillot]

I would rather avoid using mmap, because it's incompatible with #82780, and because it would be way too unsafe for my taste.
I tried to avoid touching the query result cache in anticipation for #82183.

[bjorn3]

I switched the metadata reading in cg_clif from fs::read to mmap yesterday. (bjorn3/rustc_codegen_cranelift@b1d14ca) The performance improvements for at least incr-clean were significant. Also constantly seeking is relatively expensive as it needs a syscall every time. A final benefit of mmap is that it allows the kernel to discard the data again when there is little memory left.

[bjorn3]

Also MAP_PRIVATE as used by memmap2's map_copy_read_only is a bit safer than MAP_SHARED. In addition mmap is used for the metadata reading by cg_llvm anyway.

[cjgillot]

Closing in favour of #83214