Load all DefPathHash -> DefId mappings before dumping dep graph

[Aaron1011]

Fixes #79723

[rust-highfive]

r? @petrochenkov

(rust-highfive has picked a reviewer for you, use r? to override)

[rust-log-analyzer]

The job x86_64-gnu-llvm-9 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

.................................................................................................... 9000/11238
.................................................................................................... 9100/11238
.................................................................................................... 9200/11238
..................................i.......i......................................................... 9300/11238
.........................................................................iiiiii..iiiiii.i........... 9400/11238
.................................................................................................... 9600/11238
.................................................................................................... 9700/11238
.................................................................................................... 9800/11238
.................................................................................................... 9900/11238
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 27 tests
iiiiiiiiiiiiiiiiiiiiiiiiiii

 finished in 0.061 seconds
Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
 finished in 9.829 seconds
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 116 tests
iiiiiiiiii.i.i..i..i..ii.....ii....ii..........i.iii........i.....i...i.......ii.i.ii.....iiii.....i 100/116
test result: ok. 78 passed; 0 failed; 38 ignored; 0 measured; 0 filtered out; finished in 2.27s

 finished in 2.340 seconds
Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
---
Suite("src/test/run-make") not skipped for "bootstrap::test::RunMake" -- not in ["src/tools/tidy"]
Check compiletest suite=run-make mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)

running 20 tests
Some tests failed in compiletest suite=run-make mode=run-make host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu
iiiiiiiiiiii.F......


---- [run-make] run-make/dump-dep-graph stdout ----
error: make failed
status: exit code: 2
command: "make"
stdout:
stdout:
------------------------------------------
cp main.rs /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/dump-dep-graph/dump-dep-graph/
cd /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/dump-dep-graph/dump-dep-graph
LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/dump-dep-graph/dump-dep-graph:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/dump-dep-graph/dump-dep-graph -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/dump-dep-graph/dump-dep-graph  -Z dump-dep-graph main.rs
Makefile:7: recipe for target 'all' failed
------------------------------------------
stderr:
------------------------------------------
------------------------------------------
warning: function is never used: `my_function`
 --> main.rs:1:4
1 | fn my_function() {}
  |    ^^^^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default
  = note: `#[warn(dead_code)]` on by default

thread 'rustc' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 30, kind: Other, message: "Read-only file system" }', compiler/rustc_incremental/src/assert_dep_graph.rs:239:63

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.
note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.51.0-nightly (8f785ce58 2021-01-02) running on x86_64-unknown-linux-gnu

note: compiler flags: -Z dump-dep-graph


query stack during panic:
end of query stack
warning: 1 warning emitted

make: *** [all] Error 101
------------------------------------------




failures:
    [run-make] run-make/dump-dep-graph

test result: FAILED. 7 passed; 1 failed; 12 ignored; 0 measured; 0 filtered out; finished in 0.49s



command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-make" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--suite" "run-make" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-9/bin/FileCheck" "--nodejs" "/usr/bin/node" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "9.0.0" "--llvm-components" "aarch64 aarch64asmparser aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets amdgpu amdgpuasmparser amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitstreamreader bitwriter bpf bpfasmparser bpfcodegen bpfdesc bpfdisassembler bpfinfo codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfogsym debuginfomsf debuginfopdb demangle dlltooldriver engine executionengine fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader jitlink lanai lanaiasmparser lanaicodegen lanaidesc lanaidisassembler lanaiinfo libdriver lineeditor linker lto mc mca mcdisassembler mcjit mcparser mips mipsasmparser mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmparser msp430codegen msp430desc msp430disassembler msp430info native nativecodegen nvptx nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcjit passes perfjitevents powerpc powerpcasmparser powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata remarks riscv riscvasmparser riscvcodegen riscvdesc riscvdisassembler riscvinfo riscvutils runtimedyld scalaropts selectiondag sparc sparcasmparser sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target textapi transformutils vectorize webassembly webassemblyasmparser webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info x86utils xcore xcorecodegen xcoredesc xcoredisassembler xcoreinfo xray" "--system-llvm" "--llvm-bin-dir" "/usr/lib/llvm-9/bin" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--ar" "ar" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"


failed to run: /checkout/obj/build/bootstrap/debug/bootstrap --stage 2 test --exclude src/tools/tidy
Build completed unsuccessfully in 0:31:28

[petrochenkov]

r? @michaelwoerister @tgnottingham
@bors delegate=tgnottingham

[bors]

✌️ @tgnottingham can now approve this pull request

[tgnottingham]

The DPH mapping functionality isn't inherently associated with OnDiskCache, and using OnDiskCache in non-incremental mode is kinda funny. So, you could have debug_def_path_hash_to_def_id live in GlobalCtxt, and have a method that uses it during non-incremental, and uses OnDiskCache::def_path_hash_to_def_id during incremental. That also has the nice property that OnDiskCache uses the same code path and structures to map DPHs regardless of whether or not we're dumping the dep graph.

Only issue is that OnDiskCache::def_path_hash_to_def_id doesn't currently work for foreign DefPathHashes that didn't exist in previous session, but do in this session, so that would need to be changed. But that might be nice anyway, as I can easily see someone not expecting it.

Also, before #74967, the equivalent of debug_def_path_hash_to_def_id was populated when opts.build_dep_graph() was true, i.e. when either -Z dump_dep_graph or -Z query_dep_graph was specified, but now it's only built for dump_dep_graph.

[bors]

☔ The latest upstream changes (presumably #80610) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

I hope I'll get to reviewing this next week. Thanks for the PR, @Aaron1011!

[michaelwoerister]

Thanks for trying to fix this, @Aaron1011!

I've been taking a look at the implementation and I think I understand how it works -- but I'm a bit hesitant to make things more complex for a feature like -Zdump-dep-graph which I suspect is rarely ever used. Like @tgnottingham, I'm also a bit worried about instantiating an OnDiskCache in non-incremental mode because it's rather unexpected and might lead to confusion and the need for special casing down the line.

Am I correct in the assumption that the output of -Zdump-dep-graph works fine (even without this PR) if compiling in incremental mode? If so I would suggest just emitting a warning if -Zdump-dep-graph is passed to the compiler without -Cincremental being passed too.

These dependency graph debugging options are all really old. We could probably simplify things if we just assumed that the dep-graph will only ever be built in incremental mode.

[JohnCSimon]

Ping from triage
@Aaron1011 - can you please address the build failures, merge conflicts, and comments from the reviewer? Thank you.

@rustbot label: -S-waiting-on-review +S-waiting-on-author

[Dylan-DPC-zz]

Closing this due to inactivity.