Handle autodiff for lib builds

[ZuseZ4]

Work in Progress.

Tracking:

• Tracking Issue for autodiff #124509

r? @oli-obk

[ZuseZ4]

The third (iffy) commit let's me compile and run the thin-lto pipeline for dependencies, but it runs into the following assertion:

thread 'coordinator' panicked at compiler/rustc_codegen_llvm/src/back/lto.rs:224:9:
internal error: entered unreachable code: We should never reach this case if the LTO step is deferred to the linker

Removing the assertion runs into the next one:

thread 'coordinator' panicked at /home/manuel/prog/rust-middle/compiler/rustc_codegen_ssa/src/back/write.rs:1521:29:
assertion failed: running_with_own_token > 0

That sounds like an assertion which I probably shouldn't touch.

I would have preferred to just support thin-lto instead of having to alter the compilation pipeline even more and move all other workers to fat-lto, since I was worried that would break other assumptions. But it might be easier, I'll look more into it.

The enforcing of fat-lto should be coupled to the user setting -Zautodiff=Enable, which can be checked using
config.autodiff.contains(&config::AutoDiff::Enable)

[rust-log-analyzer]

The job x86_64-gnu-llvm-18 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#21 exporting to docker image format
#21 sending tarball 27.7s done
#21 DONE 37.7s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-18]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Listening on address 127.0.0.1:4226
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-18', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-18/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
   Compiling rustc_sanitizers v0.0.0 (/checkout/compiler/rustc_sanitizers)
   Compiling rustc_hir_analysis v0.0.0 (/checkout/compiler/rustc_hir_analysis)
   Compiling rustc_traits v0.0.0 (/checkout/compiler/rustc_traits)
   Compiling rustc_codegen_ssa v0.0.0 (/checkout/compiler/rustc_codegen_ssa)
error: unused import: `AutodiffWithoutLto`
  --> compiler/rustc_codegen_ssa/src/back/write.rs:44:21
   |
44 | use crate::errors::{AutodiffWithoutLto, ErrorCreatingRemarkDir};
   |
   = note: `-D unused-imports` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(unused_imports)]`

[bors]

☔ The latest upstream changes (presumably #138177) made this pull request unmergeable. Please resolve the merge conflicts.

[ZuseZ4]

@rustbot label +F-autodiff

[Dylan-DPC]

@ZuseZ4 any updates on this? thanks

[ZuseZ4]

@Dylan-DPC are you doing PR triage, or interested? Oli and I tried for a while, but in lib builds it's hard to compile everything in one coden unit, there were some very hardcoded blockers. I don't realistically have the capacity to try to fix this again before November.

[ZuseZ4]

fixed in another pr.