Allow unwinding from OOM hooks

[Amanieu]

This is split off from #88098 and contains just the bare minimum to allow specifying a custom OOM hook with set_alloc_error_hook which unwinds instead of aborting.

See #88098 for an actual command-line flag which switches the default OOM behavior to unwind instead of aborting.

Previous perf results show a negligible impact on performance.

[rust-highfive]

r? @m-ou-se

(rust-highfive has picked a reviewer for you, use r? to override)

[m-ou-se]

From your description I assumed that this program would abort in the current version of rust, and would succeed after the change in this PR:

#![feature(alloc_error_hook)]

fn main() {
    std::alloc::set_alloc_error_hook(|_| {
        panic!("oh no");
    });
    assert!(std::panic::catch_unwind(|| {
        vec![0u8; 12345678901234567890];
    }).is_err());
    println!("i'm ok :)");
}

But it already succeeds on the current version of rust:

thread 'main' panicked at 'oh no', src/main.rs:5:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
i'm ok :)

Am I misunderstanding the purpose of this PR?

[bjorn3]

I think panicking inside the alloc error hook is currently UB.

[Amanieu]

Yes, it's currently UB due to extern "C" and #[rustc_allocator_nounwind] which imply nounwind but don't actually enforce it.

[Amanieu]

If you try using a type with Drop you will see that the drop gets optimized away because the compiler assumes the allocator doesn't unwind.

[m-ou-se]

Ah, interesting. I didn't manage to make it fail on x86_64-unknown-linux-gnu, aarch64-apple-darwin, nor x86_64-pc-windows-gnu, but it does indeed fail on aarch64-unknown-linux-musl and x86_64-unknown-linux-musl. With this patch applied, it succeeds on all of them.

@bors r+

[bors]

📌 Commit f4545cc has been approved by m-ou-se

[m-ou-se]

If you try using a type with Drop you will see that the drop gets optimized away because the compiler assumes the allocator doesn't unwind.

If I add a let _a = String::new(); in main, it seems to abort without unwinding before this patch. After this patch, it works fine.

[ehuss]

Oh, sorry, I didn't see that.

@bors r-

[hkratz]

Not really on Windows much these days, maybe someone from the Windows notification group can help?

@rustbot ping windows

[rustbot]

Error: Only Rust team members can ping teams.

Please let @rust-lang/release know if you're having trouble with this bot.

[Amanieu]

Just a backtrace at the point where it is hanging would be a good start to figuring out what the problem is.

[bjorn3]

I can see this error in the log when it tries to build rls:

error[E0635]: unknown feature `thread_local_const_init`
  --> C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\rustc-ap-rustc_span-722.0.0\src\lib.rs:23:12
   |
23 | #![feature(thread_local_const_init)]
   |            ^^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0635`.

Could this somehow be causing the hang?

Otherwise I have no idea how to go about debugging this.

No, rls fails to build on master too. See #91543.

[ehuss]

I'm able to reproduce locally. At a minimum it required llvm assertions, though it may also require parallel-compiler.

The reason it is hanging is because it pops open a helpful dialog box.

In this particular instance, it was hung building tracing-subscriber.

Here is a stack trace:

win32u.dll!NtUserWaitMessage+0x14
USER32.dll!DialogBoxIndirectParamAorW+0x431
USER32.dll!DialogBoxIndirectParamAorW+0x1a1
USER32.dll!SoftModalMessageBox+0x85b
USER32.dll!DrawStateA+0x1e25
USER32.dll!MessageBoxTimeoutW+0x1a7
USER32.dll!MessageBoxW+0x4e
rustc_driver-9acede3d415bc9ac.dll!__acrt_MessageBoxW+0x56
rustc_driver-9acede3d415bc9ac.dll!__acrt_show_wide_message_box+0x7c
rustc_driver-9acede3d415bc9ac.dll!common_assert_to_message_box<wchar_t>+0x74
rustc_driver-9acede3d415bc9ac.dll!llvm::InlineFunction+0x8366
rustc_driver-9acede3d415bc9ac.dll!llvm::InlinerPass::run+0x15d8
rustc_driver-9acede3d415bc9ac.dll!llvm::detail::PassModel<llvm::LazyCallGraph::SCC,llvm::InlinerPass,llvm::PreservedAnalyses,llvm::AnalysisManager<llvm::LazyCallGraph::SCC,llvm::LazyCallGraph & __ptr64>,llvm::LazyCallGraph & __ptr64,llvm::CGSCCUpdateResul
rustc_driver-9acede3d415bc9ac.dll!llvm::PassManager<llvm::LazyCallGraph::SCC,llvm::AnalysisManager<llvm::LazyCallGraph::SCC,llvm::LazyCallGraph & __ptr64>,llvm::LazyCallGraph & __ptr64,llvm::CGSCCUpdateResult & __ptr64>::run+0x30b
rustc_driver-9acede3d415bc9ac.dll!llvm::detail::PassModel<llvm::LazyCallGraph::SCC,llvm::PassManager<llvm::LazyCallGraph::SCC,llvm::AnalysisManager<llvm::LazyCallGraph::SCC,llvm::LazyCallGraph & __ptr64>,llvm::LazyCallGraph & __ptr64,llvm::CGSCCUpdateResu
rustc_driver-9acede3d415bc9ac.dll!llvm::DevirtSCCRepeatedPass::run+0x200
rustc_driver-9acede3d415bc9ac.dll!llvm::detail::PassModel<llvm::LazyCallGraph::SCC,llvm::DevirtSCCRepeatedPass,llvm::PreservedAnalyses,llvm::AnalysisManager<llvm::LazyCallGraph::SCC,llvm::LazyCallGraph & __ptr64>,llvm::LazyCallGraph & __ptr64,llvm::CGSCCU
rustc_driver-9acede3d415bc9ac.dll!llvm::ModuleToPostOrderCGSCCPassAdaptor::run+0x114b
rustc_driver-9acede3d415bc9ac.dll!llvm::detail::PassModel<llvm::Module,llvm::ModuleToPostOrderCGSCCPassAdaptor,llvm::PreservedAnalyses,llvm::AnalysisManager<llvm::Module> >::run+0x11
rustc_driver-9acede3d415bc9ac.dll!llvm::PassManager<llvm::Module,llvm::AnalysisManager<llvm::Module> >::run+0x2a2
rustc_driver-9acede3d415bc9ac.dll!llvm::ModuleInlinerWrapperPass::run+0x24c
rustc_driver-9acede3d415bc9ac.dll!llvm::detail::PassModel<llvm::Module,llvm::ModuleInlinerWrapperPass,llvm::PreservedAnalyses,llvm::AnalysisManager<llvm::Module> >::run+0x11
rustc_driver-9acede3d415bc9ac.dll!llvm::PassManager<llvm::Module,llvm::AnalysisManager<llvm::Module> >::run+0x2a2
rustc_driver-9acede3d415bc9ac.dll!LLVMRustOptimizeWithNewPassManager+0x1f0e
rustc_driver-9acede3d415bc9ac.dll!RNvNtNtCs8Ww9ijeY60a_18rustc_codegen_llvm4back5write35optimize_with_new_llvm_pass_manager+0x400
rustc_driver-9acede3d415bc9ac.dll!RNvNtNtCs8Ww9ijeY60a_18rustc_codegen_llvm4back3lto16run_pass_manager+0x242
rustc_driver-9acede3d415bc9ac.dll!RNvNtNtCs8Ww9ijeY60a_18rustc_codegen_llvm4back3lto20optimize_thin_module+0x2051
rustc_driver-9acede3d415bc9ac.dll!RNvMs_NtNtCs5mXDuW1fxtA_17rustc_codegen_ssa4back3ltoINtB4_16LtoModuleCodegenNtCs8Ww9ijeY60a_18rustc_codegen_llvm18LlvmCodegenBackendE8optimizeB1e_+0x2a
rustc_driver-9acede3d415bc9ac.dll!RINvNtNtCs5mXDuW1fxtA_17rustc_codegen_ssa4back5write17execute_work_itemNtCs8Ww9ijeY60a_18rustc_codegen_llvm18LlvmCodegenBackendEB19_+0x2fc
rustc_driver-9acede3d415bc9ac.dll!RINvNtNtCsbfuXEFdO0KO_3std10sys_common9backtrace28___rust_begin_short_backtraceNCINvXs0_Cs8Ww9ijeY60a_18rustc_codegen_llvmNtB1o_18LlvmCodegenBackendNtNtNtCs5mXDuW1fxtA_17rustc_codegen_ssa6traits7backend19ExtraBackendMetho
rustc_driver-9acede3d415bc9ac.dll!RINvXss_Cs6sEOyLJiBiG_8smallvecINtB6_8SmallVecARNtNtCsg0nmUIWefZS_12rustc_middle2ty3TySj8_EINtNtNtNtCscKNrPtuGzPt_4core4iter6traits7collect6ExtendBJ_E6extendINtNtNtB1y_8adapters5chain5ChainINtNtNtB1y_7sources4once4OnceBJ_
std-fbf1316f493850ac.dll!std::sys::windows::thread::impl$0::new::thread_start+0x4c
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

I'm pretty much out of my depth when it comes to digging into LLVM, though.

[nikic]

@ehuss Could you please rerun the asserting rustc command with -C save-temps and zip the results? Hopefully this is reproducible on the bitcode.

[ehuss]

Here are the temp files:
libtracing_subscriber-1e4a1d5d5521f0b9.zip

[ehuss]

I confirmed the only non-default setting needed to trigger it is llvm.assertions, and building tracing-subscriber 0.3.3 with --release.

[nikic]

Best reduction I have for now is https://gist.github.com/nikic/2ff417bd7df48d660f5af0ea47ee3c89 with -passes="cgscc(inline,early-cse<memssa>)".

Edit: https://gist.github.com/nikic/65871cd4c29ded2d1b21cebbdab3f052 asserts under opt -inline.

[nikic]

I filed llvm/llvm-project#53206.

[nikic]

Fixed upstream by llvm/llvm-project@4810051.

[Amanieu]

Thanks @nikic!

Let's try this again.

@bors r=m-ou-se

[bors]

📌 Commit 2082842 has been approved by m-ou-se

[bors]

⌛ Testing commit 2082842 with merge 719b04c...

[bors]

☀️ Test successful - checks-actions
Approved by: m-ou-se
Pushing 719b04c to master...

[rust-timer]

Finished benchmarking commit (719b04c): comparison url.

Summary: This benchmark run shows 34 relevant regressions 😿 to instruction counts.

• Average relevant regression: 0.8%
• Average relevant improvement: -0.3%
• Largest regression in instruction counts: 1.9% on incr-patched: println builds of regression-31157 opt

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression