[crater experiment] Enable LLVM assertions

[saethlin]

r? @ghost

Enabling LLVM assertions on Linux so that we can do a crater run to look for issues similar to #101585

[saethlin]

@bors try

[bors]

@saethlin: 🔑 Insufficient privileges: not in try users

[tmiasko]

@bors try

[bors]

⌛ Trying commit 1e7d884 with merge 434cce50d8e9a9de0ddaaccaca51e3e3f4bbb275...

[matthiaskrgr]

since we are doing this, should we also enable rustc debug assertions at the same time? :D

[saethlin]

@matthiaskrgr If you're volunteering to help look through what comes out, then yes. Otherwise I have little confidence in my ability to understand the Rust ICEs. They all look the same to me.

[matthiaskrgr]

finding interesting crater results can be fairly automatized; I made this script some time ago:

#!/bin/bash

EXP=$1

mkdir -p crater/$EXP
cd crater/$EXP
wget -c "https://crater-reports.s3.amazonaws.com/${EXP}/logs-archives/all.tar.gz"
tar -xzf all.tar.gz
echo "interesting findings:"
rg "LLVM ERROR"
rg "delay_span_bug"
rg "query stack during panic"
rg "internal compiler error"
#rg "RUST_BACKTRACE="

You just run main.sh pr-10147 and it will download and extract the crater logs (note: this easily by 5-10 gb when extracted 😅 ) and search for anything that looks like a crash and print it to the terminal.

( should probably add something like "Assertion `" for llvm assertions...)

[saethlin]

Finding the interesting logs has never been my issue, it's understanding what they mean.

For now I think we should just see how many regressions we get with LLVM assertions. If it's just a few (which I think is unlikely) we can then look for additional regressions with the Rust debug assertions. It seems to me like the crater queue isn't very busy.

[matthiaskrgr]

Hmm 😅 well usually I just check the bugtracker and if I don't see something that looks similar i just open an issue 🙃

[bors]

☀️ Try build successful - checks-actions
Build commit: 434cce50d8e9a9de0ddaaccaca51e3e3f4bbb275 (434cce50d8e9a9de0ddaaccaca51e3e3f4bbb275)

[est31]

FTR they were originally enabled by default, but then they were disabled by default in #45810, leaving "alt" builds for users who want to test builds with LLVM assertions enabled. Later alt builds were removed entirely however. Then, #75199 has reenabled assertions again for some CI builders (but not binaries that are shipped from my understanding).

[tmiasko]

@craterbot run mode=build-only

[craterbot]

👌 Experiment pr-101591 created and queued.
🤖 Automatically detected try build 434cce50d8e9a9de0ddaaccaca51e3e3f4bbb275
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🚧 Experiment pr-101591 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-101591 is completed!
📊 60 regressed and 8 fixed (243080 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[bjorn3]

At least one assertion failure:

• Cyborus04.more_nums.1d534ec4f0be6f83992bfd48e8e357fca8c4a79e: /checkout/src/llvm-project/llvm/include/llvm/ADT/APInt.h:1469: uint64_t llvm::APInt::getZExtValue() const: Assertion `getActiveBits() <= 64 && "Too many bits for uint64_t"'

[saethlin]

This same assertion is hit in 21 crates, but crater only detects 2 of those cases as regressions, because the other 19 crates always die in crater due to what looks like OOM. I'm guessing that crater can't tell the difference between a SIGKILL due to OOM and a SIGKILL due to an LLVM assertion.

In any case, that looks like the same assertion detecting the same bug which prompted me to open this PR. I see a fix for it has been merged so I'll manually see if those crates are fixed by it and report back.

[saethlin]

I've checked a few of the crates that have failed, and none hit the assertion anymore. I'm calling this closed.