Skip to content

Waveform testing#267

Merged
ekiwi merged 5 commits into
mainfrom
waveform-testing
Jul 1, 2026
Merged

Waveform testing#267
ekiwi merged 5 commits into
mainfrom
waveform-testing

Conversation

@Nikil-Shyamsunder

@Nikil-Shyamsunder Nikil-Shyamsunder commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Do not merge until #264

Addressing #253. Most of the diff is the new runt runt/waveform config, the new ASCII output .expect files, and a bunch of shifts in the test because I removed the typechecking noise. Here are the main interesting things:

  • in graph_interp/main.rs and interp/main.rs, add support for printing the ASCII waveform output with --ascii-waveform
  • in the AST interpreter, before stepping the simulator we collect the DontCare/Concrete inputs and outputs and stick them in a waveform state which is just a FxHashMap<PortId, Vec<BitVecValue>>. An output is DontCare if any of its combinationally dependent inputs is DontCare. We do the same logic in graph_interpreter.rs.
  • there's also logic to actually pretty print the waveform in the format you had.

Diffing AST and Graph Interp

  • We want to diff results between the AST interpreter and the IR interpreter. They have different error messages, so really all we want to see is that the AST interpreter gives the same result for each trace (pass/fail) and the same waveform ouput. The problem is that there is no way to disable the error messages or route them to stderr instead of stdout.
  • To resolve this, I switched diagnostics (warning and error messaging) to go to stderr. The results ("Trace k excuted successfully/failed") , graph serialization with --graphout, and ASCII waveforms go to stdout. The detailed error messages go to stderr. I hope this is okay, or maybe we could add a flag that toggles detailed errors on/off or how they route in the CLI. This applies to interpreter/graph-interpreter only, not to monitor/bi. I then switched the runt targets to use the binaries directly instead of cargo run, so that stderr is clean with only the error messages (it doesn't have the noise from cargo run. This way, the waveform tests can check the trace results are the same and the waveform is the same, without dealing with warning or error messages being different. All the other tests just route the stderr back in with stdout with 2>&1 so there is no change for them. It also removes all the annoying typechecking warnings from the diff, as in issue Disable Type Inference Info in test/serialization outputs #263.

There are a few tests where our ability to identify that the waveforms are actually the same is lacking. For example an output that is dependent on inputs that we're dontcare in the previous cycle. I checked the ones that are failing and they are all passing with my eye-test, but I've excluded them for now and added to the issues.

switched interpreter to output diagnostics on stderr and base errors + ascii wveform on stdout
janky thing to keep monitor on stdout only by specifying the err/out semantics in the test gen
@Nikil-Shyamsunder Nikil-Shyamsunder requested a review from ekiwi July 1, 2026 15:45
@ekiwi ekiwi merged commit e0f030a into main Jul 1, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants