Skip to content

Add a script to plot benchmark results#314

Open
bernhardmgruber wants to merge 5 commits intoNVIDIA:mainfrom
bernhardmgruber:plot_script
Open

Add a script to plot benchmark results#314
bernhardmgruber wants to merge 5 commits intoNVIDIA:mainfrom
bernhardmgruber:plot_script

Conversation

@bernhardmgruber
Copy link
Collaborator

@bernhardmgruber bernhardmgruber commented Feb 5, 2026

This script is 100% vibe-coded. I only looked at the result, since I don't understand Python. Here is a generated summary of what this PR proposes:

Add nvbench_plot.py to plot horizontal bar charts from NVBench JSON results using %SOL (global BW utilization).
Support filtering by axis values (-a with optional [pow2]) and benchmark name (-b).
Improve plot presentation with per-benchmark colors, tighter vertical spacing, and sensible default title.

Examples filtering by axis and benchmark name:

python ./python/scripts/nvbench_plot.py -b nstream -b triad -a Elements{io}[pow2]=28 .
/babelstream_ublkcp_B200.json
image

Examples with two files, filtering by axis, dark mode:

python ./python/scripts/nvbench_plot.py -a Elements{io}[pow2]=28 -a T{ct}=F32 ./babels
tream_ublkcp_B200.json ./pytorch_ublkcpy_B200.json --dark
image

Here are the input files for the plots:
pytorch_ublkcpy_B200.json
babelstream_ublkcp_B200.json

Here are the prompts that I used:

  1. “Please study the scripts in the directory python/scripts. Then look at the benchmark result babelstream_ublkcp_B200.json. Create a new script nvbench_plot.py in the above directory that can read the benchmark result and plot a horizontal bar chart, where each benchmark state is plotted as a separate box. The box length is taken from nv/cold/bw/global/utilization
  2. “If I run the script like python ./python/scripts/nvbench_plot.py ./babelstream_ublkcp_B200.json it errors, please fix the error”
  3. “Let's add a command line parameter -a where I can specify an axis name and value like -a T{ct}=I8. When specified only benchmark states are shown where the axis_value node in the JSON has an axis named T{ct} with a value of I8
  4. “Please add special handling to the -a argument handling where when I enter an like -a Elements{io}[pow2]=28 the [pow2] is parsed before the = and it converts the value after the = (28 in this case) to 2 to the power of that value (2^28 in this case)”
  5. “Let's add another command line parameter -b where i can select the benchmark name”
  6. “Make the bar color #76b900”
  7. “Please do not plot the file name next to the bar charts”
  8. “Please include the device name in the bar labels”
  9. “Let's remove the device name again”
  10. “Please remove the Device=N part from the bar label”
  11. “Can you use different colors for the bars per benchmark.”
  12. “Add a plot tile "%SOL Bandwidth". If the device of all benchmarks is the same, put the device name in the benchmark title like "%SOL Bandwidth - B200"”
  13. “I get the warning

The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use matplotlib.colormaps[name] or matplotlib.colormaps.get_cmap() or pyplot.get_cmap() instead.

Please fix it”
14. “Please reduce the empty space above the first bar and below the last bar”
15. “Please write a pull request description about what this new script can do”
16. "Add --dark mode with black background and white text."
17. "Ensure title is visible in dark mode."
18. "Remove the x‑axis description label."
19. "If -a is used, remove those axis tokens from labels and add them to the title."
20. “Please read through the nvbench code here and figure out whether it supports -a paramters where we can also specify multiple values”
21. “Please implement this”
22. “If multiple values are specified for an axis, retain the axis value in the box label”

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant