Negative values for tokens per sec for Mixtral benchmarsk

[prasad-nair-amd]

I am using the nvidia provided benchmark scripts running Mixtral 8x7B model on H100 why its reporting negative values for throughput ?

[BENCHMARK] num_samples 1
[BENCHMARK] num_error_samples 0

[BENCHMARK] num_samples 1
[BENCHMARK] total_latency(ms) 4610.37
[BENCHMARK] seq_throughput(seq/sec) 0.22
[BENCHMARK] token_throughput(token/sec) -2385.71

[BENCHMARK] avg_sequence_latency(ms) 4608.56
[BENCHMARK] max_sequence_latency(ms) 4608.56
[BENCHMARK] min_sequence_latency(ms) 4608.56
[BENCHMARK] p99_sequence_latency(ms) 4608.56
[BENCHMARK] p90_sequence_latency(ms) 4608.56
[BENCHMARK] p50_sequence_latency(ms) 4608.56

[BENCHMARK] avg_time_to_first_token(ms) 173.11
[BENCHMARK] max_time_to_first_token(ms) 173.11
[BENCHMARK] min_time_to_first_token(ms) 173.11
[BENCHMARK] p99_time_to_first_token(ms) 173.11
[BENCHMARK] p90_time_to_first_token(ms) 173.11
[BENCHMARK] p50_time_to_first_token(ms) 173.11

[BENCHMARK] avg_inter_token_latency(ms) 0.00
[BENCHMARK] max_inter_token_latency(ms) 0.00
[BENCHMARK] min_inter_token_latency(ms) 0.00
[BENCHMARK] p99_inter_token_latency(ms) 0.00
[BENCHMARK] p90_inter_token_latency(ms) 0.00
[BENCHMARK] p50_inter_token_latency(ms) 0.00

[TensorRT-LLM][INFO] Terminate signal received, worker thread exiting.
[TensorRT-LLM][INFO] Terminate signal received, worker thread exiting.
[TensorRT-LLM][INFO] Terminate signal received, worker thread exiting.
[TensorRT-LLM][INFO] Terminate signal received, worker thread exiting.

[Advait251206]

A negative token_throughput(token/sec) is not an expected benchmark result, so this usually indicates that the benchmark's token accounting has gone wrong rather than the model actually producing a negative throughput.

A few things to check:

• Verify the benchmark configuration, particularly the input and output token lengths (max_input_len, max_output_len, etc.). If the benchmark computes generated tokens as output_tokens - input_tokens and the output is unexpectedly smaller than the input (or token counts are otherwise inconsistent), the throughput calculation can become negative.
• Inspect the generated outputs to confirm that the model is actually generating the expected number of new tokens. If generation terminates immediately (e.g., EOS is produced as the first token), the benchmark's accounting may not match your expectations.
• Check the benchmark script version. There have been fixes to benchmarking utilities over time, so it's worth confirming whether you're using the latest version of TensorRT-LLM or whether this issue has already been addressed.
• Run with more than a single sample. Since your output shows num_samples 1, it's easier for accounting bugs or edge cases to surface. Testing with multiple prompts may help determine whether this is specific to one request or a systematic issue.

One thing that stands out is:

avg_inter_token_latency(ms) 0.00
token_throughput(token/sec) -2385.71

If no inter-token latencies are being recorded, it suggests the benchmark may not be observing any generated tokens at all, which would further point toward an issue in token counting or benchmark metric calculation rather than inference performance.

Could you also share:

• The exact TensorRT-LLM commit or release you're using.
• The benchmark command you ran.
• The values of max_input_len, max_output_len, and the actual prompt.
• Whether this happens only with Mixtral 8x7B or with other models as well.

That information would help determine whether this is a configuration issue or a bug in the benchmark script.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206