fix(metrics): correct params & MACs for quantized models#8
Merged
Conversation
Quantized models (INT8 fbgemm/qnnpack) have CPU-only kernels. benchmark() defaults to devices [cpu, cuda] on a CUDA host; moving a quantized model to CUDA dispatches e.g. quantized::conv2d_relu on a QuantizedCUDA tensor with no registered kernel -> PyTorch's dispatcher SEGFAULTS (uncatchable). Hits any workflow that INT8-quantizes a model and then benchmarks it. Fix: - _is_quantized() + _device_supported()/_ensure_device_supported() in core. - _run_on_devices skips unsupported devices (CUDA for quantized) -> NaN + warning: one choke point for the speed/memory/energy multi-device profilers. - Belt-and-suspenders guard at each single-device forward entry raises a catchable RuntimeError instead of segfaulting. Non-quantized models unaffected (ResNet-18 still benches cpu+cuda). Regression tests added; nbdev-test green on all 5 touched notebooks.
Quantized weights live in packed params (_packed_params), not .parameters(), so get_num_parameters returned 0; thop has no quantized-op handlers so MACs read ~0. But quantization keeps the SAME param & MAC counts (int8 vs fp32) -- only bytes and op-cost drop. Now correct: - get_num_parameters: when _is_quantized, add the numel of quantized modules' callable .weight()/.bias() accessors -> params(quant) == params(fp32). - compute_compute: route quantized models through a forward-hook MAC counter (handles quantized + fp Conv2d/Linear); the thop path is unchanged for fp models. Measured: params 0 -> 5220 (= fp32); MACs 0.008 -> 1.29M (0.99x fp32); disk still drops (int8). Protects the before/after report from fake 'params/MACs -> 0 (-100%)' rows on quantized outputs. nbdev-test green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fasterbench's param-count and MAC metrics were silently wrong for quantized models —
get_num_parametersreturned 0 (packed int8 weights aren't in.parameters()) and thop-based MACs read ≈0 (no quantized-op handlers).The conceptual fix
Quantization does not change the number of params or MACs — an int8 conv has the same weight count and multiply-accumulate count as its fp32 twin; only bytes and op-cost drop. So the counts should come out ≈ unchanged, and only model size (bytes) should shrink.
get_num_parameters— when_is_quantized, also sum the numel of quantized modules' callable.weight()/.bias()accessors.compute_compute— route quantized models through a forward-hook MAC counter (handles quantized + fpConv2d/Linear); the thop path is untouched for fp models.Measured (before → after)
Why it matters
The
optimize(..., report=True)before/after table benchmarks the optimized model — which for the defaultsize/speedrecipe is quantized. Without this, it would print misleadingparams/MACs → 0 (−100%)rows.nbdev-testgreen.