Skip to content

Cortex-M: quantize aten.matmul by rewriting it to bmm before annotation#21028

Open
rascani wants to merge 1 commit into
pytorch:mainfrom
rascani:cortexm-matmul-bmm-pr
Open

Cortex-M: quantize aten.matmul by rewriting it to bmm before annotation#21028
rascani wants to merge 1 commit into
pytorch:mainfrom
rascani:cortexm-matmul-bmm-pr

Conversation

@rascani

@rascani rascani commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

The cortex_m quantizer annotates only aten.bmm.default, but torch.matmul / the @ operator is captured as aten.matmul.default and does not decompose to bmm until to_edge -- after quantization runs -- so matmuls (for example every attention score and context product) never received qparams and stayed in fp32 on portable kernels.

Add a pre-annotation MatmulToBmmPass that rewrites aten.matmul to aten.bmm (rank-3 directly; higher ranks by folding the leading batch dims to 3D and reshaping back; rank-2 and broadcasting matmuls left unchanged), so the existing CortexMBmmCheck annotation and quantized_batch_matmul lowering handle it. No new kernel is needed -- the batch-matmul kernel already exists; this is purely an annotation-timing fix.

Test plan

Verified test-first (RED->GREEN) with rank-3 and rank-4 dialect tests and no regression on the existing bmm suite; confirmed it flips all 15 SAM mask-decoder attention matmuls from fp32 to quantizable.

Authored with Claude Code.

…annotation

The cortex_m quantizer annotates only aten.bmm.default, but torch.matmul / the @ operator is captured as aten.matmul.default and does not decompose to bmm until to_edge -- after quantization runs -- so matmuls (for example every attention score and context product) never received qparams and stayed in fp32 on portable kernels.

Add a pre-annotation MatmulToBmmPass that rewrites aten.matmul to aten.bmm (rank-3 directly; higher ranks by folding the leading batch dims to 3D and reshaping back; rank-2 and broadcasting matmuls left unchanged), so the existing CortexMBmmCheck annotation and quantized_batch_matmul lowering handle it. No new kernel is needed -- the batch-matmul kernel already exists; this is purely an annotation-timing fix.

Verified test-first (RED->GREEN) with rank-3 and rank-4 dialect tests and no regression on the existing bmm suite; confirmed it flips all 15 SAM mask-decoder attention matmuls from fp32 to quantizable.

Authored with Claude Code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rascani
rascani requested review from AdrianLundell and psiddh July 17, 2026 21:50
@pytorch-bot

pytorch-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21028

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Unclassified Failure

As of commit 21bdfd3 with merge base 7013c8d (image):

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant