mx.matmul overhead

[tsengalb99]

Do you have any benchmarks showing where the extra overhead of mx.matmul over a regular matmul is? Is it in the quantization step (calculating scales, rounding, etc.)? If so, do you know if devices with MX support will do this rounding in the hardware itself, and if so, will the overhead become negligible there because of the hardware support?

[huweim]

I have the same question. Report by QServe, the quant in GEMM loop introduces significant overhead. How about MX format? (including rounding, calculating scales, data type casting, computation with MX scaling, etc.)

I have noticed the chip Maia 100 supports MX format, it seems to provide some hardware optimization.