comparison with qualcomm ai hub model

[DongGeun123]

🐛 Describe the bug

I ran Llama-v3.2-3B-Chat(precision w4a16) from ai-hub-model on a Snapdragon 8 Gen 3 device, achieving 20 tokens/s.
For comparison, I ran inference for the Llama3.2-3B model quantized to W4A16 using executorch with the QNN backend on the same device. The performance I observed was 10 tokens/s.
Could you provide insights into what might be causing this performance difference? Are there issues with how executorch handles quantized models that could explain this gap?
Any guidance or suggestions would be greatly appreciated!

cc @cccclai @winskuo-quic @shewu-quic

[AndreaChiChengdu]

good question, i found the same issue on llama 3.1 8b via executorch qnn backend，which performance is below 1/2 compare with qualcomm qaihub chaimed
my device is xiaomi14pro with sm8650（v75 npu） 16GB ram

[kimishpatel]

cc: @cccclai

[cccclai]

Yeah we found out the model definition in llama_transformer.py isn't ideal for running llama model on qnn backend. We've started a new model definition in https://github.com/pytorch/executorch/tree/e66cdaf514e15242692073db1271aae4657f2033/examples/qualcomm/oss_scripts/llama3_2 which have better latency number

It's still wip and please expect some burden if trying them out, or maybe wait till it's more settled.

[Jack-Khuu]

Converting to discussion