Qualcomm AI Engine Direct - DRAFT for GA QWEN2.5 0.5B

[winskuo-quic]

Summary

Static Qwen2.5 0.5b enablement.

Script
python examples/qualcomm/oss_scripts/qwen/qwen.py -b build-android -s $DEVICE -m SM8750 --decoder_model qwen2_5 --prompt "Hello, how are you?" --max_seq_len 128

Stats

Speed: 128tok/sec on SM8750, seq_len = 128
Accuracy: Bad (QDQ already bad but not as bad)

Test plan

[PLEASE REMOVE] How did you test this PR? Please write down any manual commands you used and note down tests that you have written if applicable.

Author: @haowhsu-quic, @winskuo-quic

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12054

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• VolumeLimitExceeded Issue for linux.2xlarge and linux.4xlarge

❌ 1 New Failure

As of commit 0c4785b with merge base 695c7d5 ():

NEW FAILURE - The following job has failed:

• pull / test-static-llama-qnn-linux / linux-job (gh)
  test_llama_stories_110m

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[winskuo-quic]

Hi @billmguo, @cccclai,

This is a draft for qwen2.5 0.5 using static nn.Module structure.
Please notice this is just a draft as the accuracy for the model is still bad.

Thanks

[cccclai]

Thanks! If it's just the accuracy is bad, @rohansjoshi has been making progress on this. Can qwen model run with this PR (don't need to be accurate)?

[cccclai]

I think we need to have the same static llama file (maybe with some branches) instead of starting a new static qwen model

[winskuo-quic]

Thanks for reviewing the PR. We are trying to have a static decoder file that can be shared across all GA decoder models + LLama.
For this draft, we are just making sure e2e flow is working, and we will work on merging all static decoder files to 1.

[winskuo-quic]

Thanks! If it's just the accuracy is bad, @rohansjoshi has been making progress on this. Can qwen model run with this PR (don't need to be accurate)?

Yes. The can model can run with this PR.
I have an example script and stats shared in the summary section.