Qwen2.5-3B-Instruct
This document describes how to perform NPU hardware-accelerated inference of the Qwen2.5-3B-Instruct model on Qualcomm platforms using Qualcomm® Genie.
-
Source model: Qwen/Qwen2.5-3B-Instruct
-
Source model license: Qwen RESEARCH LICENSE AGREEMENT
Model Details
| Model | Quantization | Context Length |
|---|---|---|
| Qwen2.5-3B-Instruct | W4A16 | 4096 |
Supported Devices
tip
Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.
-
This example supports Qualcomm platform SoCs with v73 DSP architecture.
dsp_arch v73 -
Supported devices
Device SoC dsp_arch Fogwise® AIRbox Q900 QCS9075 v73
Download qcom-qairt Dependencies
- QCS6490
- QCS9075
Device
sudo apt install qcom-qnn-sdk-v68 qcom-genie-sdk-v68
Device
sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73
Import Environment Variables
Device
export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu
Download Model
tip
Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage
Device
pip3 install modelscope
modelscope download --model radxa/Qwen2.5-3B-Instruct-w4a16-4096-v73 --local_dir ./Qwen2.5-3B-Instruct-w4a16-4096-v73
Run Inference
Device
cd Qwen2.5-3B-Instruct-w4a16-4096-v73
Build Prompt
Prompts can be passed as a file or as a parameter.
- prompt
- prompt_file
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Device
vim chat.txt
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Run Inference
- prompt
- prompt_file
Device
genie-t2t-run -c qwen2.5-3b-instruct-htp.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Device
genie-t2t-run -c qwen2.5-3b-instruct-htp.json --prompt_file chat.txt
(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Qwen2.5-3B-Instruct/qnn231_q8280_cl4096$ genie-t2t-run -c qwen2.5-3b-instruct-htp.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Using libGenie.so version 1.14.0
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO] "Using create From Binary"
[INFO] "Allocated total size = 85753984 across 5 buffers"
[PROMPT]: <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n
[BEGIN]: A large language model (LLM) is a type of artificial intelligence model designed to understand and generate human-like text. These models are trained on vast amounts of text data, allowing them to learn patterns, contexts, and relationships within language. They can then generate text that is coherent, relevant, and sometimes even creative, given a prompt or question. Examples of of large language models include GPT-3, Claude, and Qwen.[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc
Performance Reference
You can enable performance profiling with the --profile option.
genie-t2t-run -c qwen2.5-3b-instruct-htp.json --prompt_file chat.txt --profile profile.txt
| Fogwise® AIRbox Q900 | |
|---|---|
| GenieDialog_create | 2,435,264 us |
| num-prompt-tokens | 39 |
| prompt-processing-rate | 428.81646728515625 toks/sec |
| time-to-first-token | 90,983 us |
| num-generated-tokens | 218 |
| token-generation-rate | 21.96016502380371 toks/sec |
| token-generation-time | 9,927,213 us |
| GenieDialog_free | 69,839 us |
Metric Definitions
| Metric | Definition |
|---|---|
| GenieDialog_create | Time to initialize a dialog object, including model loading, context preparation, and memory allocation. |
| num-prompt-tokens | Number of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into). |
| prompt-processing-rate | Speed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation. |
| time-to-first-token | Time elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency. |
| num-generated-tokens | Number of tokens actually output by the model in this generation, representing the length of the generated text in tokens. |
| token-generation-rate | Speed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency. |
| token-generation-time | Total time spent generating all output tokens, in microseconds (us). |
| GenieDialog_free | Time to free the dialog object, including memory release and resource cleanup. |
Official Genie Documentation
For more details on Qualcomm® Genie usage and API, refer to: