Qwen2-0.5B-Instruct
This document describes how to perform NPU hardware-accelerated inference of the Qwen2-0.5B-Instruct model on Qualcomm platforms using Qualcomm® Genie.
-
Source model: Qwen/Qwen2-0.5B-Instruct
-
Source model license: Apache 2.0
Model Details
| Model | Quantization | Context Length |
|---|---|---|
| Qwen2-0.5B-Instruct | W4A16 | 1024 |
Supported Devices
tip
Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.
-
This example supports Qualcomm platform SoCs with v73 DSP architecture.
dsp_arch v73 -
Supported devices
Device SoC dsp_arch Fogwise® AIRbox Q900 QCS9075 v73
Download qcom-qairt Dependencies
- QCS6490
- QCS9075
Device
sudo apt install qcom-qnn-sdk-v68 qcom-genie-sdk-v68
Device
sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73
Import Environment Variables
Device
export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu
Download Model
tip
Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage
Device
pip3 install modelscope
modelscope download --model radxa/Qwen2-0.5B-Instruct-w4a16-1024-v73 --local_dir ./Qwen2-0.5B-Instruct-w4a16-1024-v73
Run Inference
Device
cd Qwen2-0.5B-Instruct-w4a16-1024-v73
Build Prompt
Prompts can be passed as a file or as a parameter.
- prompt
- prompt_file
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Device
vim chat.txt
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Run Inference
- prompt
- prompt_file
Device
genie-t2t-run -c qwen2-0.5b-instruct-htp-228.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Device
genie-t2t-run -c qwen2-0.5b-instruct-htp-228.json --prompt_file chat.txt
(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Qwen2-0.5B-Instruct$ genie-t2t-run -c qwen2-0.5b-instruct-htp-228.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Using libGenie.so version 1.14.0
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO] "Using create From Binary"
[INFO] "Allocated total size = 319688704 across 1 buffers"
[PROMPT]: <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n
[BEGIN]: A large language model is a type of artificial intelligence that can generate human-like language. It is trained on large amounts of text data, which is typically of high quality and is used to generate human-like language. This model can be used for a variety of applications, such as language translation, language generation, and language prediction.[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc
Performance Reference
You can enable performance profiling with the --profile option.
genie-t2t-run -c qwen2-0.5b-instruct-htp-228.json --prompt_file chat.txt --profile profile.txt
| Fogwise® AIRbox Q900 | |
|---|---|
| GenieDialog_create | 786,644 us |
| num-prompt-tokens | 29 |
| prompt-processing-rate | 141.1432647705078 toks/sec |
| time-to-first-token | 205,491 us |
| num-generated-tokens | 81 |
| token-generation-rate | 108.5304946899414 toks/sec |
| token-generation-time | 746,347 us |
| GenieDialog_free | 80,939 us |
Metric Definitions
| Metric | Definition |
|---|---|
| GenieDialog_create | Time to initialize a dialog object, including model loading, context preparation, and memory allocation. |
| num-prompt-tokens | Number of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into). |
| prompt-processing-rate | Speed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation. |
| time-to-first-token | Time elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency. |
| num-generated-tokens | Number of tokens actually output by the model in this generation, representing the length of the generated text in tokens. |
| token-generation-rate | Speed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency. |
| token-generation-time | Total time spent generating all output tokens, in microseconds (us). |
| GenieDialog_free | Time to free the dialog object, including memory release and resource cleanup. |
Official Genie Documentation
For more details on Qualcomm® Genie usage and API, refer to: