Qwen2-1.5B-Instruct
This document describes how to perform NPU hardware-accelerated inference of the Qwen2-1.5B-Instruct model on Qualcomm platforms using Qualcomm® Genie.
-
Source model: Qwen/Qwen2-1.5B-Instruct
-
Source model license: Apache 2.0
Model Details
| Model | Quantization | Context Length |
|---|---|---|
| Qwen2-1.5B-Instruct | W4A16 | 1024 |
Supported Devices
tip
Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.
-
This example supports Qualcomm platform SoCs with v73 DSP architecture.
dsp_arch v73 -
Supported devices
Device SoC dsp_arch Fogwise® AIRbox Q900 QCS9075 v73
Download qcom-qairt Dependencies
- QCS6490
- QCS9075
Device
sudo apt install qcom-qnn-sdk-v68 qcom-genie-sdk-v68
Device
sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73
Import Environment Variables
Device
export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu
Download Model
tip
Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage
Device
pip3 install modelscope
modelscope download --model radxa/Qwen2-1.5B-Instruct-w4a16-1024-v73 --local_dir ./Qwen2-1.5B-Instruct-w4a16-1024-v73
Run Inference
Device
cd Qwen2-1.5B-Instruct-w4a16-1024-v73
Build Prompt
Prompts can be passed as a file or as a parameter.
- prompt
- prompt_file
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Device
vim chat.txt
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant
Run Inference
- prompt
- prompt_file
Device
genie-t2t-run -c qwen2-1.5b-instruct-htp-228.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Device
genie-t2t-run -c qwen2-1.5b-instruct-htp-228.json --prompt_file chat.txt
(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Qwen2-1.5B-Instruct$ genie-t2t-run -c qwen2-1.5b-instruct-htp-228.json -p '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n'
Using libGenie.so version 1.14.0
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO] "Using create From Binary"
[INFO] "Allocated total size = 328208384 across 1 buffers"
[PROMPT]: <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGive me a short introduction to large language model.<|im_end|>\n<|im_start|>assistant\n
[BEGIN]: A large language model is a type of artificial intelligence model that is designed to process and analyze large amounts of text data. They are typically trained on vast amounts of text data, such as web pages, books, and social media content, and then used to generate text based on the input they receive. They can be trained to perform a a wide range of tasks, such as language translation, sentiment analysis, and text summarization. Large language models are often trained using deep learning techniques, and can be trained to perform complex natural language processing tasks, such as language translation, sentiment analysis, and text summarization. They are typically trained using large amounts of of text data, and can be trained to to perform complex natural language processing tasks. such as language translation, sentiment analysis, and text summarization.[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc
Performance Reference
You can enable performance profiling with the --profile option.
genie-t2t-run -c qwen2-1.5b-instruct-htp-228.json --prompt_file chat.txt --profile profile.txt
| Fogwise® AIRbox Q900 | |
|---|---|
| GenieDialog_create | 1,089,633 us |
| num-prompt-tokens | 29 |
| prompt-processing-rate | 79.63685607910156 toks/sec |
| time-to-first-token | 364,158 us |
| num-generated-tokens | 162 |
| token-generation-rate | 47.145347595214844 toks/sec |
| token-generation-time | 3,436,283 us |
| GenieDialog_free | 86,358 us |
Metric Definitions
| Metric | Definition |
|---|---|
| GenieDialog_create | Time to initialize a dialog object, including model loading, context preparation, and memory allocation. |
| num-prompt-tokens | Number of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into). |
| prompt-processing-rate | Speed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation. |
| time-to-first-token | Time elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency. |
| num-generated-tokens | Number of tokens actually output by the model in this generation, representing the length of the generated text in tokens. |
| token-generation-rate | Speed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency. |
| token-generation-time | Total time spent generating all output tokens, in microseconds (us). |
| GenieDialog_free | Time to free the dialog object, including memory release and resource cleanup. |
Official Genie Documentation
For more details on Qualcomm® Genie usage and API, refer to: