Phi-2

This document describes how to perform NPU hardware-accelerated inference of the Phi-2 model on Qualcomm platforms using Qualcomm® Genie.

• Source model: microsoft/phi-2

• Source model license: MIT

Model Details

Model	Quantization	Context Length
Phi-2	W4A16	1024

Supported Devices

tip

Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.

• This example supports Qualcomm platform SoCs with v73 DSP architecture.

  dsp_arch
  v73

• Supported devices

  Device	SoC	dsp_arch
  Fogwise® AIRbox Q900	QCS9075	v73

Download qcom-qairt Dependencies

QCS6490

Device

sudo apt install qcom-qnn-sdk-v68 qcom-genie-sdk-v68

QCS9075

Device

sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73

Import Environment Variables

Device

export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu

Download Model

tip

Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage

Device

pip3 install modelscope
modelscope download --model radxa/Phi-2-w4a16-1024-v73 --local_dir ./Phi-2-w4a16-1024-v73

Run Inference

Device

cd Phi-2-w4a16-1024-v73

Build Prompt

Prompts can be passed as a file or as a parameter.

prompt

System: You are a helpful assistant\nUser: Introduce Qualcomm in 100 words\nAssistant:

prompt_file

Device

vim chat.txt

System: You are a helpful assistant\nUser: Introduce Qualcomm in 100 words\nAssistant:

Run Inference

prompt

Device

genie-t2t-run -c phi-2-htp.json -p 'System: You are a helpful assistant\nUser: Introduce Qualcomm in 100 words\nAssistant:'

prompt_file

Device

genie-t2t-run -c phi-2-htp.json --prompt_file chat.txt

(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Mistral-7B-Instruct-v0.3$ genie-t2t-run -c mistral-7b-instruct-v0_3-htp.json -p '<s>[INST] What is the most popular cookie in the world? [/INST]'
Using libGenie.so version 1.14.0

/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO]  "Using create From Binary"
[INFO]  "Allocated total size = 281051648 across 8 buffers"
[PROMPT]: <s>[INST] What is the most popular cookie in the world? [/INST]

[BEGIN]:  The most popular cookie in the world is the chocolate chip cookie, which is a type of cookie that originates from the United States. It is a small, round-shaped, and semisweet chocolate-flavored cookie that is often enjoyed for its rich, creamy, and indulgent taste. The cookie is often served as a sweet, crunchy, and luscious treat, and is often adored for its delightful, delectable, and scrumptious texture.

The chocolate chip cookie is a favorite dessert and confectionery delight that is often relished, Savoried, and Adored for its Melt-in-the-Mouth, Melt-in-the-Mind, and Melt-in-the-Moment. It is a Creamy, Creamy, and Dreamy Delight, and is often Savorized, Savorized, and Savored for its Dreamy, Delighting, and Delightful.
, its Delicious, Delectable, and Delightful.

The chocolate chip cookie is a popular, Beloved, and Adored Confectionery Delight, and is often Adored, Adored, and Adored for its Delectable, and Delightful.[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc

Performance Reference

You can enable performance profiling with the --profile option.

genie-t2t-run -c phi-2-htp.json --prompt_file chat.txt --profile profile.txt

Fogwise® AIRbox Q900
GenieDialog_create	871,031 us
num-prompt-tokens	19
prompt-processing-rate	208.463623046875 toks/sec
time-to-first-token	91,145 us
num-generated-tokens	135
token-generation-rate	20.067829132080078 toks/sec
token-generation-time	6,727,238 us
GenieDialog_free	99,989 us

Metric Definitions

Metric	Definition
GenieDialog_create	Time to initialize a dialog object, including model loading, context preparation, and memory allocation.
num-prompt-tokens	Number of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into).
prompt-processing-rate	Speed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation.
time-to-first-token	Time elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency.
num-generated-tokens	Number of tokens actually output by the model in this generation, representing the length of the generated text in tokens.
token-generation-rate	Speed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency.
token-generation-time	Total time spent generating all output tokens, in microseconds (us).
GenieDialog_free	Time to free the dialog object, including memory release and resource cleanup.

Official Genie Documentation

For more details on Qualcomm® Genie usage and API, refer to:

• Genie Official Documentation