Skip to main content

Phi-3.5-mini-instruct

This document describes how to perform NPU hardware-accelerated inference of the Phi-3.5-mini-instruct model on Qualcomm platforms using Qualcomm® Genie.

Model Details​

ModelQuantizationContext Length
Phi-3.5-mini-instructW4A164096

Supported Devices​

tip

Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.

  • This example supports Qualcomm platform SoCs with v73 DSP architecture.

    dsp_arch
    v73

  • Supported devices

    DeviceSoCdsp_arch
    Fogwise® AIRbox Q900QCS9075v73

Download qcom-qairt Dependencies​

Device
sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73

Import Environment Variables​

Device
export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu

Download Model​

tip

Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage

Device
pip3 install modelscope
modelscope download --model radxa/Phi-3.5-mini-instruct-w4a16-4096-v73 --local_dir ./Phi-3.5-mini-instruct-w4a16-4096-v73

Run Inference​

Device
cd Phi-3.5-mini-instruct-w4a16-4096-v73

Build Prompt​

Prompts can be passed as a file or as a parameter.

<|system|>You are a helpful assistant.<|end|><|user|>How to explain Internet for a medieval knight?<|end|><|assistant|>

Run Inference​

Device
genie-t2t-run -c Phi-3.5-mini-instruct-htp.json -p '<|system|>You are a helpful assistant.<|end|><|user|>How to explain Internet for a medieval knight?<|end|><|assistant|>'
(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Phi-3.5-mini-instruct$ genie-t2t-run -c Phi-3.5-mini-instruct-htp.json -p '<|system|>You are a helpful assistant.<|end|><|user|>How to explain Internet for a medieval knight?<|end|><|assistant|>'
Using libGenie.so version 1.14.0

/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO]  "Using create From Binary"
[INFO]  "Allocated total size = 816947712 across 8 buffers"
[PROMPT]: <|system|>You are a helpful assistant.<|end|><|user|>How to explain Internet for a medieval knight?<|end|><|assistant|>

[BEGIN]:  To explain the Internet to a medieval knight, you would need to break down the concept into fundamental ideas and relate them to familiar medieval scenarios. Here's a step-by터 approach:

1. **Understanding the Concept of a "Network":**
   - **Metaphor:** Explain the Internet as a vast network of roads, similar to the intricate web of trade routes and alliances that knights might travel on. Just as knights can journey from one castle to another, the Internet allows people to travel from one place to another, not on horseback, but through a system of interconnected pathways.

2. **The Role of the "Knights of the Round Table" (Internet Service Providers):**
   - **Explanation:** Describe ISPs as the local lords or guild masters who provide access to the roads. They maintain the infrastructure (like roads) and ensure that travelers (users) can move from one place to another.

3. **The "Code of Chivalry" (Internet Protocols/Standards):**
   - **Illustration:** Just as knights follow a code of conduct, the Internet has its own set of rules that ensure communication and data exchange are orderly and efficient. These rules are known as protocols, which are agreed-upon methods for knights (devices and users) to interact safely.

4. **The "Tournament of the Field" (Data Exchange):**
   - **Analogy:** When a knight competes in a tournament, he aims to win or achieve a goal. Similarly, the Internet allows individuals to send and receive information (letters, messages, scrolls) to achieve their objectives.

5. **The "Four Postern Door" (Firewall):**
   - **Security Measure:** Explain that just as a castle has a gatekeeper to protect its inhabitants from invaders, the Internet has security measures (firewalls) to protect against malicious entities.

6. **The "Tale of Two Cities" (Internet Speed and Connectivity):**
   - **Variation:** Some castles (computers) have faster horses (faster Internet speeds) and more direct routes (better connectivity) than others. This difference can affect how quickly one can send messages or travel to distant lands.

7. **The "Crossbow" (Data Transmission):**
   - **Tool:** Describe how data is sent across the Internet using a metaphor such as a crossbow. The Internet is like a vast battlefield where crossbows (data packets) are launched from one knight's (user's) position to another, carrying messages or information.

8. **The "Alchemist's Potion" (Data Encryption):**
   - **Secrecy:** Just as a potion can be concocted to remain hidden from prying eyes, data on the Internet is often encrypted, ensuring that only those with the right key (password or decryption key) can read the information.

9. **The "Dragon's Lair" (Server Farms):**
   - **Central Hub:** Explain that there are central hubs, like a dragon's lair, where vast amounts of scrolls (data) are stored. These are called servers, and they hold the knowledge and resources that knights (users) can access when they travel the Internet.

10. **The "Mercantile Guilds" (Social Networks and Online Communities):**
    - **Social Interaction:** The Internet also serves as a marketplace and a gathering place for knights to exchange news, share tales of adventure, and forge alliances, much like the social networks and online communities of today.

By using these medieval metaphors and scenarios, you can help a medieval knight grasp the abstract and complex nature of the Internet in a context they can understand. Remember, the goal is to make the explanation relatable while maintaining the essence of how the Internet functions.[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc

Performance Reference​

You can enable performance profiling with the --profile option.

genie-t2t-run -c Phi-3.5-mini-instruct-htp.json --prompt_file chat.txt --profile profile.txt
Fogwise® AIRbox Q900
GenieDialog_create2,143,046 us
num-prompt-tokens21
prompt-processing-rate122.04051971435547 toks/sec
time-to-first-token172,091 us
num-generated-tokens901
token-generation-rate9.163215637207031 toks/sec
token-generation-time98,328,012 us
GenieDialog_free122,259 us

Metric Definitions​

MetricDefinition
GenieDialog_createTime to initialize a dialog object, including model loading, context preparation, and memory allocation.
num-prompt-tokensNumber of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into).
prompt-processing-rateSpeed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation.
time-to-first-tokenTime elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency.
num-generated-tokensNumber of tokens actually output by the model in this generation, representing the length of the generated text in tokens.
token-generation-rateSpeed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency.
token-generation-timeTotal time spent generating all output tokens, in microseconds (us).
GenieDialog_freeTime to free the dialog object, including memory release and resource cleanup.

Official Genie Documentation​

For more details on Qualcomm® Genie usage and API, refer to:

    You need to be logged into GitHub to post a comment. If you are already logged in, please ignore this message.

    Radxa-docs © 2026 by Radxa Computer (Shenzhen) Co.,Ltd. is licensed under CC BY 4.0