Gemma-2-2B-it

This document describes how to perform NPU hardware-accelerated inference of the Gemma-2-2B-it model on Qualcomm platforms using Qualcomm® Genie.

• Source model: Gemma

• Source model license: Gemma Terms of Use

Model Details

Model	Quantization	Context Length
Gemma-2-2B-it	W4A16	4096

Supported Devices

tip

Refer to the SoC Architecture Reference to find the DSP architecture of your device's SoC.

• This example supports Qualcomm platform SoCs with v73 DSP architecture.

  dsp_arch
  v73

• Supported devices

  Device	SoC	dsp_arch
  Fogwise® AIRbox Q900	QCS9075	v73

Download qcom-qairt Dependencies

QCS6490

Device

sudo apt install qcom-qnn-sdk-v68 qcom-genie-sdk-v68

QCS9075

Device

sudo apt install qcom-qnn-sdk-v73 qcom-genie-sdk-v73

Import Environment Variables

Device

export ADSP_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu

Download Model

tip

Please install the modelscope Python package in a Python virtual environment. For virtual environment usage, refer to Python Virtual Environment Usage

Device

pip3 install modelscope
modelscope download --model radxa/Gemma-2-2B-it-w4a16-4096-v73 --local_dir ./Gemma-2-2B-it-w4a16-4096-v73

Run Inference

Device

cd Gemma-2-2B-it-w4a16-4096-v73

Build Prompt

Prompts can be passed as a file or as a parameter.

prompt

<bos><start_of_turn>user\nIntroduce yourself.<end_of_turn>\n<start_of_turn>model

prompt_file

Device

vim chat.txt

<bos><start_of_turn>user\nIntroduce yourself.<end_of_turn>\n<start_of_turn>model

Run Inference

prompt

Device

genie-t2t-run -c gemma-2-2b-it-htp.json -p "<bos><start_of_turn>user\nIntroduce yourself.<end_of_turn>\n<start_of_turn>model"

prompt_file

Device

genie-t2t-run -c gemma-2-2b-it-htp.json --prompt_file chat.txt

(.venv) rock@radxa-airbox-q900:/mnt/ssd/qualcomm/Gemma-2-2B-it$ genie-t2t-run -c gemma-2-2b-it-htp.json -p "<bos><start_of_turn>user\nIntroduce yourself.<end_of_turn>\n<start_of_turn>model"
Using libGenie.so version 1.14.0

/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
[INFO]  "Using create From Binary"
[INFO]  "Allocated total size = 285344256 across 3 buffers"
[PROMPT]: <bos><start_of_turn>user\nIntroduce yourself.<end_of_turn>\n<start_of_turn>model

[BEGIN]:

Hello! 👋 I'm Gemma, an AI assistant created by the Gemma team. I'm here to help you with text-based tasks, answer your questions, and generate creative content.

Just keep in mind:

* I'm still under development, so I'm always learning.
* I'm a text-only model, so I can't process images or audio.
* I don't have access to real-time information or the internet.

I'm excited to see what we can create together! 😊
[END]
/prj/qct/webtech_scratch20/mlg_user_admin/qaisw_source_repo/rel/qairt-2.42.0/release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.5.5/ipc/fastrpc/rpcmem/src/rpcmem_android.c:42:dummy call to rpcmem_deinit, rpcmem APIs will be used from libxdsprpc

Performance Reference

You can enable performance profiling with the --profile option.

genie-t2t-run -c gemma-2-2b-it-htp.json --prompt_file chat.txt --profile profile.txt

Fogwise® AIRbox Q900
GenieDialog_create	2,022,259 us
num-prompt-tokens	14
prompt-processing-rate	160.25640869140625 toks/sec
time-to-first-token	87,368 us
num-generated-tokens	170
token-generation-rate	19.333011627197266 toks/sec
token-generation-time	8,793,398 us
GenieDialog_free	105,481 us

Metric Definitions

Metric	Definition
GenieDialog_create	Time to initialize a dialog object, including model loading, context preparation, and memory allocation.
num-prompt-tokens	Number of tokens in the prompt sent to the model (i.e., the smallest unit the input text is split into).
prompt-processing-rate	Speed at which the model processes the prompt, in tokens per second (toks/sec), reflecting the efficiency of prompt analysis and output preparation.
time-to-first-token	Time elapsed from the start of processing to the generation of the first output token, reflecting the model's response latency.
num-generated-tokens	Number of tokens actually output by the model in this generation, representing the length of the generated text in tokens.
token-generation-rate	Speed at which the model generates tokens, in tokens per second (toks/sec), reflecting generation efficiency.
token-generation-time	Total time spent generating all output tokens, in microseconds (us).
GenieDialog_free	Time to free the dialog object, including memory release and resource cleanup.

Official Genie Documentation

For more details on Qualcomm® Genie usage and API, refer to:

• Genie Official Documentation