LSTM

This document describes how to run the LSTM example on the NPU.

info

Refer to Model Zoo Download for the example.

LSTM example directory structure:

$ tree ./
./
├── CMakeLists.txt
├── convert_model
│   └── dataset
├── main.cpp
├── model
│   ├── iter_0_c1_26_out0_1_1_128.tensor
│   ├── iter_0_c2_28_out0_1_1_128.tensor
│   ├── iter_0_h1_25_out0_1_1_128.tensor
│   ├── iter_0_h2_27_out0_1_1_128.tensor
│   ├── iter_0_input_24_out0_1_3_32_32.tensor
│   └── lstm_model_uint8_a733.nb
└── README.md

Model Conversion

Enter the container development environment first. See Create and Start Container in the Model Zoo download page.

info

Select the Docker image that matches the NPU:

• A733: ubuntu-npu:v2.0.10.2
• T527: ubuntu-npu:v1.8.13

Download the floating-point ONNX model from the Allwinner netdisk: http://netstorage.allwinnertech.com:5000/sharing/AmbdcCS7O

X86 Linux PC

docker exec -it model-zoo /bin/bash
cd /workspace/examples/lstm_model/convert_model/
./convert_model_env.sh
./pegasus_import.sh lstm_model
./pegasus_quantize.sh lstm_model uint8 1
export VIV_VX_ENABLE_WB_SHARE=1

VIV_VX_ENABLE_WB_SHARE=1 enables LSTM / RNN internal weight sharing and reduces the exported nb file size.

A733

X86 Linux PC

./pegasus_export_ovx_nbg.sh lstm_model uint8 a733

T527

X86 Linux PC

./pegasus_export_ovx_nbg.sh lstm_model uint8 t527

The exported model is stored in the ../model directory.

Build the Example

Then compile the example. Exit the container first, then run the commands below.

Configure the cross-compilation toolchain first.

info

Skip this step if you have already configured it in another example.

X86 Linux PC

cd ../../../0-toolchains/

Download the toolchain from this link, put it in 0-toolchains/, then run:

X86 Linux PC

tar -xvf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
cd ../examples/lstm_model/

A733

X86 Linux PC

../build_linux.sh -t a733 -s debian11

T527

X86 Linux PC

../build_linux.sh -t t527 -s debian11

Model Deployment

After compilation, the example will be installed in the install directory. You can use scp to transfer it to the board.

Configure NPU Driver

info

You can skip this step if you have already configured NPU driver in other examples.

Transfer the driver library to the board's lib directory via scp.

• A733 corresponds to the common/npuruntime/lib_linux_aarch64/A733 directory
• T527 corresponds to the common/npuruntime/lib_linux_aarch64/T527 directory

Then execute the following command to export to environment variables.

Radxa SBC

echo 'export LD_LIBRARY_PATH=$HOME/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

Run Example

After configuring the driver, you can run the example.

tip

For T527 platform, you need to first enable NPU by referring to the A5E's "Enable NPU on Board" documentation, then use the following command to grant the current user permission to use /dev/vipcore.

Radxa SBC

sudo chmod 777 /dev/vipcore

A733

Radxa SBC

cd lstm_demo_linux_a733/

Radxa SBC

chmod +x ./lstm_demo_a733
./lstm_demo_a733 -nb model/lstm_model_uint8_a733.nb -i model/iter_0_input_24_out0_1_3_32_32.tensor,model/iter_0_h1_25_out0_1_1_128.tensor,model/iter_0_c1_26_out0_1_1_128.tensor,model/iter_0_h2_27_out0_1_1_128.tensor,model/iter_0_c2_28_out0_1_1_128.tensor -st 2

The running result is as follows:

$ ./lstm_demo_a733 -nb model/lstm_model_uint8_a733.nb -i model/iter_0_input_24_out0_1_3_32_32.tensor,model/iter_0_h1_25_out0_1_1_128.tensor,model/iter_0_c1_26_out0_1_1_128.tensor,model/iter_0_h2_27_out0_1_1_128.tensor,model/iter_0_c2_28_out0_1_1_128.tensor -st 2
model_file=model/lstm_model_uint8_a733.nb
VIPLite driver software version 2.0.3.2-AW-2024-08-30
nbg name=model/lstm_model_uint8_a733.nb, size: 1068496.
create network 0: 2785 us.
prepare network: 742 us.
network: 0, loop count: 1
run time for this network 0: 1221 us.
destroy npu finished.
~NpuUint.

This performance data only calculates the time consumption of model inference. Unless otherwise specified, it does not include the time consumption of pre-processing and post-processing.

SoC	NPU	Model	Input Resolution	Network Creation Time	Network Preparation Time	Single Frame Inference Time	Post-processing Time	Total Time	Frame Rate
Allwinner A733	Vivante VIP9000	lstm_model	32×32	2.8 ms	0.7 ms	1.2 ms		4.7 ms	819.0 FPS

T527

Radxa SBC

cd lstm_demo_linux_t527/

Radxa SBC

chmod +x ./lstm_demo_t527
./lstm_demo_t527 -nb model/lstm_model_uint8_t527.nb -i model/iter_0_input_24_out0_1_3_32_32.tensor,model/iter_0_h1_25_out0_1_1_128.tensor,model/iter_0_c1_26_out0_1_1_128.tensor,model/iter_0_h2_27_out0_1_1_128.tensor,model/iter_0_c2_28_out0_1_1_128.tensor -st 2

The running result is as follows:

$ ./lstm_demo_t527 -nb model/lstm_model_uint8_t527.nb -i model/iter_0_input_24_out0_1_3_32_32.tensor,model/iter_0_h1_25_out0_1_1_128.tensor,model/iter_0_c1_26_out0_1_1_128.tensor,model/iter_0_h2_27_out0_1_1_128.tensor,model/iter_0_c2_28_out0_1_1_128.tensor -st 2
model_file=model/lstm_model_uint8_t527.nb
VIPLite driver software version 1.13.0.0-AW-2023-10-19
nbg name=model/lstm_model_uint8_t527.nb, size: 1864000.
create network 0: 2375 us.
prepare network: 235 us.
network: 0, loop count: 1
run time for this network 0: 790 us.
destroy npu finished.
~NpuUint.

This performance data only calculates the time consumption of model inference. Unless otherwise specified, it does not include the time consumption of pre-processing and post-processing.

SoC	NPU	Model	Input Resolution	Network Creation Time	Network Preparation Time	Single Frame Inference Time	Post-processing Time	Total Time	Frame Rate
Allwinner T527	Vivante VIP9000	lstm_model	32×32	2.4 ms	0.2 ms	0.8 ms		3.4 ms	1265.8 FPS