Skip to main content

RKLLM Installation

Introduction to RKLLM​

RKLLM helps users deploy LLM models to Rockchip chips quickly. Currently, it supports the rk3588/rk3576 chips. The overall framework is shown below:

rkllm_1.webp

Currently Supported Models​

Install RKLLM​

To use RKNPU, users first need to run the RKLLM-Toolkit tool on an x86 workstation to convert the trained model to the RKLLM format, then use the RKLLM C API on the development board for inference.

x86 PC Workstation​

  • (Optional) Install Anaconda

    If Python 3.8 (required version) is not installed on your system or if you have multiple Python versions, it is recommended to use Anaconda to create a new Python 3.8 environment.

    • Install Anaconda

      To check if Anaconda is installed, run the following command in the terminal. If it is installed, you can skip this step.

      $ conda --version
      conda 23.10.0

      If you see conda: command not found, Anaconda is not installed. Please refer to the Anaconda website for installation instructions.

    • Create a conda environment

      conda create -n rkllm python=3.8.2

    • Activate the rkllm conda environment

      conda activate rkllm

    • Deactivate the environment

      conda deactivate

  • RKLLM-Toolkit is a software development package for converting and quantizing Huggingface format LLM models on a PC.

    git clone -b release-v1.1.4 https://github.com/airockchip/rknn-llm.git
    pip3 install ./rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl

    If the following command runs without errors, the installation is successful:

    python3
    from rkllm.api import RKLLM

Development Board​

  • Check if the NPU driver version is 0.9.6 or higher. If it is lower, download and flash the latest Radxa 6.1 firmware.

    tip

    The default NPU driver version of radxa 6.1 firmware is 0.9.6, please update the system to 0.9.8 driver by: sudo rsetup -> System -> System Update. Please be sure to run sudo apt autopurge after system upgrade and then reboot.

    $ sudo cat /sys/kernel/debug/rknpu/version
    RKNPU driver: v0.9.8

    (Optional) Manually compile the NPU kernel

    If you are using a non-official firmware, you need to update the kernel. The RKNPU driver package supports two major kernel versions: kernel-5.10 and kernel-6.1. You can confirm the exact version number in the Makefile in the root directory of the kernel. Follow these steps to update the kernel:

    1. Download the rknpu_driver_0.9.6_20240322.tar.bz2 archive.

    2. Extract the archive and overwrite the current kernel code directory with the rknpu driver code.

    3. Recompile the kernel.

    4. Flash the newly compiled kernel to the device.

  • RKLLM Runtime provides C/C++ programming interfaces for the Rockchip NPU platform to help users deploy RKLLM models and accelerate LLM applications.

    git clone -b release-v1.1.4 https://github.com/airockchip/rknn-llm.git