Skip to main content

RKLLM Installation

RKLLM Introduction​

RKLLM helps users quickly deploy LLM models onto Rockchip chips. Currently supported chips are: RK3588/RK3576/RK3562 series chips.

The overall framework of RKLLM is as follows:

rkllm_1.webp

Currently Supported Models​

RKLLM Installation​

To use RKNPU, users need to first run the RKLLM-Toolkit on an x86 workstation to convert trained models into RKLLM format, and then perform inference on the development board using the RKLLM C API.

x86 PC Workstation​

  • (Optional) Install Anaconda

    If Python 3.11 (required version) is not installed in your system or you have multiple Python environments, it is recommended to use Anaconda to create a new Python 3.11 environment.

    • Install Anaconda

      Execute the following command in the terminal window of your computer to check whether Anaconda is already installed. If yes, skip this section.

      X86 Linux PC
      $ conda --version
      conda 24.9.2

      If you see "conda: command not found", it means Anaconda is not installed. Please refer to the Anaconda official website for installation instructions.

    • Create a conda environment

      X86 Linux PC
      conda create -n rkllm python=3.11.11

    • Enter the rkllm conda environment

      X86 Linux PC
      conda activate rkllm

    • To exit the environment

      X86 Linux PC
      conda deactivate

  • Clone the RKLLM repository

    X86 Linux PC
    git clone -b release-v1.2.1b1 https://github.com/airockchip/rknn-llm.git   && cd rknn-llm

  • Install RKLLM-Toolkit

    RKLLM-Toolkit is a software development kit that allows users to quantize and convert Huggingface-format LLM models on X86 PCs.

    X86 Linux PC
    pip3 install ./rkllm-toolkit/rkllm_toolkit-1.2.1b1-cp311-cp311-linux_x86_64.whl

    If no errors occur when executing the following commands, the installation was successful.

    X86 Linux PC
    $python3
    >>>from rkllm.api import RKLLM

Development Board​

  • Check whether the RKNPU driver version is at least 0.9.8. If it is lower than this version, download and flash the latest radxa 6.1 firmware.

    tip

    The default RKNPU driver version in the radxa 6.1 firmware is 0.9.6. Please update to version 0.9.8 via: sudo rsetup -> System -> System Update. After the update, be sure to execute sudo apt autopurge and then reboot.

    Radxa OS
    $ sudo cat /sys/kernel/debug/rknpu/version
    RKNPU driver: v0.9.8

  • (Optional) Manually compile the NPU kernel

    If you are using a non-official firmware, you may need to update the kernel. The RKNPU driver supports two main kernel versions: kernel-5.10 and kernel-6.1. You can confirm the specific version number in the Makefile at the root directory of the kernel. The specific steps to update the kernel are as follows:

    1. Download the archive file rknpu_driver_0.9.8_20241009.tar.bz2.

    2. Extract the archive and replace the rknpu driver code in the current kernel source directory with it.

    3. Recompile the kernel.

    4. Flash the newly compiled kernel to the device.

  • RKLLM Runtime provides C/C++ programming interfaces for the Rockchip NPU platform, helping users deploy RKLLM models and accelerate the implementation of LLM applications. Clone the RKLLM repository on the device side.

    Radxa OS
    git clone -b release-v1.2.1b1 https://github.com/airockchip/rknn-llm.git