Running Phi-3/Mistral 7B LLMs on Raspberry Pi 5: A Step-by-Step Guide

Vadim Bakhrenkov

12 min read

·

Sep 5, 2024

--

2

Listen

Share

Press enter or click to view image in full size

Recently, I was inspired by a video on the NVIDIA AI channel, showcasing the Gemma 2 language model from DeepMind, which has 2 billion parameters, running on a Jetson Orin Nano. Impressive stuff! This spurred me to experiment with my Raspberry Pi 5, where I chose to run the Phi-3 mini 4K Instruct, a 3.8 billion parameter LLM from Microsoft, using the ONNX Runtime GenAI framework. Despite its considerable size, this model is surprisingly nimble, delivering about 4 tokens per second. Moreover, this same setup is also capable of running Mistral 7B!

Quick video demonstration of how the Phi-3 mini 4K Instruct LLM performs real-time CPU inference on a Raspberry Pi 5, with the script executed on the device via SSH console:

Impressive, isn’t it? This is just one step closer to a future integrated with AI. The entire setup was configured using compiled ONNX Runtime libraries and a straightforward Python script, along with the open-source Phi-3 LLM, which I downloaded directly onto the Raspberry Pi 5 running the latest version of Ubuntu OS. I’m excited to share a detailed step-by-step guide in this article, along with some intriguing insights about the technology and the hardware involved.

Interesting insight: What we currently define as Large Language Models (LLMs) could be considered Small Language Models (SLMs) in future contexts. As technology progresses, what seems vast today might become the standard or even minimal tomorrow, illustrating the rapid pace of advancement in AI model development. This fluidity challenges our understanding and categorization of language models, constantly pushing the boundaries of what is considered large or small.

Although I tested this setup on Ubuntu, the instructions provided should also be applicable to Raspberry Pi OS. This guide aims to give you a clear idea of what’s possible with AI on less conventional hardware platforms and how to leverage these systems for powerful, efficient computing.

ONNX Runtime / ONNX Runtime GenAI

ONNX Runtime is a performance-built engine designed to execute machine learning models serialized in the ONNX (Open Neural Network Exchange) format. It is compatible with models trained on a variety of frameworks, including PyTorch and TensorFlow, which enhances its adaptability across different platforms.

Capable of utilizing both CPUs and GPUs, along with other hardware accelerators, ONNX Runtime’s flexibility makes it suitable for diverse computing environments — from personal computers to high-end servers.

The ONNX Runtime GenAI package is a refined version of ONNX Runtime, specifically optimized for generative AI models. It enhances performance for tasks like image generation and natural language processing by integrating specific hardware and software optimizations.

This guide will walk you through setting up the ONNX Runtime GenAI package from source on a Raspberry Pi 5, leveraging Ubuntu and the powerful Arm64 CPU for inference tasks.

Raspberry Pi 5 Hardware Overview

The Raspberry Pi 5 is equipped with a quad-core Arm Cortex-A76 processor and a 12-core VideoCore VII 800MHz GPU. The Cortex-A76 operates on the ARMv8-A architecture — a robust 64-bit platform also known as ‘AArch64.’ Renowned for its strong performance, this architecture is utilized across a spectrum of devices, from smartphones to sophisticated computing systems.

Python and Virtual Environment

Setting up a virtual environment is optional if you are already familiar with Python environments. Virtual environments are essential for project context switching, allowing you to isolate software dependencies across different projects on the same machine.

Step 1: Verify Python Installation

Start by checking if Python is installed on your system:

python3 -V

You should see output similar to Python 3.12.3.

Step 2: Create and Activate a Virtual Environment

Create a new virtual environment in your preferred directory:

python3 -m venv onnxruntime-genai-env

Activate it:

source onnxruntime-genai-env/bin/activate

Install the Virtual Environment Tool (if necessary)

If your system does not already have the virtual environment tool installed, set it up with the following command:

sudo apt-get install -y python3-venv

Step 3: Update pip

Ensure that your pip installation is up to date to avoid any compatibility issues:

pip install --upgrade pip

By following these steps, you will establish a dedicated environment tailored for working with the ONNX Runtime GenAI package, effectively isolating it from other projects and dependencies.

Install Build and Development Tools

Before you begin, ensure that your system has the necessary building packages installed. These packages, which include compilers and libraries essential for compiling software from source, may not have been previously installed on your system. Install them with the following command:

sudo apt-get install -y build-essential cmake

Next, install the development utilities needed for this guide. These tools are crucial for downloading sources and managing code:

sudo apt-get install -y curl wget tar gzip git

Executing these commands equips your system with the foundational tools required for building and managing the ONNX Runtime GenAI package. This setup ensures all prerequisites are met for a successful installation. These tools will also be invaluable for various future projects, not just this build.

Get ONNX Runtime GenAI

1. Clone the Repository

Start by cloning the ONNX Runtime GenAI project files from the official GitHub repository:

git clone https://github.com/microsoft/onnxruntime-genai.git onnxruntime-genai

Once cloned, navigate to the onnxruntime-genai directory:

cd onnxruntime-genai

2. Checkout the Specific Version

To work with the specific version required for your project, switch to version v0.3.0:

git checkout tags/v0.3.0

Note: At the time of writing, higher versions of ONNX Runtime GenAI require the nightly build of ONNX Runtime. This requirement adds complexity to the setup process. While still manageable, it may necessitate additional steps such as installing packages via NuGet or downloading pre-compiled versions from the Visual Studio servers.

3. Verify the Version

Ensure you have checked out the correct version by running:

git status

The output should confirm HEAD detached at v0.3.0.

Get ONNX Runtime

The ONNX Runtime is required to build the GenAI package. Follow these steps to download and set it up correctly. For more information, visit the ONNX Runtime GitHub releases page to access the latest version compatible with your architecture.

1. Fetch the Download Link

To obtain the download link for the version that matches your system, use the following command:

curl -s https://api.github.com/repos/microsoft/onnxruntime/releases | grep browser_download_url | grep 'aarch64' | head -n 1 | cut -d '"' -f 4

For instance, this command might output a URL such as:

• https://github.com/microsoft/onnxruntime/releases/download/v1.19.2/onnxruntime-linux-aarch64-1.19.2.tgz

This version has been tested and confirmed to work, hence it is recommended to use it to avoid compatibility issues.

2. Download the Release

Download the specified version using:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.2/onnxruntime-linux-aarch64-1.19.2.tgz

3. Extract and Setup the ONNX Runtime Files

Once downloaded, you need to extract the files and set them up within your onnxruntime-genai directory:

tar -xzvf onnxruntime-linux-aarch64–1.19.2.tgz
mv onnxruntime-linux-aarch64–1.19.2 ort

Make sure these commands are executed within the onnxruntime-genai directory to ensure correct pathing. The proper setup of the lib and include directories is crucial for the building process. Incorrect placement can result in build errors indicating that the ONNX Runtime library could not be found.

Building ONNX Runtime GenAI

Navigate to the onnxruntime-genai directory and begin the build process by executing:

sh build.sh --build_dir=build/Linux --config=RelWithDebInfo

Once the build completes, you will find the Python wheel package in the directory build/Linux/RelWithDebInfo/wheel. Further instructions will be provided in the next step.

Building may take some time, and progress will be displayed, giving you an estimate of the time remaining.

Handling potential issues (skip if no errors)

During the build, especially with pybind11, you might encounter errors related to missing Python headers such as: Imorted target “target” “pybind11::module” includes non-existent path or similar messages. This typically occurs if the system Python installation paths do not match the expected locations by the build scripts.

To resolve this, install the Python Development Version. Using the python-config tool will help specify the correct include directories needed for the build:

sudo apt-get install python3-dev
python3-config --includes

The command above should help resolve the issue and output the paths to the Python headers, which are needed for the build process. The output will look something like this:

-I/usr/include/python3.12 -I/usr/include/python3.12

Install the Built Package

Now that the package is built, it’s ready for installation. For CPU inference, you’ll first need to install NumPy:

pip3 install numpy

Next, install the ONNX Runtime GenAI wheel package from the build directory:

pip3 install build/Linux/RelWithDebInfo/wheel/onnxruntime_genai-0.3.0-cp312-cp312-linux_aarch64.whl

To verify if the installation was successful, use the following Python command:

python3 -c 'import onnxruntime_genai; print(onnxruntime_genai.Model.device_type)'

This command will output the device type that the ONNX Runtime GenAI model is using, which typically appears as:

<property object at 0xfff123abc456>

The specific output isn’t critical; what matters is that the onnxruntime_genai package has been successfully imported and is invokable.

Download Phi-3 ONNX Model

To ensure efficient inference from the outset, we’ll download the Phi-3 mini 4K instruct model in the ONNX format. This model is particularly suited for CPU inference and is optimized with CPU INT4 RTN block 32 and accuracy level 4, aligning perfectly with the requirements of this guide.

Model Background:

• Model Type: Phi-3-mini is a Transformer-based language model.
• Parameter Count: 3.8 billion.
• Training Data: High-quality, educationally valuable datasets, including NLP synthetic texts and chat data from both internal and external sources.
• Post-Training Optimization: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) for enhanced chatting capabilities, alignment, robustness, and safety.

Model Availability:

The model files are hosted on the Hugging Face portal (Phi-3 mini 4K Instruct ONNX).

We will specifically download the CPU-optimized files available here: Phi-3 CPU Files.

Method 1: huggingface-cli (Recommended method)

Using huggingface-cli for simplicity and completeness.

Install huggingface-cli:

pip install -U "huggingface_hub[cli]"

Download only the necessary data:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir ./Phi-3-mini-4k-instruct-onnx
# Download the license file
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include LICENSE --local-dir ./Phi-3-mini-4k-instruct-onnx

Alternatively, to download the entire repository:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx

Method 2: Git clone with LFS (Alternative method)

Install and configure Git Large File Storage (Git LFS) to handle large files:

sudo apt-get install git-lfs
git lfs install
# or alternatively, install via script
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash

Clone the repository:

git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx

Method 3: wget (Alternative method)

This method is less efficient but still available. It requires downloading each file individually. For example, it may look like this:

wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/blob/main/README.md

Mistral 7B ONNX Model (Optional)

For those interested in alternative options, the Mistral 7B model serves as a viable substitute for the Phi-3 model. This section will guide you on how to download and integrate the Mistral 7B model into your Raspberry Pi setup, with specific details on adjusting parameters in the inference script provided later in this guide.

Model Details:

• Optimization: The Mistral 7B Instruct v0.2 is in ONNX format, optimized for CPU performance with an INT4 RTN block 32, achieving accuracy level 4 for efficient CPU inference.
• Location: Explore the Mistral 7B model files on the Hugging Face portal at Mistral 7B model files and Mistral-7B CPU Files for the CPU-specific package.

Downloading the Model:

huggingface-cli download microsoft/mistral-7b-instruct-v0.2-ONNX --include onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4/* --local-dir ./mistral-7b-instruct-v0.2-ONNX

Performance: While the Mistral 7B ONNX model can be used interchangeably with the Phi-3 mini 4K model, it’s important to note that the Mistral 7B may exhibit slower performance compared to the Phi-3 mini 4K. The architectural differences in the Mistral model could impact real-time processing speeds and overall inference performance on the Raspberry Pi 5.

Inference

Included below is a simple script named onnxruntime-genai-test.py designed for streamlined chat conversations. This script encapsulates the minimum required steps for efficient inference.

Create file and Edit the script

1. Create the script file: touch onnxruntime-genai-test.py
2. Open the script: Open onnxruntime-genai-test.py with your preferred text editor.
3. Add the following content: Ensure the model path within the script accurately reflects where the model files were downloaded, adjusting as necessary for your setup.

# Compiled library
import onnxruntime_genai as engine

# Phi-3 mini 4K instruct model path
model_path = "Phi-3-mini-4k-instruct-onnx/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4"
# Mistral 7B instruct v0.2 model path
# model_path = "mistral-7b-instruct-v0.2-ONNX/onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4"

print(f"Loading {model_path} model\n")

model = engine.Model(f'{model_path}')
tokenizer = engine.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()

# Set the maximum number of tokens to generate
# The full example includes additional parameters for more complex configurations
search_options = {'max_length': 2048}

# Template used to format multi-turn conversations in the chat interface
chat_tpl = '<|user|>\n{input}<|end|>\n<|assistant|>'

print("Let's chat!\n")

# Multi-turn chat loop. Exit with Ctrl+C
while True:
    text = input("> User: ")
    if not text:
        print("Please, answer something")
        continue

    # Populate the chat template with user input
    prompt = f'{chat_tpl.format(input=text)}'

    input_tokens = tokenizer.encode(prompt)

    gen_params = engine.GeneratorParams(model)
    gen_params.set_search_options(**search_options)
    gen_params.input_ids = input_tokens
    generator = engine.Generator(model, gen_params)

    print("\n> Assistant: ", end='', flush=True)

    try:
        # Loop to generate and display answer tokens
        while not generator.is_done():
            generator.compute_logits()
            generator.generate_next_token()
            next_token = generator.get_next_tokens()[0]
            print(tokenizer_stream.decode(next_token), end='', flush=True)
        print('\n')
    except KeyboardInterrupt:
        print("\nCtrl+C pressed, break\n")

# Delete the generator
del generator

Script Details

This script implements a basic one-question-one-answer chat system, designed to showcase local AI capabilities using the ONNX Runtime GenAI framework.

This script is adapted from an example on the ONNX Runtime GenAI GitHub repository’s examples page. For more detailed information and guidance on adjusting inference parameters, refer to the original example.

Running the Script

To execute the script, run:

python3 onnxruntime-genai-test.py

Please be aware that the cold start may take some time as the LLM needs to initialize. This process involves loading the model, tokenizer, and necessary data into memory before inference can begin.

To stop or exit the script, use the Ctrl+C keyboard combination.

Enhancing Chat Features

The example of code above operates using the simple prompt template:

<|user|>\n{input}<|end|>\n<|assistant|>\n

For example:

<|user|>
Create a short poem about the ocean.<|end|>
<|assistant|>

Here’s a breakdown of the components:

• <|user|> and <|assistant|>: These tokens denote the roles within the conversation. <|user|> indicates that the text following it is the user's input, while <|assistant|> signals the beginning of the AI's response. This role labeling helps the model recognize who is speaking and adjust the responses accordingly.
• {input} and <|end|>: The {input} placeholder is where the actual user query or command is inserted. The <|end|> token signifies the end of the user's input, which helps the model understand where the input stops and where a response should begin.

The <|...|> tokens are specialized markers used to structure and segment text data during AI model training and in dialogue handling applications. These tokens assist the AI in differentiating between various segments of the dialogue, such as pinpointing when a user starts and stops speaking, or indicating when the AI should commence its response. This convention is specifically tailored to simplify the processing and interpretation of conversations within the model, facilitating easier management and enhancement of dialogue flows.

Multi-turn Conversation

For extended conversations that maintain context and memory, consider modifying the script based on the multi-turn conversation template:

<|user|>\n{input}<|end|>\n<|assistant|>\n{output}<|end|>\n

For example:

<|user|>
What's the future of artificial intelligence?<|end|>
<|assistant|>
The future of artificial intelligence is poised to be transformative across many sectors, including healthcare, transportation, and education. As AI continues to evolve, it is expected to become more integrated into daily life, enhancing personal and professional tasks through automation and advanced analytics.<|end|>

The main difference from the template above is the inclusion of the {output} token and data:

• {output}: This is where the AI's response or output is inserted, along with the <|end|> token. It represents what the AI has generated in response to the user's input, providing information, answering questions, or continuing the conversation.

If you’re interested in adapting the script to handle more complex interactions or maintain a chat history, let’s discuss further in the comments below.

Conclusion

Thank you for reading! Your feedback and support are crucial to authors like myself. If you found this guide helpful, please give it a clap and consider subscribing for more content like this.

This guide completes our journey through setting up on-device inference with the Raspberry Pi 5. If you’re interested in more projects with Raspberry Pi, check out my other guides:

• Easy Guide to Text-to-Speech on Raspberry Pi 5 Using Piper TTS

If you’re using different platforms, you might also find my guide for ARM-based Macs useful:

• Running Phi-3/Mistral 7B LLMs on a Silicon Mac locally: A Step-by-Step Guide

Let’s stay connected! For more insights and ongoing discussions, follow me on:

• LinkedIn
• Twitter

Your support makes these resources possible and fuels future content. Thanks again for your time and engagement!