State-of-the-art AI model runs on tiny Raspberry Pi Zero computer – impractical, but it works

Enthusiast Binh Pham has created a USB device based on a Raspberry Pi Zero single-board computer, which locally runs a large artificial intelligence language model that generates literary texts.

Image source: youtube.com/@build_with_binh

The author developed the software part of the project using the llama.cpp library and the llamafile utility, which are designed to output large AI language models. This turned out to be a difficult task, since the Raspberry Pi Zero has only 512 MB of RAM and a processor with an outdated ARMv6 architecture, which prevented the project from compiling in its original version. To get around these limitations, the enthusiast had to convert the ARMv8-optimized instruction set to llama.cpp and remove all notations and optimization mechanisms intended for modern hardware.

Binh Pham placed the Raspberry Pi Zero single-board computer and an expansion board with a USB port for connecting to modern computers in a 3D-printed case. Due to the modest computing resources, it was necessary to limit the context to 64 tokens and use models containing from 15 million to 136 million parameters. The smallest one — Tiny15M — showed the highest speed among all the tested models: 223 ms per token. For the Lamini-T5-Flan-77M, this figure was 2.5 s per token, and for the SmolLM2-136M — 2.2 s per token.

At such a speed, the device can hardly be called practical, but the author of the project decided not to stop there. He considered that controlling AI through the command line interface is not convenient enough, and suggested a more convenient way. To send a request, the user needs to create an empty text file in the specified location, the name of which serves as a request to the model. Having found the file, the system sends a request to the AI ​​and writes its response to the contents of the same file.

With his project, Binh Pham decided to show what interaction with local AI models could be like in the future.