Running Language Models on Low-Power CPUs: A Surprising Success
Running Ollama on a 15W CPU sounded ridiculous until I got it working with decent results
Xda-developers
Image: Xda-developers
Using a low-power mini PC with a 15W Intel Core i5-10210U CPU, the author successfully runs local language models like Ollama. While performance is not on par with cloud-based systems, it demonstrates that budget-friendly setups can still yield decent results for specific tasks.
- 01The author transitioned from a high-power setup to a mini PC to reduce energy consumption, achieving satisfactory performance with local language models.
- 02The Intel Core i5-10210U, despite its limitations, allows for running models under 10 billion parameters without excessive swapping.
- 03The setup includes Proxmox for virtualization and Open WebUI for model management, showcasing a practical approach to local AI.
- 04The Qwen3 model achieved around 4 tokens per second, which is sufficient for non-time-sensitive tasks.
- 05While local models can be effective, they are not a substitute for the responsiveness of cloud-based AI solutions.
Advertisement
In-Article Ad
The article explores the feasibility of running local language models on a low-power mini PC equipped with a 15W Intel Core i5-10210U CPU. The author details their transition from a high-power setup to a more energy-efficient solution, highlighting the surprising effectiveness of the mini PC for specific tasks. Using Proxmox and Open WebUI, they successfully installed and configured models like Qwen3 and Qwen2.5coder. Despite the limitations of the hardware, the Intel Core i5-10210U managed to run models with up to 10 billion parameters without significant delays. The performance of Qwen3 was measured at approximately 4 tokens per second, which, while not ideal for daily use, was adequate for the author's needs. The article emphasizes that while local setups can be viable for certain applications, they lack the speed and efficiency of cloud-based AI, making them suitable primarily for users willing to wait for responses.
Advertisement
In-Article Ad
This approach can make AI technology more accessible to individuals and small businesses by reducing hardware costs.
Advertisement
In-Article Ad
Reader Poll
Would you consider using a low-power PC for AI tasks?
Connecting to poll...
Read the original article
Visit the source for the complete story.