Needle2: A 14MB Agentic LLM That Runs at 800 Tokens Per Second on a Raspberry Pi 5
Cactus Compute has released Needle2, a 45-million-parameter language model compressed to 14 megabytes via CQ2-bit quantisation. It runs at over 800 tokens per second on a Raspberry Pi 5 in prefill mode and over 500 tokens per second in decode mode — no GPU, no cloud connectivity required.
The model is designed specifically for agentic tasks on constrained hardware: wearables, robots, smart home controllers, phones, and automotive systems. Weights are available on Hugging Face under Apache 2.0.
Scale in Context
14MB puts Needle2 in a category with almost no competition. The smallest frontier-tier open models — the sub-2B class — land around 1.5GB after quantisation. Gemma 4’s 2B edge model, which ships pre-installed on iPhones, weighs in at 1.5GB. Needle2 is roughly 100x smaller than that.
The parameter count — 45M — is closer to a BERT-class encoder than a modern instruction-following decoder. What makes Needle2 distinctive is not the intelligence ceiling but the reliability floor: it is designed to execute tool calls, follow multi-step instructions, and recover from failures in agentic loops, on hardware where even a 1GB model is impractical.
Benchmark Numbers
Cactus has not published head-to-head accuracy comparisons against larger edge models on standard benchmarks. The published performance metrics are throughput-focused:
- 800+ tokens/second prefill on Raspberry Pi 5
- 500+ tokens/second decode on Raspberry Pi 5
An interactive playground at cactuscompute.com lets developers test Needle2 across its target domains — wearables, robots, smart home, phones, and automotive — before downloading weights.
Technical Architecture
The 14MB footprint comes from CQ2-bit quantisation, a compression scheme that reduces weight precision below the 4-bit floor that most edge-model compression stops at. At full float precision, a 45M-parameter model would require approximately 180MB. The CQ2-bit approach cuts that to 14MB while preserving enough fidelity for the structured agentic tasks the model targets.
The model is not a stripped-down version of a larger checkpoint. It was trained for agentic reliability at its native 45M parameter scale.
The Edge AI Gap
Current deployable AI breaks into two tiers with a large gap between them: cloud-hosted frontier models (tens of billions of parameters, $0.15 to $75 per million tokens) and on-device small models that run locally but require a 1-8GB footprint. Phones and laptops can fit the second tier. Microcontrollers, wearables, and embedded robotics systems cannot.
Needle2 targets the gap. An always-on wearable that needs to parse voice commands and call local APIs cannot pull from the cloud every time and cannot store a gigabyte model in flash. 14MB is a realistic fit for that constraint.
How much useful agentic work a 45M-parameter model can do in production — without retrieval, without tool scaffolding, on genuinely complex multi-step tasks — remains the open question. The Cactus playground is the quickest path to an answer.