GLM-52 897 —
GPT-56SC 873 —
CL-OP5X 865 -0.9%
GROK-46H 865 -0.9%
GEM-37FH 865 -0.9%
GPT-56T 861 —
GLM-5 856 —
MUSE-SPK 841 —
QWEN-38X 824 -2.3%
GPT-6A 820 —
KIMI-K3X 810 -1%
CL-FAB5H 787 -0.9%
CL-OP5H 764 -0.9%
CL-OP46H 742 -0.9%
CL-OP47H 733 -1.1%
GEM-38FH 676 -1%
CL-OP47 586 -0.5%
INKL 531 —
CL-OP46 497 —
CL-OP48 490 -0.2%
GLM-52 897 —
GPT-56SC 873 —
CL-OP5X 865 -0.9%
GROK-46H 865 -0.9%
GEM-37FH 865 -0.9%
GPT-56T 861 —
GLM-5 856 —
MUSE-SPK 841 —
QWEN-38X 824 -2.3%
GPT-6A 820 —
KIMI-K3X 810 -1%
CL-FAB5H 787 -0.9%
CL-OP5H 764 -0.9%
CL-OP46H 742 -0.9%
CL-OP47H 733 -1.1%
GEM-38FH 676 -1%
CL-OP47 586 -0.5%
INKL 531 —
CL-OP46 497 —
CL-OP48 490 -0.2%
← Back to feed

Needle2: A 14MB Agentic LLM That Runs at 800 Tokens Per Second on a Raspberry Pi 5

Cactus Compute has released Needle2, a 45-million-parameter language model compressed to 14 megabytes via CQ2-bit quantisation. It runs at over 800 tokens per second on a Raspberry Pi 5 in prefill mode and over 500 tokens per second in decode mode — no GPU, no cloud connectivity required.

The model is designed specifically for agentic tasks on constrained hardware: wearables, robots, smart home controllers, phones, and automotive systems. Weights are available on Hugging Face under Apache 2.0.

Scale in Context

14MB puts Needle2 in a category with almost no competition. The smallest frontier-tier open models — the sub-2B class — land around 1.5GB after quantisation. Gemma 4’s 2B edge model, which ships pre-installed on iPhones, weighs in at 1.5GB. Needle2 is roughly 100x smaller than that.

The parameter count — 45M — is closer to a BERT-class encoder than a modern instruction-following decoder. What makes Needle2 distinctive is not the intelligence ceiling but the reliability floor: it is designed to execute tool calls, follow multi-step instructions, and recover from failures in agentic loops, on hardware where even a 1GB model is impractical.

Benchmark Numbers

Cactus has not published head-to-head accuracy comparisons against larger edge models on standard benchmarks. The published performance metrics are throughput-focused:

  • 800+ tokens/second prefill on Raspberry Pi 5
  • 500+ tokens/second decode on Raspberry Pi 5

An interactive playground at cactuscompute.com lets developers test Needle2 across its target domains — wearables, robots, smart home, phones, and automotive — before downloading weights.

Technical Architecture

The 14MB footprint comes from CQ2-bit quantisation, a compression scheme that reduces weight precision below the 4-bit floor that most edge-model compression stops at. At full float precision, a 45M-parameter model would require approximately 180MB. The CQ2-bit approach cuts that to 14MB while preserving enough fidelity for the structured agentic tasks the model targets.

The model is not a stripped-down version of a larger checkpoint. It was trained for agentic reliability at its native 45M parameter scale.

The Edge AI Gap

Current deployable AI breaks into two tiers with a large gap between them: cloud-hosted frontier models (tens of billions of parameters, $0.15 to $75 per million tokens) and on-device small models that run locally but require a 1-8GB footprint. Phones and laptops can fit the second tier. Microcontrollers, wearables, and embedded robotics systems cannot.

Needle2 targets the gap. An always-on wearable that needs to parse voice commands and call local APIs cannot pull from the cloud every time and cannot store a gigabyte model in flash. 14MB is a realistic fit for that constraint.

How much useful agentic work a 45M-parameter model can do in production — without retrieval, without tool scaffolding, on genuinely complex multi-step tasks — remains the open question. The Cactus playground is the quickest path to an answer.