← Back to feed

Artificial Analysis launched a mobile phone AI benchmark on August 24, in partnership with Liquid AI. The benchmark covers 41 models tested on iPhone 17 Pro under standardised 4-bit-or-smaller quantisation, measuring both intelligence and inference speed in a single controlled environment.

What was tested

The benchmark has two sides. Intelligence is measured using five evaluations covering function calling, knowledge recall, and reasoning — chosen specifically for tasks phone-scale models are expected to handle. Inference is measured by Liquid AI: output speed in tokens per second, latency to first token, and memory footprint under realistic phone load.

Both sides use the same build — quantised to 4-bit or smaller, loaded from llama.cpp — so intelligence and inference numbers apply to identical model builds. A model qualifies for the benchmark only if it fits inside 8 GB of memory after quantisation, including KV cache at 8K context.

41 model builds were tested. 33 successfully ran on iPhone 17 Pro.

Launch results

At the 16K context limit (the primary results set, because 64K does not fit in mobile phone memory):

• Nanbeige4.2-3B: 63 intelligence score. iPhone 17 Pro: 21.4 seconds per standard prompt, 4.0 GB memory.
• LFM2.5-2.6B: 63 intelligence score. iPhone 17 Pro: 8.0 seconds per standard prompt, 2.3 GB memory.
• Ornith-1.0-9B: 62 intelligence score.
• Qwen3.5 9B (Reasoning): 61 intelligence score, but hits the 16K limit on 29% of benchmark tasks, which reduces its score.

LFM2.5-2.6B matches Nanbeige4.2-3B on intelligence while running 2.7x faster and using 43% less memory. That gap is large enough to change user-perceived performance in interactive applications.

At the 64K context limit (published as a supplementary result, not the primary leaderboard):

• Ling 3.0 Flash: 66
• LFM2.5-2.6B: 64
• Qwen3.5 9B (Reasoning): 64

The 64K context results show different relative rankings because models with strong long-context reasoning benefit from the higher limit, but Qwen3.5 9B (Reasoning)‘s memory demands mean 64K does not fit on a phone — making the 16K primary results the operationally relevant comparison.

Why this matters

On-device AI benchmarks have historically been fragmented: each lab tests its own models with proprietary harnesses, on different devices, with different precision levels. The AA and Liquid AI benchmark standardises quantisation (4-bit or smaller), device (iPhone 17 Pro), and build process (llama.cpp), so results are directly comparable across models.

The benchmark also exposes a common issue: many high-scoring reasoning models have context windows much larger than what a phone can hold. Qwen3.5 9B (Reasoning) hits the 16K ceiling on nearly a third of benchmark tasks, which lowers its practical score relative to what cloud-based evaluations would show.

Artificial Analysis says it plans to expand coverage with more devices beyond iPhone 17 Pro, additional inference frameworks beyond llama.cpp, and evaluations purpose-built for on-device use cases. Speech Arena, evaluating speech-to-speech models on real-world conversational tasks, was also announced alongside the benchmark launch.

Key numbers

• 41 model builds tested; 33 successfully ran on iPhone 17 Pro
• Quantisation: 4-bit or smaller
• Memory threshold: 8 GB including KV cache at 8K context
• Primary context limit: 16K (64K results published separately)
• LFM2.5-2.6B: 63 intelligence, 8.0s per prompt, 2.3 GB
• Nanbeige4.2-3B: 63 intelligence, 21.4s per prompt, 4.0 GB
• Ornith-1.0-9B: 62 intelligence
• Qwen3.5 9B (Reasoning): 61 intelligence (16K-limited on 29% of tasks)