OpenMobile-2 Scores 79.9 on AndroidWorld, Outperforms Gemini 3.1 Pro at 70.7 With a 27B Open Model
A team from Nanjing University, SenseTime, and four partner institutions has released OpenMobile-2, an open smartphone agent that outscores Google’s Gemini 3.1 Pro Preview on the AndroidWorld benchmark. The 27-billion-parameter model posted 79.9 on AndroidWorld against Gemini 3.1 Pro Preview’s 70.7. The paper appeared on arXiv on October 9, 2026, and drew broader coverage after The Decoder reported on it October 25.
The gap over the base model it was built on is notable. Qwen3.6-27B, the foundation, scored 67.1 on AndroidWorld. Post-training with the team’s mobile agent recipe added 12.8 points.
Benchmark Results
| Model | AndroidWorld | MobileWorld (GUI) | SPA-Bench L3 |
|---|---|---|---|
| OpenMobile-2-27B | 79.9 | 50.4 | 59.6 |
| OpenMobile-2-9B | 74.0 | 38.5 | 48.9 |
| Gemini 3.1 Pro Preview | 70.7 | 58.1 | 70.2* |
| GPT-5.6 Sol | — | 70.1 | 68.1* |
| Qwen3.6-27B (base) | 67.1 | 35.9 | 31.9* |
*Author-reproduced scores, not independently verified.
The AndroidWorld lead over Gemini does not carry to the harder benchmarks. On MobileWorld, OpenMobile-2-27B scores 50.4, 7.7 points behind Gemini’s 58.1 and 19.7 points behind GPT-5.6 Sol’s 70.1. On SPA-Bench L3, a real-device test, the 27B model posts 59.6 against Gemini’s 70.2 and GPT-5.6 Sol’s 68.1.
The 9B model trails across all three tests but remains competitive on AndroidWorld at 74.0, above Gemini’s 70.7.
What the Team Released
The training recipe is fully open. The team published:
- A playground of more than 110 apps
- 11,600 training trajectories covering real device interactions
- Over 2,000 verifiable tasks with ground-truth outcomes
- Two model checkpoints (27B and 9B)
The open recipe is significant because smartphone agent training data is expensive to collect and has no large-scale public equivalent. Prior open work on mobile agents has typically relied on synthetic trajectories or small curated sets.
Why AndroidWorld vs MobileWorld Diverge
AndroidWorld tests task completion across real Android apps. MobileWorld is a GUI interaction benchmark with a heavier emphasis on navigation precision and UI element identification. The OpenMobile-2 models were optimized against the real-device trajectory distribution, which aligns better with AndroidWorld’s task structure. The gap on MobileWorld suggests that the training data does not transfer cleanly to the GUI-click precision that closed models handle well.
SPA-Bench L3 is a real-device evaluation covering longer task horizons. The 10-point gap to Gemini on SPA-Bench indicates that multi-step coherence and recovery from errors remain harder for the 27B model than the benchmark headline suggests.
Context
The result lands in a competitive mobile agent market. Closed models from Google and OpenAI lead on the harder evaluations, but OpenMobile-2 is the first open model to cross 79% on AndroidWorld and to beat a frontier multimodal model on that benchmark. For teams building smartphone automation in regulated environments or on-device, the fully open training pipeline is the practical differentiator, not the AndroidWorld score.
None of the reported figures have been independently verified. The comparison scores for Gemini and GPT-5.6 Sol on MobileWorld and SPA-Bench were reproduced by the authors, not taken from the originating labs’ evaluations.