← Back to feed

Artificial Analysis published AA-Robotics v0.1 on October 14, marking the first time the leaderboard has tracked frontier LLMs performing direct robotic manipulation rather than text, code, or image tasks. Five models were put in control of a 6-axis robot arm and scored against eight tabletop tasks designed to test control, spatial reasoning, and adaptability.

The Results

Model	AA-Robotics Score
GPT-6 Astra	69.4
GPT-6.1 Sol	64.6
Gemini 4 Argon	55.3
Claude Opus 5.5	48.8
Claude Fable 5.1	46.3

The scoring is 0-100 per task. Full completion earns 100; partial credit captures progress short of that.

GPT-6 Astra wins overall but with visible ceilings: it scored 100 on pressing the red button, placing the toy, and writing “hello world,” then fell to 0 on measuring a box. Every model scored 0 on the box-measuring task. “Tidy the blocks into the box” produced partial scores across the board, with no model clearing it cleanly.

Task-Level Breakdown

Task	Astra	Sol	Argon	Opus 5.5	Fable 5.1
Press red button	100	100	100	100	100
Place crocodile toy	100	100	100	100	100
Pour drink into cup	100	0	100	25	30
Tidy blocks into box	40	100	40	40	40
Shape matching challenge	44	100	0	38	100
Write hello world	100	100	45	45	0
Build the tallest tower	71	0	57	25	0
Measure the box	0	17	0	17	0

Sol scored 100 on tasks Astra only partially completed (tidy blocks, shape matching). Argon poured a drink but failed the shape challenge entirely. Fable completed shape matching but returned 0 on writing and tower building. No model’s ranking held consistently across all eight tasks.

Cost and Speed

GPT-6 Astra averaged 30 minutes and $16.36 per task. Artificial Analysis estimates models are “nearly two orders of magnitude” worse than human baselines on cost and speed combined. The benchmark is not testing whether LLMs can manipulate robots for production use — it is testing whether they can do it at all.

The gap to production remains large. A human completing comparable tabletop tasks requires neither 30 minutes nor $16 per attempt. The benchmark frames this as expected: the research question is whether frontier models can control robot arms for simple objectives, not whether they are ready to replace human operators.

What AA Is Building Toward

AA-Robotics v0.1 initially uses a coding agent harness, meaning models write and execute code that sends commands to the arm rather than operating in real-time closed loops. This makes scores more comparable to SWE-bench-style evaluations than to direct motor-control work. The team notes that frontier LLMs “have indeed recently become capable of directly manipulating robot arms for simple tasks” — a claim that would have been harder to make two years ago.

The benchmark will expand. Tasks, species of robot hardware, and real-time control loops are all signaled as future additions. This is version 0.1 in the true sense: the category is new enough that even defining what constitutes a fair benchmark requires iteration.

The ranking order here — Astra, Sol, Argon, Opus 5.5, Fable 5.1 — differs from how these models rank on pure language and coding benchmarks. Fable 5.1 leads LiveBench and FrontierCode but places last here. Sol places second despite trailing Opus 5.5 and Astra on many coding tasks. Spatial reasoning under physical constraints is drawing out a different ordering, which is the most useful signal this benchmark has produced so far.