← Back to feed

Artificial Analysis released Optima on August 13, a platform that lets teams construct benchmarks from their own workloads and run them against leading models in a single pass. The company built its reputation on independent LLM evaluation infrastructure — GDPval-AA, AA-Briefcase, the AA Intelligence Index, CursorBench — and is now packaging that methodology as a self-service product.

The core problem Optima addresses is familiar to anyone who has tried to use published benchmark scores to pick a model: aggregate rankings measure what labs optimise for, not what your pipeline actually does. An Epoch AI analysis cited by The Decoder found that implementation details rarely disclosed in benchmark reports — prompt wording, temperature settings, scaffold choice — can shift the same model’s score by meaningful amounts. On agentic benchmarks like SWE-bench, swapping the agent scaffold alone accounted for up to 15 percentage points.

How It Works

There are three paths into Optima. Teams can upload an existing evaluation dataset from their own files or from Hugging Face. They can import agent traces from observability platforms — Arize, Braintrust, and Langfuse are supported at launch. Or they can describe their use case and provide sample inputs and outputs, and Optima generates suggested tasks and rubric criteria from that description.

Once the benchmark is defined, it runs across selected frontier models simultaneously under identical conditions. Results are reported across three axes: quality score, cost per task, and time per task. The cost-per-task column is the one that matters most in production decisions — it converts quality comparisons into spend comparisons on your specific workload rather than on AA’s representative task distribution.

Grading uses two methods from AA’s existing benchmark work. Rubric grading scores against objective criteria ($0.125 per criterion per model run). Pairwise grading uses the same judge panel that powers AA-Briefcase, where you select preferred responses from a sample and Optima extends those preferences across the full test set ($0.375 per match).

Benchmarks Are Only As Good As Their Design

Optima does not solve the underlying methodology problem — it shifts the responsibility. A custom benchmark built from a biased sample of traces, or with rubric criteria that reward style over correctness, will produce misleading results just as a public benchmark can. The Epoch AI analysis also found that the choice of judge model moves scores substantially; using Optima’s pairwise grading anchors the judge to a fixed panel, which at least makes the evaluation consistent across runs.

What Optima does do is give teams working evidence for their specific case rather than forcing a mapping from GDPval-AA or AA-Briefcase to a domain those benchmarks weren’t designed for. For workloads involving structured document processing, niche legal workflows, or domain-specific code generation, that translation has always been unreliable.

Optima is available now at artificialanalysis.ai/optima. Pricing is raw token cost of the models tested with no markup, plus the per-criterion and per-match grading fees.