Artificial Analysis Launches AA-AgentPerf: Production Hardware Benchmark for the Agent Era
Artificial Analysis published its first results from AA-AgentPerf on June 13, a benchmark designed specifically for AI providers running multi-agent deployments at scale. The core premise: peak throughput numbers are straightforward to inflate; what actually matters for production is how many agents a system can sustain simultaneously while preserving the speed and latency that users experience.
The benchmark is live and open for submissions.
What It Measures
AA-AgentPerf runs real coding-agent trajectories — not synthetic prompts — and fixes service-level targets derived from Artificial Analysis’s serverless API benchmarking data. Two metrics define each tier: P25 output speed (tokens per second) and P95 time to first token (TTFT). The benchmark then finds the maximum concurrency a system can sustain without violating those targets.
The discovery method is an exponential ramp followed by binary search: double the agent load until a target breaks, then converge on the precise maximum concurrency. Metrics are computed over steady-state windows where all agents have been active for at least 30 seconds.
Tier Structure
DeepSeek V4 Pro (max) is the first provider with published results, across three tiers:
| Tier | Output Speed | P95 TTFT |
|---|---|---|
| Tier 1 | 20 tokens/s | ≤ 10s |
| Tier 2 | 60 tokens/s | ≤ 5s |
| Tier 3 | 180 tokens/s | ≤ 3s |
GPT-OSS-120B (high) is listed as coming soon, with a substantially different tier structure:
| Tier | Output Speed | P95 TTFT |
|---|---|---|
| Tier 1 | 100 tokens/s | ≤ 5s |
| Tier 2 | 250 tokens/s | ≤ 3s |
| Tier 3 | 500 tokens/s | ≤ 2s |
| Tier 4 | 2,000 tokens/s | ≤ 1s |
The contrast is substantial. GPT-OSS-120B’s Tier 1 (100 tokens/s, TTFT ≤5s) is already faster than DeepSeek V4 Pro’s Tier 3 ceiling. A system that can serve 2,000 tokens per second with sub-1s TTFT can sustain far more parallel agents at acceptable quality than one constrained to 180 tokens/s with a 3s TTFT budget. The number of concurrent agents an operator can run before degrading user experience scales roughly linearly with throughput at a fixed latency tier.
Why This Benchmark Matters
Existing inference benchmarks focus on latency or throughput in isolation — time to first token, sustained output speed, or maximum tokens per second at maximum concurrency. None of those metrics tell an operator how many agents they can actually run at production quality simultaneously.
AA-AgentPerf reframes the question. Instead of asking “what is your peak throughput?” it asks “how many agents can your infrastructure serve while meeting the service level your users actually require?” The distinction matters because peak throughput is trivially achievable by queuing everything and letting each agent crawl — a pattern that looks good on paper but produces unusable systems.
Using real coding-agent trajectories (rather than single-shot prompts) also captures the bursty token consumption patterns of agentic workloads: tool calls, long context builds, re-reads of prior output. Those patterns don’t show up in standard benchmarks but they determine capacity in production.
Open for Submissions
The benchmark is live and accepting new providers. Artificial Analysis says the methodology will expand to additional model types beyond coding agents. The decision to anchor tier targets to existing market data means the tiers reflect actual service quality thresholds rather than arbitrary numbers — providers already operating in the Tier 3 range for DeepSeek V4 Pro will recognise those numbers as close to what production customers receive today.
DeepSeek V4 Pro’s published results give the first comparison anchor. GPT-OSS-120B’s pending results, when they arrive, will establish whether high-throughput American inference infrastructure can serve meaningfully more concurrent agents per dollar than current Chinese open-weight deployments.