← Back to feed

The agent evaluation landscape has been fragmented since agentic AI took off: SWE-bench covers coding, tau-bench covers customer service tasks, Terminal-Bench covers shell and CLI work, and dozens of domain-specific evaluations have emerged without a common infrastructure. AgentBeats is an attempt to consolidate them.

The platform, developed under the Agentified Agent Assessment (AAA) paradigm, functions as a centralised registry where benchmarks can be “agentified” — made runnable against any participating agent via standardised interfaces — and where results are published automatically. Berkeley’s Responsible Decentralized Intelligence (RDI) lab, which has a track record in agent systems research, anchors the academic legitimacy of the effort.

The AgentX-AgentBeats Competition

The associated AgentX-AgentBeats competition runs under joint organisation by Berkeley RDI and the Agentic AI MOOC, with over $1M in prizes and resources backed by named AI sponsors. The competition structure provides external validation incentive for labs to submit their agents for standardised evaluation — addressing one of the long-standing problems in agent benchmarking: developers self-select which benchmarks they run and report.

13 Agent Domains

The registry spans:

• Coding — software engineering tasks
• Web — browser-use and web navigation
• Computer Use — desktop and OS-level tasks
• Research — multi-step information synthesis
• Game — strategic and planning tasks
• DeFi — on-chain and financial protocol interaction
• Cybersecurity — offensive and defensive security tasks
• Healthcare — clinical and medical decision support
• Finance — financial analysis and transaction tasks
• Legal — legal reasoning and document tasks
• Agent Safety — evaluating boundary adherence and safe operation
• Multi-agent — coordination and communication between agents
• Other — catch-all for emerging task types

The category breadth is notable. Most agent evaluation infrastructure to date has focused on coding (SWE-bench, Terminal-Bench, CursorBench) because software task correctness is objectively measurable. Including healthcare, legal, and agent safety alongside DeFi puts AgentBeats in the position of having to establish grading methodologies for domains where ground truth is harder to define.

AAA Paradigm and tau2-bench Integration

The AAA paradigm (Agentified Agent Assessment) is the theoretical foundation. It defines how benchmarks should be adapted for consistent cross-agent evaluation — standardising how environments are set up, how agents interact with tasks, and how scores are surfaced into the registry. The platform already integrates tau2-bench natively (tau2-bench is the successor to tau-bench, which measures retail and telecom customer service task completion).

The platform provides centralised infrastructure for executing assessments at scale, real-time observability of agent runs, and automatic score publication. The goal is a shared source of truth — removing the incentive for labs to cherry-pick evaluations.

Why Now

Agent evaluation has become commercially significant. Claude Managed Agents ($0.08/hr), OpenAI Codex Pro, and Cursor 3 all compete on agent capability claims. The absence of a standardised cross-domain registry means every lab’s benchmark comparisons are produced under different conditions, with different scaffolds, and different definitions of what counts as a pass. A registry backed by a credible academic institution with competitive incentive structures is a credible attempt to change that.

Whether the 13-domain scope holds up under the practical difficulty of benchmark design for healthcare and legal — where evaluation quality matters as much as the scores — remains to be seen. The coding-centric benchmarks at the top of the registry are the most mature; the expansion into regulated domains will be the harder test.