← Back to feed

A new benchmark registry called AgentBeats is building infrastructure to evaluate AI agents across 13 task categories, extending well beyond the software-engineering focus of SWE-bench and Terminal-Bench. The platform at agentbeats.dev lists participating agents, assessment results, and benchmark metadata in a searchable registry, and accepts community contributions via a documented “agentify” workflow.

Thirteen Categories, Not One

The SWE-bench era of agent evaluation treated software engineering as the canonical hard task. AgentBeats frames the problem differently. Its 13 categories are:

Coding — Web — Computer Use — Research — Game — DeFi — Cybersecurity — Healthcare — Finance — Legal — Agent Safety — Multi-agent — Other

The inclusion of DeFi, Healthcare, Finance, Legal, and Agent Safety reflects how agent deployment has evolved in 2026. Enterprises running agents in legal document review or clinical decision support need benchmarks calibrated to those domains, not proxy scores from GitHub issue resolution. The coding category serves as the baseline; the remaining twelve represent the domains where agents are actually being bought.

Agent Safety as a standalone category is notable. It places alignment and boundary-testing on equal footing with capability benchmarks inside the same taxonomy, rather than treating safety evaluation as a separate process that runs on different infrastructure.

Open Contribution Model

AgentBeats supports external benchmark contributions through what it calls an “agentify” process — a step-by-step workflow for converting existing benchmarks into the platform’s evaluation format. Benchmark owners can publish their assessment infrastructure to the registry, making their results comparable across the agent landscape.

This positions AgentBeats as a coordination layer rather than a single-benchmark authority. Instead of requiring labs or evaluation teams to submit results to one central benchmark, it provides a standard format that any benchmark operator can adopt. The community Discord at discord.gg/uqZUta3MYa handles support for teams going through the contribution process.

The AgentX Competition

AgentBeats is also running the AgentX-AgentBeats competition, a competitive evaluation layer built on top of the registry. The structure suggests that benchmark participation is not just passive listing — agents can compete across categories, giving the registry a competitive dimension alongside its reference function.

Why Vertical Benchmarks Matter Now

The benchmarks that shaped the first wave of frontier model development — MMLU, HumanEval, SWE-bench — were designed when the primary question was whether models could perform domain-general reasoning and code generation. That question is mostly answered.

The current evaluation gap is vertical: teams deploying agents in legal, healthcare, or financial settings cannot directly translate a SWE-bench score into a reliability estimate for their workload. The gap has been visible in production deployments where models with strong aggregate scores fail on domain-specific tasks because the benchmarks were never calibrated to those tasks.

AgentBeats attempts to close that gap by building shared evaluation infrastructure rather than asking each vertical to develop its own assessment methodology from scratch. The 13-category taxonomy is a starting point, not a ceiling — the “Other” category and open contribution model are designed to absorb new task types as agent deployment continues to expand.