← Back to feed

Endor Labs released the Agent Security League on April 15, a public leaderboard that measures AI coding agents on two independent axes: whether the code works, and whether it does so without introducing exploitable vulnerabilities. The results expose a gap that benchmark tables built around SWE-bench Verified do not capture.

The Numbers

The benchmark extends Carnegie Mellon’s SusVibes framework — 200 real-world tasks drawn from 108 open-source projects, covering 77 Common Weakness Enumeration (CWE) vulnerability classes. Test harnesses run against leading agents including Cursor, Codex, and Claude Code.

Key findings from the initial run:

• 87% of AI-generated code contains at least one security vulnerability
• OpenAI Codex with GPT-5.4 scored highest on security correctness: 17.3%
• Cursor with Claude Opus 4.6 scored highest on functional correctness: 84.4% — but produced secure code only 7.8% of the time
• The gap between functional and security scores holds across every agent-model combination tested

Why This Is Structurally Distinct From SWE-Bench

SWE-bench Verified tests whether an agent can resolve a GitHub issue. A resolved issue means the test suite passes. It does not measure whether the fix introduces a buffer overflow, an injection point, or a privilege escalation vector. An agent that scores 87.6% on SWE-bench Verified and 7.8% on security correctness is not contradicting itself — those two numbers measure entirely different things.

The Agent Security League is the first benchmark to run both measurements simultaneously on the same tasks, making the functional-security gap visible and comparable across systems.

What The Security Failures Look Like

The 77 CWE classes cover injection flaws, memory safety issues, improper input validation, race conditions, and path traversal among others. The benchmark was built on real code from real repositories, not synthetic exercises, which means the vulnerability patterns reflect what models actually produce when solving production engineering tasks.

Agents that top the functional leaderboard are generating code that passes CI/CD and ships. The security benchmark is measuring what happens after that code reaches production.

The Market Context

AI-generated code is running in production at scale. GitHub Copilot recently reported that 40% of new code in repositories using its service is AI-assisted. Cursor reports similar figures for its user base. If the 87% vulnerability rate generalises across production deployments — a reasonable assumption given the diversity of the tasks in the benchmark — the security debt accumulating in codebases using AI agents is larger than most engineering teams are measuring.

The Agent Security League will update continuously as new agents and models are released. Current results are at endorlabs.com/research/ai-code-security-benchmark.