← Back to feed

Epoch AI and METR have published MirrorCode, a long-horizon coding benchmark designed to measure what bug-fix and feature-completion evals cannot: whether AI agents can rebuild entire programs from behavioral specification alone.

The context matters. Most agentic coding benchmarks cap per-task inference at $1 to $10, even when the underlying task would take a skilled engineer weeks. That spending ceiling systematically underestimates frontier capability. MirrorCode removes it.

What the Benchmark Tests

In a MirrorCode task, an agent receives a target program’s public interface and documentation. It must reimplement the entire program without access to the original source code, without internet access, and without ever seeing the held-out end-to-end test cases used to judge its output. Results must match the original exactly — including on tests the agent never observed during development.

The benchmark spans 25 programs across six domains: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. The regularly updated leaderboard runs the 15 medium and large programs in two implementation languages each, for 30 task instances per model, with a budget of 10 billion tokens and 7 days per attempt.

Where Models Stand

The strongest model evaluated scores 56% across the benchmark.

The flagship demonstration: Claude Opus 4.7 reimplemented gotree, a bioinformatics toolkit with approximately 16,000 lines of Go code and more than 40 commands. It completed the task in 14 hours at a cost of $251. Epoch AI estimates the same task would take a human engineer without AI assistance 2 to 17 weeks. The best-performing implementation passed 2,000 of 2,001 test cases — failing a single edge case in a niche date-annotation command.

At the other end of the scale: a single attempt on the largest MirrorCode program consumed 19 days of compute and cost $2,600. No existing benchmark accounts for this range. Capping inference at $10 would make the task unsolvable by design.

The Contamination Problem

MirrorCode reimplements open-source programs, which means frontier models have likely seen the original codebases in pretraining. The authors ran a memorization screen and found that models succeeded at programs where the screen showed no evidence of memorization, and failed on programs where contamination was likely. The paper concludes results were not dominated by memorization but cannot rule out partial contributions from training data exposure.

That caveat is honest and structurally important. Any benchmark using real-world software faces this tension. MirrorCode’s end-to-end, held-out test design at least makes shallow lookup attacks ineffective — a model cannot simply reproduce source code and pass, it must produce working behavior.

What It Measures That SWE-bench Does Not

SWE-bench Verified tests single-issue resolution on real GitHub repositories. Terminal-Bench tests multi-step shell tasks. MirrorCode tests something different: the ability to build from a blank slate toward a complex, behaviorally defined target, across days, without human feedback.

The authors frame this as measuring transformative potential directly. Their argument: once AI agents can reliably reimplement existing programs from behavioral specs, the logical extension is building novel programs from requirements. The benchmark is not designed to be solved soon — at 56% for the strongest model, there is significant headroom.

MirrorCode was co-developed with METR. The lead authors are Tom Adamczewski, David Owen, and David Rein, with infrastructure contributions from Rasmus Faber-Espensen. Code and 22 of the 25 target programs are open-source on GitHub; three are held out as a private test set. The paper (arXiv:2606.30182) was posted in late June and updated July 17.

Key Numbers

Metric	Value
Top score (full benchmark)	56%
Claude Opus 4.7 on gotree	14 hours, $251
Human equivalent (gotree)	2–17 weeks
Largest single attempt	19 days, $2,600
Leaderboard budget	10B tokens / 7 days per attempt
Target programs	25 (22 open-source, 3 private)
Leaderboard tasks (ML, 2L)	30 instances