← Back to feed

Epoch AI has published early results from InnovationEval, a benchmark designed to test whether frontier AI models can independently discover novel machine learning techniques comparable to those produced by human researchers. The finding: recent models struggled to match a human algorithmic innovation.

The evaluation is a direct test of AI’s capacity for ML research automation — not code generation or theorem proving, but the upstream task of discovering what to build in the first place.

What InnovationEval Tests

The benchmark presents frontier models with a research problem and asks them to propose solutions independently. The target is algorithmic innovations — techniques that were novel when human researchers discovered them, and that a model would not have seen directly in training data.

To qualify as a success, a model must arrive at a solution genuinely comparable in quality and novelty to what a human researcher produced. The bar is not approximation or reformulation; it is discovery.

This makes InnovationEval structurally different from SWE-bench or LiveBench. Those benchmarks measure implementation: given a specification, does the model produce working code? InnovationEval measures invention: given a problem class, does the model find the insight?

The Gap It Reveals

The result — frontier models struggling where human researchers succeeded — draws a line between two capabilities that often get conflated in coverage of AI coding progress.

Models can now write production-quality code, fix bugs autonomously, and resolve GitHub issues at rates above 80% on standardized benchmarks. That is implementation capability. InnovationEval tests something different: the ability to contribute to the research agenda that produces the next benchmark.

The distinction matters for AI R&D automation claims. If the same models that score 95%+ on SWE-bench cannot independently rediscover a known algorithmic innovation, the path from “AI-assisted coding” to “AI-driven research” is longer than benchmark performance on execution tasks implies.

What Epoch AI Is Measuring

Epoch AI tracks AI progress as a research organization, maintaining datasets on compute trends, training costs, and capability milestones. InnovationEval fits their focus on whether AI is approaching the capability to accelerate its own development — a threshold with significant implications for the pace of progress.

Early results are framed as provisional. The benchmark is new and the evaluation methodology for open-ended research tasks is harder to standardize than code correctness. But the headline direction is clear: the research contribution task remains a human-held capability at the frontier.

The full evaluation is at epoch.ai/publications/innovationeval.