ByteDance EdgeBench: Agent Performance Follows a Scaling Law Over 12-Hour Runs — Opus 4.8 Leads at 43.6
Most agent benchmarks measure a single attempt. You give an agent a problem, it either solves it or it doesn’t, you record the pass rate. ByteDance Seed’s EdgeBench, released July 2, runs the agent for twelve hours or more and measures how it improves through feedback.
The difference matters. A model that fails immediately but recovers through self-correction is far more valuable in production than one that scores well on a one-shot evaluation but stalls when its first approach fails. EdgeBench is the first large-scale attempt to benchmark that property at scale.
What EdgeBench Measures
134 tasks across six categories: scientific discovery and machine learning, systems and software engineering, combinatorial optimization, professional knowledge work, formal mathematics (Lean 4 and Coq theorem proving), and interactive simulation environments including NetHack and OpenTTD. Every task requires at least 12 continuous hours of agent operation. The evaluation harness is called SForge; 51 tasks and the framework are public, the full 134 available by request.
The headline statistic is the 12-hour leaderboard on the 51-task open subset:
| Model | EdgeBench Score (12hr) | Systems/SWE |
|---|---|---|
| Claude Opus 4.8 | 43.6 | 62.0 |
| GPT-5.5 | 42.7 | — |
| GPT-5.4 | 34.3 | — |
| GLM-5.1 | (trailing) | — |
| DeepSeek-V4-Pro | (trailing) | — |
The 0.9-point gap between Opus 4.8 and GPT-5.5 is within noise. What is notable is GPT-5.4’s position — a 9-point gap below GPT-5.5 suggests the model generation transition improved long-horizon resilience more than short-horizon benchmarks captured.
The Scaling Law Claim
The paper’s central claim is the one that will attract the most scrutiny: performance follows a log-sigmoid scaling law as a function of interaction time, fit at R²=0.998 across all 134 tasks. Behind that curve fit is approximately 38,000 hours of recorded agent-environment interaction — the data set needed to make this claim plausible rather than just large.
If the curve generalises, “how long should I let this agent run” becomes a quantitative question rather than an intuition. ByteDance reports that agent learning speed has roughly doubled every three months across model generations.
The honest caveats: this is a self-published benchmark from the same lab that wrote the paper. The full 134-task set is not yet independently replicable without emailing for access. The dataset card does not specify exactly how each competing model was scaffolded or how many seeds were run. Independent replication of the scaling claim — not just the leaderboard scores — is needed before treating R²=0.998 as a fact about AI agents rather than a fact about ByteDance’s evaluation setup.
Why Long-Horizon Benchmarks Matter Now
Every major benchmark in the current evaluation landscape measures one-shot or few-step performance. SWE-bench Verified tracks whether an agent can close a GitHub issue in a single attempt. Terminal-Bench measures execution accuracy inside a shell session. These benchmarks have driven an enormous amount of progress.
But the tasks enterprises actually deploy agents against — multi-week data analysis projects, ongoing software maintenance, research synthesis — require agents to operate across dozens of sessions, adapt based on partial failures, and hold planning context across long time horizons. EdgeBench is measuring the thing that will determine whether the current generation of frontier agents is economically viable at production scale, not just impressive in demos.
Claude Opus 4.8 leading on systems and software engineering (62.0) specifically matters because it means Anthropic’s current flagship is the most resilient model when given 12 hours and a real codebase. That is a different claim than topping SWE-bench Verified.