Senior SWE-Bench: Frontier Models Cap at 24% When Given Realistic Engineer Tasks
Snorkel AI launched Senior SWE-Bench on July 1 to expose a specific failure mode: frontier coding agents perform well when given ticket-style specs, and fall apart when asked to work like actual engineers.
The premise is straightforward. Instead of instructions like “add a validate_email function that checks RFC 5322 compliance and returns a boolean,” Senior SWE-Bench gives agents messages like a Slack thread from a product manager that mentions a user complaint, references two unrelated files, and ends with “can you look into this?” Tasks draw from real pull requests merged after February 2026, contributed by engineers with hundreds of commits in those repositories. Median instruction length is 31% of SWE-Bench Pro’s already-sparse guidance.
The benchmark has 100 tasks — 50 public, 50 private to block contamination. It introduces a validation agent that writes behavioral tests adapted to each solution, scores “tasteful solves” by combining runtime correctness with codebase-specific quality checks, and includes load-bearing practices that go unstated in instructions — conventions the repo maintainers would notice even if no test caught them.
Results
Claude Opus 4.8 leads at 24.0% tasteful solve rate (pass@1), burning an average of 117,000 tokens per task. GPT-5.5 trails on tasteful solves but achieves the highest raw correctness rate, using just 36,000 tokens at peak effort — three times less output to reach second place.
The divergence matters. Opus 4.8 aligns better with what repo maintainers actually shipped: it reads the codebase, finds abstractions worth preserving, and patches against the grain of the surrounding code. GPT-5.5 grinds through correctness but produces what the benchmark calls “bloated patches” — code that passes tests while ignoring design conventions that experienced engineers would spot immediately.
GPT-5.5 shines on feature tasks involving Python backends and TypeScript frontends, hitting tasteful rates above 50% in some categories. Opus 4.8 scores roughly 50% higher on taste than its predecessor, Opus 4.7.
The Failure Mode
More than 75% of attempts across both models fail senior-level correctness and taste standards combined. The most common failure: picking the wrong root cause in bug investigations. At least 12% of all trials across agents misdiagnosed the issue and wrote a patch that addressed a symptom rather than the cause. That number is likely higher — it only counts cases where the wrong diagnosis was clearly traceable in the trajectory.
The benchmark’s runtime investigation tasks are the hardest tier. They require agents to reason from user-reported symptoms alongside logs, profiling data, and reproduction steps — the kind of debugging context a senior engineer would interpret in five minutes and an agent with no muscle memory of the codebase gets wrong repeatedly.
Why This Benchmark Matters Now
Standard SWE-Bench Verified sits at 88.7% for GPT-5.5 and clamped-at-ceiling for Opus 4.8 and Fable 5. Senior SWE-Bench resets the ceiling to 24%. That is not a contradiction — it is a measurement artifact. Benchmarks with detailed specs measure whether agents can implement what’s described. Senior SWE-Bench measures whether they can figure out what to implement at all.
The gap between the two scores is the gap between current AI coding agents and senior software engineers. On Snorkel’s tasks, that gap is still three to four times larger than leaderboard numbers suggest.
Key Numbers
| Model | Tasteful Solve Rate | Avg Tokens / Task |
|---|---|---|
| Claude Opus 4.8 | 24.0% | 117,000 |
| GPT-5.5 | ~22% (est.) | 36,000 |
Senior SWE-Bench is open-source. The private 50-task split is held by Snorkel to maintain contamination resistance.