← Back to feed

Dex Horthy published the October 2026 SlopCodeBench results on October 15, running six frontier model-effort combinations across all 36 problems and 196 checkpoints in the full benchmark for the first time. The benchmark tests coding agents on programs with evolving requirements: a model writes an initial solution, receives new requirements, updates the code, and is scored on whether both new behavior and old behavior still pass.

The most notable finding: increased thinking effort did not reliably buy more correct code, and in two of three comparable pairs it reduced strict pass rates while multiplying cost.

Results Table

Campaign	Strict Passes	Strict Rate	Core Rate	Estimated Cost
GPT-6 Astra XHigh	32/196	16.33%	69.4%	$758.62
GPT-6 Sol Medium	30/196	15.31%	67.9%	$245.69
GPT-6 Astra Medium	29/196	14.80%	66.3%	$261.05
GPT-6.1 Sol Medium	29/196	14.80%	65.8%	$178.51
Opus 5.5 Medium	28/196	14.29%	74.0%	$193.47
GPT-5.6 Sol XHigh	28/196	14.29%	70.9%	$753.30
Fable 5.1 XHigh	26/196	13.27%	72.5%	$1,550.07
GPT-6 Sol XHigh	26/196	13.27%	68.4%	$517.32
Opus 5.5 XHigh	25/196	12.76%	71.9%	$817.04
GLM 5.3 XHigh	24/196	12.24%	66.3%	$365.13

Strict pass rate = every test category passes, including regression tests. Core pass rate = all tests in a nonempty Core category pass, without requiring regression to hold.

The xHigh Problem

The effort comparison reveals a consistent pattern across two of three tested pairs:

Model	Strict: Medium → XHigh	Core: Medium → XHigh	Cost Multiplier
Opus 5.5	28 → 25	145 → 141	4.22x
GPT-6 Sol	30 → 26	133 → 134	2.11x
GPT-6 Astra	29 → 32	130 → 136	2.91x

Astra is the only model where XHigh strictly improved results — gaining 3 strict passes at 2.91x the cost. For Opus 5.5, XHigh produced fewer correct checkpoints and fewer core passes at more than four times the price. GPT-6 Sol XHigh cost twice as much as Medium and returned 4 fewer strict passes.

Horthy’s framing is careful: one campaign per setting cannot isolate effort as the sole cause versus run-to-run variation. These are single runs. But the directional result across two of three pairs, combined with the cost differentials, argues against treating XHigh as a reliable upgrade on real codebases with evolving requirements.

Value Standout: GPT-6.1 Sol Medium

GPT-6.1 Sol Medium posted 29 strict passes — identical to Astra Medium — at $178.51 versus $261.05 for Astra Medium and $758.62 for Astra XHigh. The cheapest run in the table achieves the same strict count as a model costing 1.5x to 4.2x more. Fable 5.1 XHigh, the most expensive campaign at $1,550, produced 26 strict passes — 3 fewer than GPT-6.1 Sol Medium.

What SlopCodeBench Measures

SlopCodeBench differs from SWE-bench in that it does not give models a complete, static problem description. Requirements arrive incrementally, and later checkpoints test both the new behavior and whether old tests still pass — the regression condition that causes models with high core rates to still fail on strict. Opus 5.5 Medium has the highest core rate in the table (74%) but only 28 strict passes, because regression failures pull its strict count down despite generally correct new-feature implementation.

The benchmark was introduced in the research paper arXiv 2603.24755. This is the third public run Horthy has published; the first covered 3 problems, the second 6. This is the first full 36-problem run, making it the first directly comparable result against the official research findings.

For context on how far the benchmark ceiling sits from deployed software quality: Horthy notes that models left large amounts of scratch code, temp files, and generated copies in the codebase during each campaign. These were counted in the quality metrics rather than excluded. A strict pass on SlopCodeBench does not mean clean, production-ready code — it means tests passed, including regression. Getting to 100% would require near-perfect code discipline across 36 problems with shifting requirements. No model is close.