Databricks Benchmarked 10 Coding Agents on Its Own Multi-Million-Line Codebase — GLM 5.2 Ties Opus 4.8 at 34% Lower Cost
Databricks published the results of an internal coding agent benchmark built on actual pull requests from its own engineering teams — multi-million lines of code spanning Python, Go, TypeScript, Scala, Rust, Java, Bazel, and Protobuf. The exercise surfaced patterns that public leaderboards consistently obscure.
Three Tiers, One Surprise
Models clustered into three distinct capability tiers. The top tier — covering the most complex tasks — includes Claude Opus 4.8, GPT-5.5, and GLM 5.2. That last entry is the headline finding.
GLM 5.2, Z.ai’s open-weight 744B MoE model, was statistically tied with Opus 4.8 on task completion across all complexity levels. Its cost advantage is substantial: $1.28 per task versus $1.94 for Opus 4.8 — a 34% reduction with no measurable quality drop on Databricks’ workloads.
The middle tier handled common operational tasks efficiently. The bottom tier was cost-competitive on low-complexity work but fell apart on anything requiring deeper reasoning.
Token Price Is Not Task Price
The benchmark’s most practically useful finding: price per token is a poor proxy for price per task.
Sonnet 5 costs roughly 1.7x less per token than Opus 4.8. On Databricks’ production tasks, Sonnet 5 came out to $2.09 per task versus Opus 4.8’s $1.94 — and scored six points lower on task completion (81% vs 87%). The reason: Sonnet 5 consumed 1.9x more tokens to get there. It read more, iterated longer, and ultimately cost more despite the headline price advantage.
This inverts the common engineering intuition. Larger models with stronger reasoning can complete tasks in fewer, more targeted steps. Token efficiency at the model level often beats token price at the billing level.
The Harness Problem
The benchmark ran the same model with the same thinking effort through two different harnesses — Claude Code/Codex versus a simpler harness called Pi. Cost per task differed by more than 2x. Quality was identical.
Pi sent roughly 3x less context per turn. It managed working state tighter, finishing tasks in fewer round-trips. The lesson is not that simpler harnesses always win — it’s that context management is a primary cost driver independent of model choice. Databricks has been investing in Omnigent, an orchestration layer designed to make model-and-harness swapping seamless for exactly this reason.
Why Public Benchmarks Don’t Answer This
Databricks built its own benchmark because SWE-Bench and Terminal-Bench couldn’t. Both have contamination risk as their tasks enter training data. Neither reflects a 10-language enterprise codebase with Bazel build systems, Protobuf schemas, and deep service dependencies.
The internal dataset was constructed from recent engineer PRs, filtered for human-written commits with high-quality test suites. About 25% of tasks were tagged low-complexity; 60% were medium. Expensive frontier models were the default across the board — the benchmark found significant efficiency headroom in routing lower-complexity work to Haiku-class and GPT-5.4 Mini-class models.
Key Numbers
| Model | Cost/Task | Completion | Tier |
|---|---|---|---|
| GLM 5.2 | $1.28 | ~87% | 1 |
| Opus 4.8 | $1.94 | 87% | 1 |
| Sonnet 5 | $2.09 | 81% | 1 |
| GPT-5.5 | Top tier | Top tier | 1 |
The immediate practical outcome: Databricks is pushing more routine engineering work to Haiku-class models, and has started deploying GLM 5.2 as a daily driver for mid-complexity tasks given its frontier-tier quality at open-weight pricing.