OpenAI's MultiLangBench: GPT-6 Astra Hits 87.4% Across 20 Languages, GPT-6.1 Sol Reaches 97.9% on COBOL Migration
OpenAI published results on October 19 for two new evaluations targeting low-resource programming languages: MultiLangBench and a COBOL migration test. Both address a practical gap that standard benchmarks like SWE-bench ignore — how models perform on languages with thin training corpora but substantial real-world footprint.
COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions globally. Fortran underpins weather forecasting and nuclear reactor safety analysis. These are not toy targets.
MultiLangBench: 20 Languages, One Task
MultiLangBench tests whether a model can build a SQLite-compatible database engine from scratch in each of 20 programming languages. It measures new software construction across a breadth of language paradigms, from mainstream to obscure.
| Model | Average Pass Rate | vs. GPT-5.6 Sol | COBOL | OCaml | Fortran | Visual Basic |
|---|---|---|---|---|---|---|
| GPT-5.6 Sol (baseline) | 26.8% | — | 10.9% | 25.6% | 21.4% | 27.5% |
| GPT-6 Luna (max effort) | 40.3% | +13.5 pp | 17.1% | 34.1% | 31.4% | 41.6% |
| GPT-6 Astra (max effort) | 87.4% | +60.6 pp | 72.1% | 85.8% | 83.5% | 89.4% |
GPT-6 Astra exceeds 83% in 19 of the 20 languages tested. COBOL remains the hardest at 72.1% — the only language where Astra does not clear that floor. The jump from GPT-5.6 Sol’s 26.8% to Astra’s 87.4% is not incremental; it crosses a practical threshold in most languages.
GPT-6 Luna offers a lower-cost path. At max reasoning effort it averages 40.3%, with improvements in every language over GPT-5.6 Sol. OpenAI estimates Luna’s API cost at $0.77 per sample versus $3.48 for GPT-5.6 Sol — approximately 78% cheaper. For teams where Astra’s cost per run is prohibitive, Luna shows meaningful gains in COBOL (17.1%) and Visual Basic (41.6%) that did not exist in the prior generation.
OpenAI is releasing the MultiLangBench evaluation setup and full test suite on GitHub.
COBOL Migration: GPT-6.1 Sol at 97.9%
The second evaluation tests whether a model can migrate existing COBOL programs to another language while preserving behavior — a harder task than code generation because correctness is defined by the original program’s semantics, not a pass/fail test suite.
| Model | Effort Level | Accuracy | Cost vs. GPT-5.6 Sol |
|---|---|---|---|
| GPT-5.6 Sol | Low | 5.4% | baseline |
| GPT-6.1 Sol | Low | 86.6% | -39% |
| GPT-6.1 Sol | High | 97.9% | — |
GPT-6.1 Sol at low reasoning effort reaches 86.6%, against GPT-5.6 Sol’s 5.4% at the same setting — a 16x improvement. At higher reasoning effort it reaches 97.9%, approaching saturation of the evaluation. OpenAI’s current pricing for GPT-6.1 Sol is $2/$10 per million input/output tokens, and the low-effort COBOL migration costs 39% less than GPT-5.6 Sol at the equivalent setting.
What Changes Practically
Legacy code modernization is a large, underserved professional market. Teams managing COBOL mainframes — banks, insurers, government agencies — have historically relied on specialized contractors or hand migration. At 97.9% behavioral fidelity on migration, GPT-6.1 Sol at high effort is at a level where human review becomes the bottleneck rather than the primary workflow.
The MultiLangBench construction task and the migration task test complementary skills. A model that can write a working database engine in Fortran can likely handle new feature development in those languages. A model that can migrate COBOL can handle modernization. Together they describe a model capable of full-lifecycle work in language categories that were effectively out of reach for AI tooling a year ago.
The evaluation setup and test suite going to GitHub means independent replication will follow quickly.