ViBench 2.0: GPT-6.1 Sol Completes 65% of Full Product Builds, No Open-Weight Model Finishes Any
Replit shipped ViBench 2.0 on October 28, 2026: a benchmark that asks AI coding agents to build entire software products from an empty repository, then continue adding features to their own code.
The measure is strict. A build passes only if every browser test plan passes: not 90%, all of them. Seventeen products model real applications, including Uber, GitHub, and Google Docs. Each product has an average of 14 feature requests and 32 test plans. Four independent runs per product per model were executed by Fireworks AI at medium effort.
Results
| Model | Builds Fully Passing | Cost per Build |
|---|---|---|
| GPT-6.1 Sol | 65% | ~$12 |
| GPT-6 Astra | 60% | ~$90 |
| Claude Opus 5.5 | 49% | not stated |
| Open-weight models | 0% | N/A |
The top three each passed over 90% of individual test plans. The gap is not raw coding ability but build coherence: finishing a product with no broken feature across 32 test plans while continuing to extend code that accumulated across prior turns.
GPT-6.1 Sol’s cost advantage is stark. At $12 per build versus Astra’s $90, Sol completes 65% of builds at roughly one-seventh the price of a model that closes 5 percentage points fewer. That gap, already documented in LiveBench 2.0, DeepSWE, and OSWorld, widens here when the task is sustained multi-turn product development rather than isolated coding tasks.
No open-weight model completed a single full build. The open-vs-closed gap in isolated code evaluation has narrowed over 2026; ViBench’s multi-turn, full-product frame reintroduces a large margin that simpler coding benchmarks have been obscuring.
Design
ViBench 2.0 covers 241 feature requests and 544 browser test plans across its 17 products. Builds started from an empty project; the model had to initialize, implement, and iterate, accumulating context and prior code decisions across each feature addition. This is the key methodological shift from first-generation coding benchmarks: the model’s own prior outputs become inputs.
Runs were single-attempt (pass@1). A build counted only if all test plans in the final product passed. Individual feature pass rates were also tracked, giving a secondary view of where models break down, but the headline metric demands completion.