GLM-52 897 —
GPT-56SC 873 —
CL-OP5X 865 -0.9%
GROK-46H 865 -0.9%
GEM-37FH 865 -0.9%
GPT-56T 861 —
GLM-5 856 —
MUSE-SPK 841 —
QWEN-38X 824 -2.3%
GPT-6A 820 —
KIMI-K3X 810 -1%
CL-FAB5H 787 -0.9%
CL-OP5H 764 -0.9%
CL-OP46H 742 -0.9%
CL-OP47H 733 -1.1%
GEM-38FH 676 -1%
CL-OP47 585 -0.7%
INKL 531 —
CL-OP46 496 -0.2%
CL-OP48 490 -0.2%
GLM-52 897 —
GPT-56SC 873 —
CL-OP5X 865 -0.9%
GROK-46H 865 -0.9%
GEM-37FH 865 -0.9%
GPT-56T 861 —
GLM-5 856 —
MUSE-SPK 841 —
QWEN-38X 824 -2.3%
GPT-6A 820 —
KIMI-K3X 810 -1%
CL-FAB5H 787 -0.9%
CL-OP5H 764 -0.9%
CL-OP46H 742 -0.9%
CL-OP47H 733 -1.1%
GEM-38FH 676 -1%
CL-OP47 585 -0.7%
INKL 531 —
CL-OP46 496 -0.2%
CL-OP48 490 -0.2%
← Back to feed

Microsoft ThinkingBox: Best Agent Hits 65% Pass@1, Collapses to 25% Across 20 Consecutive Trials

Microsoft released ThinkingBox on August 19, an open-source sandbox framework that evaluates AI agents on actual database state changes rather than self-reported outcomes. The paired benchmark, ThinkingBox-Bench, surfaces a gap that per-task pass rates obscure: the best available agent succeeds 65.36% of the time on a first attempt, but only 25.25% of the time when required to succeed across all 20 trials of the same task.

Microsoft’s label for this: the discovery-reliability gap.

What ThinkingBox Measures

ThinkingBox-Bench covers 507 tasks across five business domains. Each task was run through 20 separate trials per model. Scoring uses two metrics:

  • Pass@1: Did the agent succeed on its first attempt?
  • Pass^20: Did the agent succeed on every single trial of that task?

The divergence between these numbers is the finding. A model that scores 65% pass@1 looks usable. A model that scores 25% pass^20 is not trustworthy for unsupervised deployment. It will randomly fail three-quarters of the time on tasks it has previously demonstrated it can solve.

Results

Microsoft tested 12 proprietary and open-weight models. The best-performing model achieved:

  • Pass@1: 65.36%
  • Pass^20: 25.25%

No individual model breakdown was published in the initial release. Microsoft’s framing treats the discovery-reliability gap itself as the primary finding, not which model performed best.

Why This Matters

Most agentic benchmarks report a single run per task. Pass@1 is informative for understanding ceiling performance, but it does not reflect what happens in production, where agents run the same classes of task repeatedly under varying conditions.

The 40-point collapse from pass@1 to pass^20 suggests that current agents are highly sensitive to conditions that are not visible in single-shot evaluation: minor prompt variation, state changes in the environment, different starting configurations, or stochastic model behavior. For enterprise deployments in the five business domains ThinkingBox targets, this translates to workflows that appear to work during evaluation and fail inconsistently in production.

ThinkingBox differs from existing agent benchmarks by grounding evaluation in verifiable database state. Rather than asking an agent whether it completed a task, the framework checks whether the database actually changed in the expected way. That approach eliminates a class of errors where agents correctly report success on tasks they have not actually completed.

The framework and benchmark are available at github.com/microsoft/thinkingbox. Microsoft appears to be positioning ThinkingBox as a candidate industry standard rather than an internal evaluation tool. The open-source release was accompanied by a detailed methodology post from Principal ML Engineer Liang-Chun Tsai, who called for other labs to adopt the pass^20 metric alongside pass@1 in published benchmark tables.