← Back to feed

Microsoft released ThinkingBox on August 19, an open-source sandbox framework that evaluates AI agents on actual database state changes rather than self-reported outcomes. The paired benchmark, ThinkingBox-Bench, surfaces a gap that per-task pass rates obscure: the best available agent succeeds 65.36% of the time on a first attempt, but only 25.25% of the time when required to succeed across all 20 trials of the same task.

Microsoft’s label for this: the discovery-reliability gap.

What ThinkingBox Measures

ThinkingBox-Bench covers 507 tasks across five business domains. Each task was run through 20 separate trials per model. Scoring uses two metrics:

• Pass@1: Did the agent succeed on its first attempt?
• Pass^20: Did the agent succeed on every single trial of that task?

The divergence between these numbers is the finding. A model that scores 65% pass@1 looks usable. A model that scores 25% pass^20 is not trustworthy for unsupervised deployment. It will randomly fail three-quarters of the time on tasks it has previously demonstrated it can solve.

Results

Microsoft tested 12 proprietary and open-weight models. The best-performing model achieved:

• Pass@1: 65.36%
• Pass^20: 25.25%

No individual model breakdown was published in the initial release. Microsoft’s framing treats the discovery-reliability gap itself as the primary finding, not which model performed best.

Why This Matters

Most agentic benchmarks report a single run per task. Pass@1 is informative for understanding ceiling performance, but it does not reflect what happens in production, where agents run the same classes of task repeatedly under varying conditions.

The 40-point collapse from pass@1 to pass^20 suggests that current agents are highly sensitive to conditions that are not visible in single-shot evaluation: minor prompt variation, state changes in the environment, different starting configurations, or stochastic model behavior. For enterprise deployments in the five business domains ThinkingBox targets, this translates to workflows that appear to work during evaluation and fail inconsistently in production.

ThinkingBox differs from existing agent benchmarks by grounding evaluation in verifiable database state. Rather than asking an agent whether it completed a task, the framework checks whether the database actually changed in the expected way. That approach eliminates a class of errors where agents correctly report success on tasks they have not actually completed.

The framework and benchmark are available at github.com/microsoft/thinkingbox. Microsoft appears to be positioning ThinkingBox as a candidate industry standard rather than an internal evaluation tool. The open-source release was accompanied by a detailed methodology post from Principal ML Engineer Liang-Chun Tsai, who called for other labs to adopt the pass^20 metric alongside pass@1 in published benchmark tables.