← Back to feed

Primer, a finance-focused AI agent system, topped the BigFinanceBench leaderboard at 79.1% final-answer accuracy. The model underneath — GPT-5.5 — scores 55.8% without the harness. The 23.3-point gap did not close when Primer’s evaluators ran the same benchmark against Claude Fable 5, Claude Opus 4.8, and GPT-5.6. The harness outperforms the next frontier model regardless of which model the harness runs.

What BigFinanceBench Measures

BigFinanceBench contains 928 analyst-grade tasks graded on working and final answer: splitting private equity distributions between investors and managers, rebuilding divisional figures after a reporting change, calculating liquidity while excluding a credit facility announced after the reference date. It is graded on method, not just the number.

GPT-5.5 leads all raw models at 55.8%. Inside Primer’s harness, the same model hits 79.1%.

Where the Gap Comes From

The 23.3-point improvement breaks into two components:

Retrieval: +16.3 points. Primer goes to the actual SEC filing rather than reasoning over whatever the benchmark question hands it. When a task asks about Q3 liquidity, the agent retrieves the relevant balance sheet line rather than working from the question’s partial context. The model gets the right number before it reasons.

Calculation: +9.3 points. Primer knows how a company defines its own metrics. It excludes a credit line announced six weeks after the quarter closed rather than counting it as liquidity — the kind of domain knowledge that prevents a correct computation from a wrong starting figure.

The Benchmark’s Own Answer Key Was Wrong 12% of the Time

Primer’s evaluators found errors in 12% of BigFinanceBench’s questions during verification. In some cases the benchmark marked a correct answer wrong because the reference answer itself contained finance or data errors. A model can reason correctly on those questions and still be graded as failing. That figure has been published in Primer’s methodology blog and has been submitted to the benchmark maintainers.

What Frontier Model Upgrades Did Not Fix

The gap is structural, not a gap that a better base model closes. When Primer’s team evaluated Fable 5 and GPT-5.6 against the benchmark, neither closed the distance to the harness score. A larger model still retrieves the same wrong context from the question text if it has no access to the filing. Domain-specific retrieval and calculation pipelines are doing the work that model scale alone cannot.

That finding has a practical implication for enterprise AI buyers: evaluating a frontier model on a financial benchmark without an agent harness does not predict deployed performance. The dominant variable is the scaffolding, not the model ranking.

Key Numbers

• BigFinanceBench score, raw GPT-5.5: 55.8%
• BigFinanceBench score, Primer + GPT-5.5: 79.1%
• Retrieval contribution: +16.3 points
• Calculation contribution: +9.3 points
• Answer key errors found: 12% of questions
• Benchmark tasks: 928 questions graded on working and final answer
• BigFinanceBench: the only finance eval listed on the OpenAI GPT-5.6 launch page