Gemini 3.5 Flash's 1M-Token Context Window: 77% Retrieval Accuracy at 128K, 27% at Full Scale — and Google Skipped General Reasoning Benchmarks
Gemini 3.5 Flash shipped on May 19 at Google I/O. The launch positioned it as a mid-tier model that beats the previous Pro tier on agentic benchmarks while running at Flash speeds. The context window — marketed at one million tokens — was a centrepiece of the pitch.
Independent evaluation published between May 20 and May 22 tells a more complicated story on both fronts.
The Context Window Caveat
The MRCR v2 benchmark measures a model’s ability to retrieve specific information embedded in long documents — the practical test of whether a large context window is usable or just a capacity claim.
At 128K tokens, Gemini 3.5 Flash scores 77.3% on MRCR v2. That is below Gemini 3.1 Pro’s 84.9% at the same context length, meaning the previous flagship was already more reliable at long-range retrieval. At 1M tokens, Flash drops to 26.6%.
Gemini 3.1 Pro does not publish a 1M-token MRCR score because it does not support a 1M context window. The relevant comparison is this: a model with a 1M-token context window retrieves information at one-third the accuracy of a shorter-context predecessor at the 128K length where both models operate.
For teams building document analysis pipelines, regulatory compliance tools, or long-codebase agent workflows on the assumption that “1M context” means reliable full-document retrieval, this number matters. At practical 128K lengths, Flash performs adequately. Beyond 200-300K tokens, retrieval starts to fail in ways that are difficult to predict per-document.
What Google Did and Did Not Benchmark at Launch
Six benchmarks appeared in Google’s I/O presentation. Every one of them targets agentic or coding scenarios:
| Benchmark | Gemini 3.5 Flash | Notes |
|---|---|---|
| Terminal-Bench 2.1 | 76.2% | 3rd on independent leaderboard |
| MCP Atlas | 83.6% | Vendor-reported; leads Claude Opus 4.7 (79.1%) |
| Finance Agent v2 | 57.9% | Vendor-reported |
| GDPval-AA Elo | 1,656 | Agentic task quality score |
| SWE-Bench Pro | 55.1% | 3rd; Claude Opus 4.7 at 64.3%, GPT-5.5 at 58.6% |
| CharXiv Reasoning | 84.2% | Scientific chart reasoning |
Absent from the launch: MMLU, GPQA Diamond, MATH-500, or any competition mathematics benchmark. For Gemini 3.1 Pro, Google published a broader suite including consistent reasoning scores alongside agentic numbers. That pattern broke with 3.5 Flash.
Independent evaluators noted the gap within hours. On ARC-AGI-2 — abstract reasoning puzzles that are a clean proxy for generalisation — Gemini 3.5 Flash scores 72.1%, compared to GPT-5.5 at 84.6%. On Humanity’s Last Exam, Flash scores 40.2% against 46.9% for GPT-5.5. The benchmarks Google chose to publish were the ones where Flash leads or competes. The benchmarks it did not publish are where the reasoning gap to frontier models is most visible.
The Agentic Case Is Real
The performance omissions do not negate the genuine wins. Flash scores 83.6% on MCP Atlas, leading Claude Opus 4.7 by 4.5 points and GPT-5.5 by 8.3 points. That is the best public result on a benchmark that measures real multi-step tool-driven workflows — the use case that most enterprise agent stacks actually deploy. At 289 tokens per second and $1.50 per million input tokens (or $0.15 for cached input), the cost arithmetic for orchestration-heavy pipelines favours Flash over every frontier alternative.
The specific use case where Flash is the clear choice: agent chains running many sequential API calls per workflow, where throughput matters more than peak reasoning capability. High-volume document classification, batch structured extraction, RAG-heavy pipelines with lots of cached context. Speed stops being a bottleneck, and the $0.15/M cached input rate is roughly 10% of Claude Sonnet 4.6’s standard input price.
The SWE-Bench Pro Gap
On SWE-Bench Pro — 1,865 real-world code changes across four languages, each requiring at least 10 lines of modification — Flash scores 55.1%, ranking third. Claude Opus 4.7 leads at 64.3%. GPT-5.5 sits at 58.6%.
Nine points behind the current leader on the benchmark most representative of production software engineering work is not a rounding error. For teams running autonomous code modification agents on complex multi-file repositories, Flash is not the top choice.
Google confirmed at I/O that Gemini 3.5 Pro is in internal testing. If 3.5 Pro addresses the reasoning regression and the long-context retrieval accuracy, the full 3.5 family becomes the default tier for general use. Until then, Flash is an excellent tool for what it was designed for — and a misleading benchmark story for everything else.
Key Numbers
- MRCR v2 retrieval: 77.3% at 128K tokens → 26.6% at 1M tokens
- Gemini 3.1 Pro at 128K: 84.9% MRCR v2 (no 1M window)
- Terminal-Bench 2.1: 76.2% (3rd independently; GPT-5.5 leads at 78.2%)
- SWE-Bench Pro: 55.1% (3rd; Claude Opus 4.7 at 64.3%)
- MCP Atlas: 83.6% (leads Claude Opus 4.7 at 79.1% and GPT-5.5 at 75.3%)
- ARC-AGI-2: 72.1% (GPT-5.5: 84.6%)
- Humanity’s Last Exam: 40.2% (GPT-5.5: 46.9%, Claude Opus 4.7: 41.4%)
- Throughput: 289 tok/s (4x faster than Claude Opus 4.7 and GPT-5.5)
- Pricing: $1.50/M input, $9.00/M output, $0.15/M cached input
- Intelligence Index: 55 (5th globally; GPT-5.5 at 60, Claude Opus 4.7 and Gemini 3.1 Pro at 57)
- Gemini 3.5 Pro: confirmed in internal testing, expected June 2026