DeepSeek V4's Architecture Paper: 90% Less KV Cache at 1M Tokens, a New Residual Path, and Muon at Scale
The four existing DeepSeek V4 articles on this site covered arena ELO, SWE-bench scores, pricing, and the Apache 2.0 licence. The architecture paper changes the conversation. DeepSeek V4 is not just a better model — it is a different cost curve.
The Problem It Solves
Standard attention has a quadratic memory problem. At 1 million tokens, every layer must compare the current token against the entire prior context at full resolution. KV cache — the stored key-value pairs for that history — grows linearly with context length and quadratically with layer depth. For a 1.6 trillion-parameter model reading a million-token document, that becomes prohibitive. V3.2 (671B total / 37B active) already struggled with this. V4 Pro (1.6T total / 49B active) would have been worse under the old architecture.
Three Changes
Compressed-Block Attention (CBA). V4 replaces standard self-attention in many layers with a two-tier system. Nearby tokens are compared at full resolution. Older context is compressed into blocks, and the model attends to those compressed summaries instead of the raw token sequence. The further back in the context, the more aggressively the compression. Some layers use this compressed-block approach; others compress even more aggressively and skip the full-resolution nearby comparison entirely. The model no longer reads the whole past at full detail — it uses a layered memory system.
New residual path. At great depth and scale, residual connections — the skip-paths that carry information across transformer layers — become numerically unstable. V4 introduces a redesigned residual architecture intended to remain stable as parameter count and depth increase. The paper frames this as a prerequisite for the rest of the efficiency work: the attention and routing changes are only useful if training converges.
Muon optimizer at scale. Muon is a second-order-adjacent optimizer that has shown strong results at smaller scales but had not been validated at frontier model size. V4 applies it at 1.6T parameters. DeepSeek’s claim is that Muon handles the gradient dynamics from the new attention and routing changes better than AdamW would at this scale.
The Result
At a 1M-token context, DeepSeek V4 Pro uses approximately 27% of the single-token compute and 10% of the KV cache required by V3.2 under the same workload. That is not a 10% efficiency gain — it is an order-of-magnitude reduction in memory pressure for long-context inference.
On the Artificial Analysis Intelligence Index, V4 Pro scores 52 (up from 42 for V3.2), a 10-point gain. V4 Flash — at 284B total / 13B active — scores 47, above V3.2 and at Claude Sonnet 4.6 level despite being substantially smaller. On GDPval-AA, Artificial Analysis’s agentic real-world tasks benchmark, V4 Pro (Max) scores 1554, leading all open-weight models ahead of GLM-5.1 (1535), MiniMax-M2.7 (1514), and Kimi K2.6 (1484).
Why It Matters
Open-weight long-context inference has been constrained by hardware economics. A 1M-token call on a model the size of V3.2 requires significant GPU memory and compute time. V4’s architecture paper proposes that the constraint is not just hardware — it is attention design. If 90% of the KV cache at 1M tokens is waste that can be removed without proportional quality loss, the economics of deploying large open-weight models at long context change materially.
The paper is also the first large-scale validation of Muon at frontier model size. That has implications beyond DeepSeek: if Muon proves stable at 1.6T, expect other labs to evaluate it for future training runs.
Key Numbers
| Metric | V3.2 | V4 Pro (Max) | V4 Flash (Max) |
|---|---|---|---|
| Parameters (total/active) | 671B / 37B | 1.6T / 49B | 284B / 13B |
| Intelligence Index (AA) | 42 | 52 | 47 |
| KV cache at 1M tokens (vs V3.2) | 100% | 10% | — |
| Compute at 1M tokens (vs V3.2) | 100% | 27% | — |
| GDPval-AA Agentic | — | 1554 (#1 open) | 1388 |