Eagle 3.1 Fixes Speculative Decoding's Long-Context Collapse — 2x Acceptance Length, 2.03x Throughput
Speculative decoding has been one of the most effective latency-reduction techniques in production LLM serving — a small draft model proposes tokens that a larger target model verifies in parallel, cutting time-to-first-token and increasing throughput when drafts are accepted. The technique works well in controlled benchmarks. It degrades visibly in production: different chat templates, long-context inputs, unusual system prompts.
Eagle 3.1, released jointly by the EAGLE team, vLLM team, and TorchSpec team, is the first version that systematically addresses why.
The Root Cause: Attention Drift
The EAGLE team traced production instability to a phenomenon they call attention drift. As speculation depth increases — meaning the draft model generates more tokens ahead of the target — the drafter’s attention progressively shifts away from sink tokens and toward its own generated tokens. Two compounding issues cause this:
- The fused input representation becomes imbalanced at depth: higher-layer hidden states from the target model dominate the drafter’s input signal.
- Hidden-state magnitude grows across speculation steps due to an unnormalized residual path, making each successive step less stable.
The result is degrading acceptance rates the deeper the speculation goes — particularly acute in long-context workloads where the instability compounds over more tokens.
Two Architectural Fixes
Eagle 3.1 introduces two changes to address both issues:
- FC normalization after each target hidden state before the FC layer
- Post-norm hidden state feedback into the next decoding step
The post-norm design makes the drafter behave as though it is recursively re-invoked at each depth step, rather than accumulating state through an unnormalized stack. The architecture change is small; the effect is not.
Results
On Kimi K2.6 (NVFP4) with vLLM on a GB200 (TP=4), measured on SPEED-Bench coding tasks:
| Concurrency | Throughput vs. Eagle 3 |
|---|---|
| C=1 | 2.03× |
| C=4 | 1.71× |
| C=16 | 1.66× |
In long-context workloads specifically, Eagle 3.1 achieves up to 2× longer acceptance length compared with Eagle 3. The speedup is consistent across the concurrency range — the improvement does not dissolve under production load.
The team also open-sourced an Eagle 3.1 draft model for Kimi K2.6 on HuggingFace (lightseekorg/kimi-k2.6-eagle3.1-mla), providing a reference implementation for labs running K2.6 at scale.
Deployment
Eagle 3.1 ships as a config-driven extension of vLLM’s existing Eagle 3 code path. Backward compatibility with existing Eagle 3 checkpoints is preserved — upgrading requires no scaffold changes. The support has merged to vLLM main and ships in the upcoming v0.22.0 release, available via nightly builds now.
Enabling it requires adding a speculative config to the serve command:
--speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}'Why It Matters
Speculative decoding is a primary lever for inference cost and latency at scale. A 2x throughput increase on a production-grade model like Kimi K2.6 translates directly to halved serving costs per request at low concurrency — the regime that matters most for interactive agent use cases where users wait for responses. The attention drift diagnosis also explains why prior Eagle generations saw benchmark-to-production gaps that were hard to reproduce reliably, giving infrastructure teams a concrete architectural cause rather than deployment folklore.
TorchSpec, the training framework used to train the Eagle 3.1 draft model for K2.6, is also open-sourced and now supports Eagle 3.1 and future speculative decoding algorithms — lowering the barrier for other labs to train custom draft models for their serving targets.