← Back to feed

Speculative decoding has been one of the most effective latency-reduction techniques in production LLM serving — a small draft model proposes tokens that a larger target model verifies in parallel, cutting time-to-first-token and increasing throughput when drafts are accepted. The technique works well in controlled benchmarks. It degrades visibly in production: different chat templates, long-context inputs, unusual system prompts.

Eagle 3.1, released jointly by the EAGLE team, vLLM team, and TorchSpec team, is the first version that systematically addresses why.

The Root Cause: Attention Drift

The EAGLE team traced production instability to a phenomenon they call attention drift. As speculation depth increases — meaning the draft model generates more tokens ahead of the target — the drafter’s attention progressively shifts away from sink tokens and toward its own generated tokens. Two compounding issues cause this:

1. The fused input representation becomes imbalanced at depth: higher-layer hidden states from the target model dominate the drafter’s input signal.
2. Hidden-state magnitude grows across speculation steps due to an unnormalized residual path, making each successive step less stable.

The result is degrading acceptance rates the deeper the speculation goes — particularly acute in long-context workloads where the instability compounds over more tokens.

Two Architectural Fixes

Eagle 3.1 introduces two changes to address both issues:

• FC normalization after each target hidden state before the FC layer
• Post-norm hidden state feedback into the next decoding step

The post-norm design makes the drafter behave as though it is recursively re-invoked at each depth step, rather than accumulating state through an unnormalized stack. The architecture change is small; the effect is not.

Results

On Kimi K2.6 (NVFP4) with vLLM on a GB200 (TP=4), measured on SPEED-Bench coding tasks:

Concurrency	Throughput vs. Eagle 3
C=1	2.03×
C=4	1.71×
C=16	1.66×

In long-context workloads specifically, Eagle 3.1 achieves up to 2× longer acceptance length compared with Eagle 3. The speedup is consistent across the concurrency range — the improvement does not dissolve under production load.

The team also open-sourced an Eagle 3.1 draft model for Kimi K2.6 on HuggingFace (lightseekorg/kimi-k2.6-eagle3.1-mla), providing a reference implementation for labs running K2.6 at scale.

Deployment

Eagle 3.1 ships as a config-driven extension of vLLM’s existing Eagle 3 code path. Backward compatibility with existing Eagle 3 checkpoints is preserved — upgrading requires no scaffold changes. The support has merged to vLLM main and ships in the upcoming v0.22.0 release, available via nightly builds now.

Enabling it requires adding a speculative config to the serve command:

--speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}'

Why It Matters

Speculative decoding is a primary lever for inference cost and latency at scale. A 2x throughput increase on a production-grade model like Kimi K2.6 translates directly to halved serving costs per request at low concurrency — the regime that matters most for interactive agent use cases where users wait for responses. The attention drift diagnosis also explains why prior Eagle generations saw benchmark-to-production gaps that were hard to reproduce reliably, giving infrastructure teams a concrete architectural cause rather than deployment folklore.

TorchSpec, the training framework used to train the Eagle 3.1 draft model for K2.6, is also open-sourced and now supports Eagle 3.1 and future speculative decoding algorithms — lowering the barrier for other labs to train custom draft models for their serving targets.