← Back to feed

For most of vLLM’s history, speculative decoding was a CUDA-first feature. AMD users running the ROCm stack had access to the same model weights but not the same inference optimization toolkit. A blog post published August 23 closes that gap.

The post, written by the vLLM team, documents speculative decoding measurements on AMD Instinct MI300X and MI355X GPUs using the ROCm open software platform — the first systematic treatment of the feature on AMD hardware.

What Speculative Decoding Does

Standard autoregressive decoding advances one token per model step. Each output token requires a full forward pass through the target model, making long generations expensive in wall-clock time.

Speculative decoding separates the process into two stages. A lightweight draft component — faster than the target model by design — proposes several candidate tokens in sequence. The target model then verifies those candidates in a single forward pass. When draft tokens are accepted, the system commits multiple output tokens from one target-model step. When a draft token is rejected, the target model supplies the next correct token and the remaining candidates from that proposal are discarded.

The output distribution is mathematically equivalent to the target model running without speculative decoding. No quality trade-off is made. The potential benefit is latency and throughput — specifically in workloads where the draft model predicts correctly at a useful rate.

Five Methods, One Blog Post

The vLLM post evaluates five drafting approaches, each with a different design for how the draft component interacts with the target model:

Native MTP (Multi-Token Prediction): The target model itself proposes multiple tokens simultaneously, without a separate draft model. The draft tokens are generated from a prediction head that shares weights with the target model.

Gemma 4 MTP: Applies the multi-token prediction approach using a Gemma 4 draft checkpoint, decoupled from the target model architecture.

EAGLE-3: An autoregressive drafting method where the draft component uses features from the target model’s intermediate layers to generate candidate tokens one at a time. EAGLE-3 tends to achieve high acceptance rates on code-heavy workloads.

DFlash: A flash-attention-based approach optimized for throughput in batch serving scenarios.

DSpark: A hybrid that combines parallel and autoregressive proposal stages, aiming to balance per-request latency and batch throughput.

The post makes clear that gains vary substantially by method, model family, draft checkpoint, workload, and acceptance behavior. No single configuration dominated across all measured dimensions, which is consistent with findings from CUDA-side evaluations of the same techniques.

The AMD Hardware Context

MI300X and MI355X sit at the high end of AMD’s Instinct server GPU lineup. The MI300X, with 192GB of HBM3 memory, is the card AMD has positioned against NVIDIA’s H100/H200 for large-model inference. The MI355X is the more recent generation with higher memory bandwidth.

ROCm is AMD’s software stack analogous to CUDA: the runtime, compiler, and library layer that makes GPU compute accessible to frameworks like vLLM and PyTorch. One persistent criticism of AMD’s AI inference position has been that software support lags NVIDIA — fewer optimized kernels, less mature tooling, delayed feature parity. The speculative decoding blog post addresses one specific gap in that list.

The techniques now available on AMD were originally developed and optimized for CUDA. Bringing them to ROCm required engineering work at the kernel and attention-layer level, not just a configuration flag.

NVIDIA’s Simultaneous Efficiency Push

The AMD blog arrived as NVIDIA was making its own inference efficiency claims on a separate front.

NVIDIA announced 1.9x throughput improvements for local inference on RTX GPUs carrying at least 24GB of VRAM. The gains come from new XQA (Cross-Query Attention) kernels combined with enhanced speculative decoding support, improving both prefill throughput on long documents and token generation speed for streaming responses.

On the data center side, NVIDIA reported 1.4x throughput improvement for Qwen3.6-27B running in vLLM on DGX Spark platforms, and a 20% improvement on DeepSeek V4 Flash inference.

The NVIDIA improvements are delivered through driver and library updates rather than new hardware — a software-layer optimization that benefits existing installed base.

The Pattern

Both announcements point to the same underlying dynamic: inference optimization is moving from hardware-specific tricks toward a software-layer discipline that any capable silicon can implement. Speculative decoding, once seen as an advanced feature requiring careful CUDA tuning, is now documented and reproducible on AMD’s stack.

For infrastructure teams evaluating AMD as an alternative to NVIDIA for inference workloads, the vLLM blog removes one objection. The practical question remains whether AMD’s kernel optimization cadence can match NVIDIA’s, and whether the acceptance rates measured in the vLLM post translate to production workload distributions — which tend to be harder on speculative methods than academic benchmarks.

The five-method comparison is a starting point, not a verdict. Teams running MI300X or MI355X clusters can now benchmark the specific combination that matches their model and traffic pattern rather than relying on generalizations from CUDA results.