DiffusionGemma: Google Ships 26B MoE That Hits 1,000+ Tokens/Second by Generating Blocks, Not Tokens
Google released DiffusionGemma on June 10 under Apache 2.0. The model is a 26B parameter Mixture of Experts that activates only 3.8B parameters per inference call and generates text using a diffusion approach rather than sequential token prediction.
The headline number is 1,000+ tokens per second on a single NVIDIA H100, with 700+ tokens per second on a consumer RTX 5090. For comparison, standard autoregressive models running locally typically generate 50-150 tokens per second on the same hardware.
How it works
Standard language models generate text like a typewriter: one token at a time, left to right, each prediction depending on all previous predictions. This is efficient at scale in cloud serving — a server can batch thousands of user requests to keep the GPU saturated. On local hardware with a single user, the same model leaves the GPU largely idle, waiting for each sequential output.
DiffusionGemma inverts this. Instead of predicting the next token, the model starts with a 256-token block of noise and iteratively refines the entire block simultaneously — the same approach image diffusion models use for pixels. The GPU processes a much larger chunk of work in each forward pass, which saturates local hardware far more effectively.
The result: the decode bottleneck shifts from memory bandwidth to compute, which is where modern accelerators have the most headroom.
Specs
| Property | Value |
|---|---|
| Total parameters | 26B (MoE) |
| Active parameters | 3.8B per forward pass |
| VRAM (quantized) | 18GB |
| H100 speed | 1,000+ tokens/second |
| RTX 5090 speed | 700+ tokens/second |
| Tokens per forward pass | 256 |
| License | Apache 2.0 |
| Base architecture | Gemma 4 family |
What it’s good for
Bi-directional attention is the feature that differentiates DiffusionGemma from speed-optimized autoregressive models. Because all 256 tokens in a block are generated in parallel, every token can attend to all others — including tokens that haven’t been finalized yet. This is structurally impossible in standard left-to-right models.
Domains where this helps:
- Code infilling: filling a gap in the middle of existing code, where the context on both sides of the insertion point matters.
- In-line editing: revising a paragraph in the middle of a document.
- Non-sequential structures: amino acid sequences, mathematical graphs, constraint satisfaction problems. Google’s demo shows a fine-tuned version solving Sudoku — a task standard models struggle with because the solution requires reasoning about the whole grid simultaneously.
The quality trade-off
Google is explicit that DiffusionGemma is lower quality than standard Gemma 4 for most production tasks. The model trades output fidelity for speed and parallel generation. For applications where quality is paramount, Gemma 4 remains the recommendation.
The primary use case is local, low-concurrency inference where speed and interactivity are more important than maximum accuracy — real-time editing tools, development environments, research workflows, and hardware-constrained deployments.
In cloud high-QPS serving, autoregressive models can be deployed to saturate compute efficiently through batching. DiffusionGemma’s throughput advantage shrinks in those configurations and can result in higher per-request cost.
Context
DiffusionGemma builds on Google DeepMind’s earlier Gemini Diffusion research. The open release and Apache 2.0 license mean fine-tuning for specific domains is straightforward. Google notes that even modest fine-tuning can close the quality gap significantly for specialized tasks, as the Sudoku demo demonstrates.