ZEDA Cuts 50% of MoE Expert Compute on Already-Deployed Models — 20% Faster Inference, Near-Zero Accuracy Loss
Every Mixture-of-Experts model in production — Qwen3, GLM, Mixtral, DeepSeek V4 — already skips most experts per token by design. MoE’s entire value proposition is sparse activation: route each token to a small subset of experts, ignore the rest. The efficiency gains over dense models are real.
What ZEDA (Zero-Expert Self-Distillation Adaptation) identifies is that even within the activated subset, a large fraction of tokens don’t need those experts at all. The model spends compute on easy tokens — structured code, common phrases, simple math — that could bypass expert processing entirely. ZEDA adds an exit route: a set of zero-output experts that return nothing, letting tokens skip computation when the model’s routing signal says they don’t need it.
A paper from Tsinghua and CAS researchers submitted May 18 shows this works on fully post-trained models, with no pretraining required.
What ZEDA Does
The method injects parameter-free zero-output experts into each MoE layer. These are not pruned experts or compressed weights — they output exactly zero. The model is then adapted through two-stage self-distillation: the original static MoE becomes a frozen teacher, and the augmented dynamic version learns when it can safely route to a zero expert. A group-level balancing loss prevents collapse (all tokens routing to zero).
The adaptation does not touch the base model’s parameters. It adds routing capacity that existing models lack: the ability to say “this token needs nothing from this layer.”
Results
Tested on Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following:
| Model | Expert FLOPs eliminated | Inference speedup | Accuracy delta |
|---|---|---|---|
| Qwen3-30B-A3B | >50% | ~1.20x | marginal |
| GLM-4.7-Flash | >50% | ~1.20x | marginal |
Against the strongest prior dynamic MoE baseline, ZEDA outperforms by 6.1 points on Qwen3-30B-A3B and 4.0 points on GLM-4.7-Flash.
The paper also characterises where compute is actually needed. Expert budget did not track task difficulty monotonically. Structured code fragments and routine math used fewer active experts; tokens where the teacher and student disagreed, or where the model faced uncertainty, used more. The zero-expert routing effectively learns a proxy for “how much does this token need processing.”
Why It Matters for Deployed Models
The critical point is the retrofit angle. Techniques that cut inference cost typically require either:
- Retraining from scratch with a different architecture
- Post-training distillation into a smaller model
- Quantisation (quality-cost tradeoff)
ZEDA does none of these. It adapts a model that is already deployed — trained, aligned, and in production — and adds dynamic routing capacity through a lightweight adaptation pass. For operators running Qwen3 or GLM at scale, this is a meaningful operational lever that doesn’t require a new model version.
The 1.20x end-to-end speedup is also a real number, not a FLOP count on a whiteboard. End-to-end benchmarks include memory movement and kernel launch overhead, which frequently dominate naive FLOP savings. A 50% expert FLOP reduction delivering a 20% end-to-end speedup is a reasonable conversion ratio and consistent with what memory-bound inference workloads typically allow.
Citation
Lv, X. et al. “Post-Trained MoE Can Skip Half Experts via Self-Distillation.” arXiv:2605.18643. May 18, 2026.