Orthrus Posts 7.8x Inference Speedup on Qwen3 With Zero Accuracy Loss
A paper posted to arXiv on May 12, 2026, and released as open weights on May 13, claims a 7.8x inference speedup on Qwen3 models with no degradation in output quality — a result that, if it holds at scale, meaningfully changes the economics of deploying open-weight models.
The method, called Orthrus, uses a dual-view architecture: the original autoregressive model is kept frozen, and a diffusion-based parallel decoding component is fine-tuned on top. Both views share the same key-value cache, which eliminates the memory overhead that typically makes speculative decoding expensive at long context lengths.
The Numbers
Average speedups from the published benchmarks:
| Model | Avg. Speedup |
|---|---|
| Orthrus-Qwen3-1.7B | 4.25x |
| Orthrus-Qwen3-4B | 5.20x |
| Orthrus-Qwen3-8B | 5.36x |
| Peak (generation tasks) | 7.8x |
On the MATH-500 benchmark, Orthrus-Qwen3-8B achieves approximately 6x speedup over the Qwen3-8B baseline with no accuracy drop — compared to Fast-dLLM-v2, which achieves similar speeds but with measurable accuracy degradation on complex reasoning tasks.
What Makes It Different
Existing speculative decoding approaches (EAGLE-3, DFlash) use a separate draft model that must maintain its own KV cache. As context grows, that draft cache becomes a significant memory cost and slows acceptance rates. Orthrus sidesteps this by sharing the base model’s cache natively across both the autoregressive and diffusion views, resulting in O(1) additional memory overhead rather than the O(n) scaling of traditional speculative decoding.
The diffusion component is trained by fine-tuning only 16% of total model parameters. The base model weights are never modified. This means the exact output distribution of the original Qwen3 model is preserved by construction — the authors call it an “exact intra-model consensus mechanism.”
The claim is strict losslessness: not approximate quality retention, but identical predictive distribution to the frozen base. The benchmarks are consistent with this: MATH-500 accuracy curves for Orthrus and the Qwen3 baseline overlap.
The Practical Ceiling
The peak 7.8x figure is a best-case result on generation-heavy tasks. Real serving workloads vary. A published independent test of DART (a different diffusion-inspired speculative decoding method) on Qwen3-14B on a single Blackwell GPU achieved 1.97x over the autoregressive baseline at 512-token generation length, falling to 1.49x at 4,000 tokens and 1.16x at 8,000 tokens. Context-length decay is a known pattern across all speculative decoding approaches.
Orthrus has not yet published long-context scaling curves. The 41-star GitHub repository and recent arXiv submission suggest an early-stage project, not production-ready inference infrastructure. The test at HuggingFace is against Qwen3 1.7B–8B, not the larger Qwen3-30B or Qwen3-235B models where inference cost matters most.
Why It’s Worth Tracking
Speculative decoding has been one of the few inference optimization levers that doesn’t require hardware or quantization tradeoffs. EAGLE-3 and DFlash have already shipped in production serving stacks (SGLang supports DFlash for Qwen3-Coder-30B-A3B). If Orthrus’s losslessness guarantee and memory efficiency claims replicate at larger model sizes and longer contexts, it represents a meaningful step forward in the inference efficiency stack — particularly for cost-sensitive open-weight deployments where every token saved translates directly to margin.
The paper is at arXiv, weights are on HuggingFace, and it appeared on Hacker News front page on May 16. Whether it replicates outside the authors’ test environment is the question worth watching over the next few weeks.