← Back to feed

DeepSeek has released DeepInfer on GitHub under the deepseek-ai organisation. The repository is described as the high-performance inference framework that powers DeepSeek’s own large-scale production services — not a research prototype, but the operational stack the company runs in production.

Previous open-source releases from DeepSeek have been components: FlashMLA for multi-head latent attention, DeepGEMM for matrix operations, DeepEP for expert parallelism communication, and DSpark for speculative decoding. DeepInfer packages all of these together into a deployable inference server, adding the orchestration layer that ties them into a working system.

What DeepInfer Contains

The framework ships with six custom kernel libraries: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. Each addresses a different computational bottleneck in serving large mixture-of-experts models.

On top of the kernels, DeepInfer includes:

Expert Parallelism + Load Balancing (EPLB): Distributes expert computation across devices and actively rebalances load to avoid idle capacity. This is particularly relevant for V4.1 Flash, which uses a sparse MoE architecture where naive dispatch creates uneven GPU utilisation.

DSpark with confidence-scheduled verification: The speculative decoding system DeepSeek open-sourced in July 2026 is integrated and configurable via environment variable (DEEPINFER_ENABLE_MTP_VARLEN=1). Confidence estimates schedule draft token verification dynamically, with the maximum draft length controlled by DEEPINFER_USE_NEXTN_DECODE (default: 5).

SWA bounded replay: Reduces decoder prefill work by replaying only a bounded suffix of the sliding window attention window. DeepSeek sets the default at 140 retained tokens, covering the SWA window plus maximum backtracking length.

Asynchronous scheduling: Overlaps CPU batch scheduling with device execution, enabled by default. The goal is zero scheduling overhead on the critical path.

Batch-invariant inference: Produces identical model outputs regardless of batch composition — useful for auditing, regression testing, and reproducible deployments. Enabled via DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).

Deterministic sampling: Produces identical outputs for identical inputs and sampling parameters. Requires batch invariance plus a fixed seq_seed in SamplingParams.

Structured outputs: Constrains generation to a regex, JSON schema, or EBNF grammar via the SamplingParams interface.

The initial release supports one model: DeepSeek-V4.1-Flash.

Why This Is Different From DSpark

DSpark, released in July 2026, was a speculative decoding algorithm. It showed 60-85% inference speedups on V4 without retraining by using draft tokens and confidence-based verification. That was a technique paper with accompanying code.

DeepInfer is an inference server. It includes DSpark as one component but adds the full operational layer: parallelism strategies, kernel dispatch, scheduling, memory management, and output guarantees. The gap is the difference between publishing a fast engine component and publishing the car.

For organisations running V4.1 Flash on their own hardware, DeepInfer means they can now use the same inference configuration DeepSeek runs internally rather than integrating component libraries separately into vLLM or TGI.

What Is Not Included

The repository does not currently include weights. V4.1 Flash weights are available separately; the open-weight release of V4.1 Flash vision from September 2026 covers the multimodal variant. DeepInfer is the runtime, not the model.

Support is limited to V4.1 Flash at launch. Previous V4 variants and the larger V4-Pro model are not listed as supported targets. Whether that expands depends on DeepSeek’s roadmap.