← Back to feed

The model weights landed on Hugging Face last week. Now the paper explains why they work.

Moonshot AI’s Kimi K3 technical report drops enough architecture detail to distinguish it from a typical frontier release announcement. The headline numbers — 93.4% SWE-bench Verified, 79.2 on LiveBench, 2.8T parameters at $3/$15 per million tokens — were covered at launch. The paper reveals what had to be rebuilt to get there.

A New Attention Mechanism

The most significant departure from standard transformer design is the KDA (Kimi Delta Attention) mechanism. Each transformer block runs three KDA layers and one global MLA (Multi-head Latent Attention) layer. KDA replaces full softmax attention with a bounded-decay mechanism — the bounded decay makes difficult causal attention tiles Tensor Core-friendly without special-casing, and it enables NoPE.

NoPE stands for No Position Encoding. Kimi K3 removes Rotary Position Embedding entirely. Most long-context models spend significant engineering effort extending or modifying RoPE to support ranges beyond 8K-32K. Extending RoPE is finicky and produces degradation at extreme context lengths. Kimi K3 sidesteps the problem: positional information is carried implicitly through KDA’s bounded decay curves rather than injected through frequency-based encoding.

The tradeoff is that the attention mechanism must now handle positional semantics through learned decay patterns rather than explicit signals. The paper claims this is more efficient and produces cleaner behaviour at 1M token context lengths, where RoPE interpolation schemes typically strain.

Depth-wise Attention Residuals

Standard transformer architectures accumulate state through a single residual stream — each layer writes its output to a running sum that passes to the next layer. Kimi K3 adds Attention Residuals, allowing each layer to selectively retrieve representations from earlier blocks rather than being constrained to only what the incrementally-updated residual stream carries forward.

The effect is that the model attends across depth in the network, not only across tokens in the sequence. Earlier block representations — potentially capturing different abstraction levels — remain directly accessible rather than being compressed into the residual accumulation. For long-horizon tasks where early reasoning context needs to persist through many downstream computation steps, this matters.

Training for Agent Stability

The training regime is designed around a specific failure mode: agents that learn to exploit a fixed evaluation scaffold rather than developing genuine task-completion capability.

During RL training, Kimi K3 was exposed to varied scaffolding. Tools, prompts, memory configurations, context management strategies, skills, and subagent structures all change across training trajectories. There is no fixed harness to overfit to.

Partial rollouts use resumable microVM sandboxes — unfinished environment states persist across training iterations. An agent trajectory that gets partway through a task and fails can be continued from that state in the next iteration rather than being reset. This enables training on long-horizon tasks that would otherwise require prohibitively long single rollouts.

The result on benchmarks: 91.2% on BrowseComp, which tests real browsing agent capability over multi-step search tasks. The model trained on varied scaffolding generalises across scaffold types at inference time.

MoE Stability at Scale

896 experts, 16 active per token — a sparsity ratio of roughly 1.8%. At this scale, expert routing instability is a documented failure mode: routers can collapse to using a small subset of experts, or produce training loss spikes when load imbalance triggers cascade effects.

The paper describes a Quantile Balancing scheme that computes the routing bias directly from the load distribution rather than optimising a proxy objective. The target: each expert’s observed token-load matches its target percentile. Auxiliary stability measures include a half-width latent expert path, RMSNorm on expert outputs, and bounded SiTU-GLU activations.

Context Curriculum

The 1M token context was not extended from a shorter window. The training curriculum runs 8K to 64K, then 256K to 1M, using synthetic multimodal tasks where relevant evidence is deliberately scattered across the full sequence at every stage. The model was forced to develop genuine long-range dependency resolution rather than learning to ignore the distant past.

Numbers

• SWE-bench Verified: 93.4% (3rd globally behind Mythos 5 at 95.5% and Fable 5 at 95.0%)
• GPQA Diamond: 93.5%
• BrowseComp: 91.2%
• LiveBench: 79.2 overall, 62.2 agentic coding
• Context: 1M tokens
• Pricing: $3/M input, $15/M output
• Architecture: 2.8T parameters, 896-expert MoE, 16 active per token
• Efficiency: 2.5x intelligence per compute unit versus Kimi K2
• License: free for products under 100M MAU or under $20M ARR; commercial terms apply above those thresholds

Weights are available at HuggingFace (moonshotai/Kimi-K3). The technical report is at github.com/MoonshotAI/Kimi-K3.