AMD Trains Its Own MoE on Instinct GPUs and Opens Every Checkpoint — But the Weights Aren't Commercially Free
AMD released Instella-MoE-16B-A3B on August 1, a 16-billion-parameter Mixture-of-Experts language model trained from scratch on its own Instinct MI300X and MI325X GPUs. The release includes weights from every training checkpoint, data mixture configurations, and full training code. AMD is using this model to demonstrate that its ROCm software stack can train competitive models without touching NVIDIA hardware or CUDA.
Architecture
Instella-MoE is a decoder-only model with 27 layers, 16 attention heads, and a 128,000-token vocabulary. Each MoE layer uses 2 shared experts plus 6 routed experts selected from 64 candidates, activating 2.8 billion parameters per token against 16 billion total.
Two systems-level contributions distinguish the release:
Gated Multi-head Latent Attention (Gated MLA): An attention variant that reduces key-value cache size, delivering a 39.2% reduction in time to first token compared to standard grouped-query attention at this model scale.
FarSkip-Collective: A connectivity pattern for MoE routing that cuts cross-node communication overhead. AMD reports a 12.7% training throughput improvement from this change.
Benchmark Results
The base checkpoint averages 76.7 across standard evaluations, the highest among fully open models at this size class — above Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), and OLMo-3-7B (70.1). It trails Qwen3.5-4B-Base at 79.5, which is smaller in active parameters.
Post-training stages improve the instruction-following score from 71.58 (SFT) to 73.22 (Think), above OLMo-3-7B-Think (71.97) and Gemma-4-E4B Think (70.47). IFEval rises from 77.08 to 83.70.
Long-context results: HELMET average 41.5, RULER 79.4. SWE-bench numbers are not reported.
The License Gap
The headline “fully open” requires a correction. The model weights are released under ResearchRAIL, which restricts commercial deployment. The training code carries an MIT license, which does not. The distinction matters for enterprise users: teams can study the architecture and reproduce training freely, but cannot ship Instella-derived products commercially without AMD’s explicit approval under ResearchRAIL terms.
AMD has used ResearchRAIL on prior releases including earlier Instella checkpoints. The pattern suggests AMD views model weights as a research instrument and marketing proof-point for its GPU stack, not as a commercial product in themselves.
The Hardware Angle
The training infrastructure is the actual story AMD is making here. Running a competitive MoE training run end-to-end on MI300X and MI325X, with ROCm and AMD’s own Primus and Miles systems libraries, is a direct counter to the repeated claim that CUDA’s software moat makes NVIDIA hardware the only viable option for frontier AI training. The Gated MLA and FarSkip-Collective improvements came out of training-efficiency work on AMD’s own chips, not as abstractions designed for NVIDIA and then ported.
Whether the 12.7% throughput improvement translates at the scale of frontier model training — where the top labs run hundreds of thousands of GPUs — remains open. At 16B parameters, the proof is preliminary but it is at least hardware-demonstrated rather than simulated.