AMD's Instella-MoE: a fully open 16B/2.8B MoE, data and code too
Weights at every training stage, the 7.1T-token data mix, and MIT code — all trained on AMD Instinct. The catch: research-only license, no GGUF yet.
Copy markdown
Fully open, top to bottom
AMD released Instella-MoE-16B-A3B — 16B total, 2.8B active — publishing weights from every training stage, the full data mixtures, and the training code. Its Think variant scores 73.22 on instruction benchmarks, edging OLMo-3-7B-Think (71.97) with far fewer active parameters.
2.8B active means it runs light
Only 6 of 64 experts (plus 2 shared) fire per token, so it serves roughly like a 3B model while holding 16B of knowledge. It ships as BF16 safetensors with a 64K context and runs today via Transformers, vLLM, or SGLang — but there's no GGUF yet, so laptop and CPU users will need to convert.
The catch: a research-only license
The weights ship under ResearchRAIL — academic and research use only, not commercial — while the training and inference code is MIT. You can study, fine-tune, and benchmark it, but you can't ship the weights in a product. AMD also flags it isn't for safety-critical, medical, or high-factual-accuracy use.
Trained end-to-end on AMD, in the open
7.1 trillion tokens on Instinct MI300X and MI325X with ROCm, using AMD's open Primus (training) and Miles (RL) stacks. A new FarSkip-Collective trick overlaps expert communication with compute for a 12.7% pretraining speedup and up to 39.2% faster time-to-first-token — a credible, reproducible non-NVIDIA training path now fully in the open.