AMD Instella-MoE: A Fully Open 16B Mixture-of-Experts Model Trained Entirely on AMD GPUs

Instella-MoE is a mixture-of-experts language model carrying 16 billion total parameters, built from scratch on AMD Instinct MI300X and MI325X accelerators and released on July 24. What separates it from most releases labeled "open" is the scope of what AMD actually handed over. The package covers model weights captured at every stage of training, the training configurations behind them, the data mixtures used, intermediate checkpoints, and inference code — a level of disclosure that places it among the most transparent large-scale model releases to date.

That distinction matters because "open" has been stretched thin across the industry. Plenty of models ship with downloadable weights and nothing else, leaving researchers unable to reproduce results or study how a model arrived at its final behavior. Instella-MoE goes the other direction, exposing the intermediate states and recipe details that make replication and analysis possible.

Sparse Activation Keeps Inference Costs Low

The model activates only 2.8 billion of its 16 billion parameters for any given token. That is the core economic argument for a mixture-of-experts design: total capacity stays large, but the compute burden at inference time tracks closer to a dense model several times smaller.

The practical effect is that Instella-MoE can hold the knowledge and specialization of a 16-billion-parameter system while costing something nearer to a 3-billion-parameter one to run. AMD positions the result as the strongest performer among fully open-source models in its size class.

Context handling is also worth noting. The model supports windows of up to 64,000 tokens, putting long-document processing and extended multi-turn work within reach rather than treating them as edge cases.

Benchmark Results Against Comparable Open Models

On standard evaluations, the base model posted an average score of 76.7. That figure places it ahead of two well-known reference points in the open ecosystem:

The comparison against Moonlight-16B-A3B is the more telling one. That model belongs to the open-weight category rather than the fully open one, meaning Instella-MoE is holding its ground against systems that disclose considerably less about how they were built.

Six Stages From Pre-Training to Reinforcement Learning

The training pipeline runs through six distinct phases rather than a single monolithic run:

1. Pre-training — the foundational pass that establishes core language capability.
2. Mid-training — a refinement stage layered on top of the base run.
3. Long-context extension — the phase that pushes the usable window out to 64K tokens.
4. Supervised fine-tuning — task-directed training on curated examples.
5. Direct preference optimization — alignment tuning driven by preference data.
6. Reinforcement learning — the final stage shaping model behavior.

Sequencing training this way, with checkpoints preserved at each transition, is part of what makes the release useful to researchers. Anyone studying how capabilities emerge across a training run has the artifacts to trace that progression instead of inferring it from a finished model.

Two Architectural Techniques Behind the Efficiency Gains

Instella-MoE introduces two methods that AMD developed for this release, both aimed at squeezing more throughput out of the hardware.

Gated Multi-head Latent Attention

This approach attaches a learned gate to the attention mechanism, giving the model a way to selectively filter out attention responses that carry little useful signal. Rather than treating every attention output as equally worth propagating, the gate suppresses the ones that contribute least.

FarSkip-Collective

FarSkip-Collective addresses a different bottleneck: the communication overhead that shows up when experts are distributed across devices during training. The method restructures connectivity so that communication overlaps with computation instead of stalling it.

The measured results are concrete. Pre-training ran 12.7% faster. On the inference side, time-to-first-token dropped by as much as 39.2% — a latency improvement users would feel directly in interactive applications.

Where This Fits in AMD's ROCm Strategy

Instella-MoE is not a standalone experiment. It extends the earlier Instella series, a set of 3-billion-parameter models AMD introduced in 2025, and it follows ZAYA1 — a large-scale mixture-of-experts model that Zyphra trained on MI300X hardware last year.

Taken together, these efforts read as a sustained campaign to establish that AMD silicon and the ROCm software stack can carry frontier-scale AI development end to end. That claim has been the sticking point for years. Nvidia's dominance in AI training rests as much on the maturity of the CUDA ecosystem as on the chips themselves, and no amount of raw specification parity resolves the question of whether real models can be trained on the alternative.

Training a competitive model from scratch on your own accelerators, then publishing the full recipe, is a direct answer to that question. It shifts the argument from marketing claims to reproducible evidence.

Licensing and the Road Ahead

Instella-MoE ships under a Research RAIL license, scoping its use to academic and research work rather than unrestricted commercial deployment. That framing is consistent with the release's apparent purpose: demonstrating capability and enabling study rather than competing for production workloads.

AMD has indicated it intends to keep building on this line of work, with larger models and additional efficiency improvements on the roadmap.