Back to Articles a]:hidden"> Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs Community Article Published October 1, 2026 Upvote 25 +19 Kyle Wiggers Ai2Comms Follow allenai :last-child]:mb-0">Building a training stack around how MoEs actually work","children":[],"isValid":true,"title":"Building a training stack around how MoEs actually work"},{"id":"scaling-and-optimizing-moe-training","label":"Scaling and optimizing MoE training","children":[],"isValid":true,"title":"Scaling and optimizing MoE training"},{"id":"scaling-into-the-trillion-parameter-range","label":"Scaling into the trillion-parameter range","children":[],"isValid":true,"title":"Scaling into the trillion-parameter range"},{"id":"built-for-the-next-generation-of-olmo-open-for-everyone","label":"Built for the next generation of Olmo, open for everyone","children":[],"isValid":true,"title":"Built for the next generation of Olmo, open for everyone"}],"classNames":"not-prose hidden lg:block"}"> Building a training stack around how MoEs actually work Scaling and optimizing MoE training Scaling into the trillion-parameter range Built for the next generation of Olmo, open for everyone 📄 Tech Report | 💻 Code | 🧩 Interactive demo
Today we’re releasing Olmo-core 3, a significant upgrade to our framework for developing large language models featuring a redesigned open mixture-of-experts (MoE) training system.
Olmo-core 3 is designed to scale MoE training into the trillion-parameter range while preserving computational efficiency. It’s one of the core systems behind the next generation of Olmo, and part of our ongoing commitment to open up the tools and training infrastructure behind each new model.
Training large AI models takes a lot of compute, driving up costs and energy use and putting advanced model development out of reach for many academic researchers and smaller labs. MoE models offer a more efficient approach—they can contain many more learned components, or parameters, without requiring every input to use all of them. But the full model still has to be stored across GPU memory and updated during training, and directing inputs to the right experts – the specialized components within an MoE – across a cluster creates its own communication and coordination costs. As MoEs grow, those costs can erode much of the computational advantage of using only part of the model for each input.
Olmo-core 3 is built to close that gap. In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%.
The same infrastructure has been benchmarked at over one trillion total parameters.
Building a training stack around how MoEs actually work Olmo-core has evolved with each generation of Olmo.
Our work on sparse models goes back to OlmoE, which used an MoE architecture with 64 routed experts. Olmo 3, by contrast, used a dense architecture, meaning nearly all of the model was active for every token and its training stack was built around that design. Olmo-core 3 extends the framework with a training system designed for much larger MoE models.
Our earlier MoE implementation in Olmo-core used fully sharded data parallelism (FSDP), configured to gather and reshard model weights for each small batch of training data. Olmo-core 3 switches to a system based on distributed data parallelism (DDP). It keeps experts resident on GPUs and routes the relevant data to them, avoiding that repeated weight gathering.
NVIDIA’s Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation. In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput.
Scaling and optimizing MoE training Olmo-core 3 combines several techniques for distributing large MoEs across GPU clusters with optimizations that make routing and computation more efficient.
Three techniques determine how the model and its training state are split across hardware:
Together, these techniques allow an MoE to scale without requiring every GPU to keep the entire model and its training state in memory.
Olmo-core 3 also reduces the cost of routing data to the right experts and running their computations. Rowwise expert parallelism places routed data directly into expert input buffers, minimizing the extra work needed to rearrange it. GPU-resident routing keeps routing metadata on the GPUs, so the CPU can queue work without waiting for that information to be copied back. And grouped GEMM combines many small expert computations so GPUs can execute them more efficiently.
Finally, Olmo-core 3 supports MXFP8, a lower-precision number format that represents some values with fewer bits. This can reduce computation and the amount of data moved between GPUs, as long as those savings outweigh the cost of converting between number formats.