AiAnyTool - AI Tools Directory LogoAiAnyTool - AI Tools Directory Logo
AIAnyToolAI Tools Directory
Full ListAI CategoriesAI News Today
Loading theme toggle
Submit an AISign up
AiAnyTool - AI Tools Directory LogoAiAnyTool - AI Tools Directory Logo
AiAnyTool

Artificial intelligence for everyone. Discover, compare, and leverage over 19,000 verified AI tools.

Updated daily with manual verification

Resources

  • AI Deals & Discounts
  • AI News Today
  • Tutorials & Guides
  • MCP Servers Directory

Directory

  • Full AI List
  • AI Categories
  • Top 10 Best AI
  • New AI Tools
  • Submit an AI Tool →

About & Legal

  • About AiAnyTool
  • Contact Support
  • Privacy Policy
  • Terms of Service

© 2026 AiAnyTool.com — All rights reserved.

The definitive artificial intelligence directory and search engine.

HomeAI NewsOpen Source

Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Hugging FaceOpen SourceOctober 1, 2026(Oct 1)5 min read
Share:
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Full Coverage Details

Hugging Face Blog
Back to Articles a]:hidden"> Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs Community Article Published October 1, 2026 Upvote 25 +19 Kyle Wiggers Ai2Comms Follow allenai :last-child]:mb-0">Building a training stack around how MoEs actually work","children":[],"isValid":true,"title":"Building a training stack around how MoEs actually work"},{"id":"scaling-and-optimizing-moe-training","label":"Scaling and optimizing MoE training","children":[],"isValid":true,"title":"Scaling and optimizing MoE training"},{"id":"scaling-into-the-trillion-parameter-range","label":"Scaling into the trillion-parameter range","children":[],"isValid":true,"title":"Scaling into the trillion-parameter range"},{"id":"built-for-the-next-generation-of-olmo-open-for-everyone","label":"Built for the next generation of Olmo, open for everyone","children":[],"isValid":true,"title":"Built for the next generation of Olmo, open for everyone"}],"classNames":"not-prose hidden lg:block"}"> Building a training stack around how MoEs actually work Scaling and optimizing MoE training Scaling into the trillion-parameter range Built for the next generation of Olmo, open for everyone 📄 Tech Report | 💻 Code | 🧩 Interactive demo Today we’re releasing Olmo-core 3, a significant upgrade to our framework for developing large language models featuring a redesigned open mixture-of-experts (MoE) training system. Olmo-core 3 is designed to scale MoE training into the trillion-parameter range while preserving computational efficiency. It’s one of the core systems behind the next generation of Olmo, and part of our ongoing commitment to open up the tools and training infrastructure behind each new model. Training large AI models takes a lot of compute, driving up costs and energy use and putting advanced model development out of reach for many academic researchers and smaller labs. MoE models offer a more efficient approach—they can contain many more learned components, or parameters, without requiring every input to use all of them. But the full model still has to be stored across GPU memory and updated during training, and directing inputs to the right experts – the specialized components within an MoE – across a cluster creates its own communication and coordination costs. As MoEs grow, those costs can erode much of the computational advantage of using only part of the model for each input. Olmo-core 3 is built to close that gap. In one benchmark, we increased the expert pool from 8 to 128 while still selecting only four experts per token – the small units of text a language model processes – keeping the number of active parameters per token roughly fixed at about 3.2B. Total parameter capacity grew from 4.6B to 47B, while training throughput fell by less than 5%. The same infrastructure has been benchmarked at over one trillion total parameters. Building a training stack around how MoEs actually work Olmo-core has evolved with each generation of Olmo. Our work on sparse models goes back to OlmoE, which used an MoE architecture with 64 routed experts. Olmo 3, by contrast, used a dense architecture, meaning nearly all of the model was active for every token and its training stack was built around that design. Olmo-core 3 extends the framework with a training system designed for much larger MoE models. Our earlier MoE implementation in Olmo-core used fully sharded data parallelism (FSDP), configured to gather and reshard model weights for each small batch of training data. Olmo-core 3 switches to a system based on distributed data parallelism (DDP). It keeps experts resident on GPUs and routes the relevant data to them, avoiding that repeated weight gathering. NVIDIA’s Megatron-Core is an established option for training large MoEs. Olmo-core 3 brings an integrated MoE training stack to the framework behind Olmo, with a redesign that improves throughput over our earlier FSDP-based implementation. In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput. Scaling and optimizing MoE training Olmo-core 3 combines several techniques for distributing large MoEs across GPU clusters with optimizations that make routing and computation more efficient. Three techniques determine how the model and its training state are split across hardware: Together, these techniques allow an MoE to scale without requiring every GPU to keep the entire model and its training state in memory. Olmo-core 3 also reduces the cost of routing data to the right experts and running their computations. Rowwise expert parallelism places routed data directly into expert input buffers, minimizing the extra work needed to rearrange it. GPU-resident routing keeps routing metadata on the GPUs, so the CPU can queue work without waiting for that information to be copied back. And grouped GEMM combines many small expert computations so GPUs can execute them more efficiently. Finally, Olmo-core 3 supports MXFP8, a lower-precision number format that represents some values with fewer bits. This can reduce computation and the amount of data moved between GPUs, as long as those savings outweigh the cost of converting between number formats.
H

Hugging Face

Original Reporting Source

Read Full Article at Source

AI Tools Mentioned in this Story

Explore features, ratings, and alternatives from our 19,000+ AI tools catalog.

Explore Directory

AssistLoop

Free plan with 150 message credits and basic features, Basic at $25/month for 3,000 credits and lead collection, Pro at $109/month for 15,000 credits and advanced actions, Business with custom pricing for 50,000+ credits and unlimited agents.

AssistLoop's AI chatbots slash response times by 70% and boost satisfaction, scaling support for e-commerce and SaaS without extra hires.

AlternativesView Tool

GetResponse AI Website Builder

Free + from $10.23/mo

Design professional websites effortlessly with AI.

AlternativesView Tool

HyperTxt

Lite plan at $15/month with 20 credits, Pro at $39/month (most popular) with 100 credits and priority support, Pro+ at $125/month with 500 credits and advanced features, free trial available on all paid plans.

Hypertxt.ai is an AI-powered SEO content writing tool that helps create high-quality, optimized content for improved search engine rankings. It offers various features to streamline the content creation process, from keyword research to content optimization and plagiarism checks.

AlternativesView Tool
← Back to AI News RadarExplore Trending Stories
Trending AI Stories
View All
#1

Signal blocks screenshots on iOS 27 and Android to stop AI agents from reading private chats

The Decoder • 3h ago
#2

Google's Gemini can now run Mac apps and finish desktop chores in the background while you keep working

The Decoder • 4h ago
#3

Anthropic mathematician claims to have solved prestigious Millennium Prize problem with Claude

The Verge AI • 6h ago
#4

Apple is hiring engineers to build its own AI agent

The Verge AI • 8h ago
#5

Apple joins the race for AI agents that control your device while raising new barriers for competitors

The Decoder • 8h ago

More in Open Source

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

DaisyRose-27B: Pushing the Frontier of Post-Training Efficiency

2d ago
Why your fine-tuning job takes longer than the FLOPs say

Why your fine-tuning job takes longer than the FLOPs say

2d ago
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

2d ago
Hugging Face and OpenRouter join forces to make open-weight models easy to run

Hugging Face and OpenRouter join forces to make open-weight models easy to run

4d ago