Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das, Bharat Kaul · 2026
Read Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer by Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das, Bharat Kaul (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00785.txt.
Introduction
Vooturi and colleagues pretrain language models from scratch on Aurora, an exascale machine with 127,488 Intel Ponte Vecchio GPU tiles. They use their Optimus training library for the work. They first train 1-billion dense and 7-billion mixture-of-experts models on 3072 tiles for 4 trillion tokens, then scale to 220-billion-parameter models. On the largest model, scaling efficiency reaches around 90 percent at 12288 tiles, aided by custom kernels and a sharded optimizer.
Catalog record
| CREATOR | Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das, Bharat Kaul |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34458 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.00785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das, Bharat Kaul Abstract Abstract : Pretraining Large Language Models (LLMs) from scratch requires massive amount of compute. Aurora super computer [ 1 ] is an ExaScale machine with 127,488 Intel PVC (Ponte Vechio) GPU tiles. In this work, we showcase LLM pretraining on Aurora at the scale of 1000s of GPU tiles. Towards this effort, we developed Optimus, an inhouse training library with support for standard large model training techniques. Using Optimus, we first pretrained Mula-1B, a 1 Billion dense model and Mula-7B-A1B, a 7 Billion Mixture of Experts (MoE) model from scratch on 3072 GPU tiles for the full 4 trillion tokens of the OLMoE-mix-0924 dataset. We then demonstrated model scaling by pretraining three large MoE models Mula-20B-A2B, Mula-100B-A7B, and Mula-220B-A10B till 100 Billion tokens on the same dataset. On our largest model Mula-220B-A10B, we pushed the compute scaling from 384 to 12288 GPU tiles and observed scaling efficiency of around 90% at 12288 GPU tiles. We significantly improved the runtime performance of MoE models using custom GPU kernels for expert computation, and a novel EP-Aware sharded optimizer resulting in training speedups up to 1.71x. As part of the Optimus library, we also developed a robust set of reliability and fault tolerant features to improve training stability and continuity at scale. 1 Training Large Models Training a model with P parameters in BF16 mixed precision using the AdamW optimizer requires atleast 16P bytes of memory - 2P for weights, 2P for gradients, 4P for FP32 master weights, and 8P for FP32 optimizer states. As a result, training a small 7B model with Pytorch DDP is infeasible on PVC GPU tiles as 112 GB ( 16\times 7 ) memory requirement is beyond the 64 GB memory capacity available on a PVC GPU tile. To overcome this limitation and enable large model training, several techniques such as sharded data parallelism (e.g., ZeRO [ 10 ] , FSDP [ 12 ] ), model parallelism (Tensor [ 11 ] , Expert [ 6 ] , and Pipeline [ 5 ] ), context parallelism [ 7 ] , activation checkpointing [ 3 ] etc. have been proposed. In this section, we will go through the subset of the techniques we employed and specific implementations of them in our Optimus training library. Sharded Optimizer (SO) : In PyTorch DDP, the optimizer states are fully replicated on every data-parallel (DP) rank. After the backward pass, gradients are synchronized across ranks using an all-reduce operation. With the full gradient available locally, every rank independently updates its copy of the model weights. In contrast, when using DP with a sharded optimizer, the optimizer states are partitioned (sharded) across DP ranks instead of being replicated. As a result, reduce-scatter operation is used to synchronize the gradients, distributing only the relevant gradient shards to each rank—specifically, the gradients corresponding to the parameters whose optimizer states it owns. Each rank updates only its assigned parameter shard. Finally, an all-gather operation is used to share the updated parameter shards so that all DP ranks once again hold a consistent and complete copy of the model. Sharding optimizer states allows us to train small MoE models with just DP. Tensor Parallelism (TP) : In TP, the model is partitioned horizontally. A decoder block in a dense transformer model has two modules : attention and MLP. In TP, the attention heads are divided in the attention module and intermediate size is divided in the MLP modules. To accumulate partial output activations and input gradients, we need to do allredue after/before attention and MLP in forward/backward pass on activations/gradients. Apart from decreasing the memory requirement for the GPU, TP allows us to scale the training to more number of GPUs without increasing the global batch size. Expert Parallelism (EP) : A decoder block in a Mixture-of-Experts (MoE) model has two modules : attention and SparseMoE. The SparseMoE module contains multiple experts along with a router. In Expert Parallelism (EP), the experts are divided across the EP ranks, while the non-expert parameters are replicated on each rank. Because expert parameters make up the majority of the model’s weights, distributing them allows us to train medium sized MoE models. In EP, global batch size scales with the number of GPUs just like DP. Pipeline Parallelism (PP) : In Pipeline Parallelism (PP), the model is partitioned vertically across layers. Each rank processes forward/backward for it’s corresponding layer block and sends the activation/gradient to the next/previous rank. In PP, the input batch is divided into micro-batches and these are scheduled across PP ranks using different scheduling strategies. PP truly allows us to train arbitrary large models as the division is across layers. Selective Activation Checkpointing (SAC) : In forward pass, intermediate activations need to be stored to compute gradients in the backward pass. These intermediate activations can take up significant amount of memory. In SAC, for the selected blocks of the forward pass, only the input activation to the selected block is stored instead of all the activations in that block. During the backward pass for the selected block, the forward pass is recomputed from the stored input and then the backward pass is computed. In our Optimus training library, • We implemented two reference huggingface models allenai/OLMo-1B-hf and allenai/OLMoE-1B-7B-0924 for the dense and MoE models respectively with model parallelism (Tensor, Expert and Pipeline Parallelism). • We implemented gpipe [ 5 ] , 1f1b [ 8 ] , and interleaved-1f1b [ 9 ] schedules for pipeline parallelism (PP) and enabled support for doing expert load balancing auxiliary loss for MoE model training with PP. • We implemented selective activation checkpointing for three modules : norm layer, attention block, SparseMoE block with an option for selecting one or more modules during training. • We developed a drop-in optimized FastSparseMoEBlock module that significantly speeds up MoE model training when compared to the SparseMoEBlock module in Hugging face OLMoE model. (See Section 3.1) • We implemented sharded optimizer for AdamW and also developed EPSO, a novel expert parallelism (EP) aware sharded optimizer that results in better performance when EP is used for training. (See Section 3.2) 2 Results We first pretrained Mula-1B dense and Mula-7B-A1B MoE models on the 4 trillion token OLMoE-Mix-0924 dataset [ 2 ] . We then demonstrate MoE model scaling with model sizes ranging from 20B to 220B parameters, and compute scaling from 384 to 12288 GPU tiles. Mula models follow the same model architecture as OLMo [ 4 ] and OLMoE [ 2 ] for dense and MoE models respectively. Mula- Mula- Mula- Mula- Mula- 1B 7B-A1B 20B-A2B 100B-A7B 200B-A10B Layers 16 16 32 48 64 Hidden size 2048 2048 2048 3072 3072 Attention heads 16 16 16 24 24 Head size 128 128 128 128 128 Intermediate size 8192 1024 1024 1536 1536 Experts - 64 96 144 240 Chosen Experts - 8 8 8 8 Total parameters 1.3 B 6.9 B 20 B 100 B 220 B Active parameters 1.3 B 1.3 B 2.4 B 7.6 B 10 B Table 1 : Model configurations. 2.1 Mula-1B and Mula-7B-A1B We pretrained Mula-1B and Mula-7B-A1B on 256