Pier: Efficient Large Language Model pretraining with Relaxed Global Communication

Shuyuan Fan, Zhao zhang · 2025

Read Pier: Efficient Large Language Model pretraining with Relaxed Global Communication by Shuyuan Fan, Zhao zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17849.txt.

Introduction

Shuyuan Fan and Zhao Zhang present Pier, a training method that cuts the global communication slowing down pretraining of large language models. Synchronizing many processors often dominates cost, so Pier builds on an approach that syncs less often, with local training steps in between.

Two fixes to the outer update, warming up its momentum and then decaying it, keep training stable and preserve model quality. Across GPT models of several sizes, Pier speeds training several fold on large graphics clusters, with no loss on validation or on a suite of downstream tasks. The design also supports mixing data and tensor parallelism.

Catalog record

CREATORShuyuan Fan, Zhao zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pier: Efficient Large Language Model pretraining with Relaxed Global Communication Shuyuan Fan, Zhao zhang Abstract Global communication, such as all-reduce and all-gather, is the prominent performance bottleneck in large language model (LLM) pretraining. To address this issue, we present Pier, an efficient and scalable optimizer with relaxed global communication. Pier is built upon DiLoCo, which leverages an inner optimizer within groups of processors and an outer optimizer that requires global communication. To preserve the convergence and model performance, Pier incorporates two key techniques for the outer optimizer: momentum warmup and momentum decay. Pier employs an efficient and scalable system architecture to enable complex parallelization strategies in LLM pretraining. We examine the model performance and runtime reduction of Pier using the GPT model family (e.g., small, medium, XL, and 7B) and the OpenWebText dataset with a suite of thirteen downstream tasks. With data parallel strategy, Pier speeds up GPT-2 XL training by up to 2.7x-3.7x on 256 NVIDIA A100 GPUs and 1.2x-1.9x on 64 GH200 Superchips, respectively, without degradation of validation loss or downstream task performance. With data parallel and tensor parallel, Pier reduces the time cost GPT-2 7B model training by 54.5% on 128 A100s. I Introduction Major stakeholders are designing large language models (LLMs) with ever-increasing size, driven by the scaling law [ 12 ] , which states that the model’s capability improves with scale. The state-of-the-art LLMs [ 1 , 11 , 9 ] are trained with tens of thousands of high-end graphic processing units (GPUs) in complex distributed strategies, including data parallel [ 13 ] , tensor parallel [ 26 ] , pipeline parallel [ 10 , 19 ] , and context parallel [ 17 ] . Communication is the prominent bottleneck in LLM pretraining, as simply evidenced by the low scaling efficiency of 42.7% and 34.6% for GPT-2 1.5B training with 32 NVIDIA A100s and 64 NVIDIA GH200 superchips using the AdamW optimizer with standard data parallel, respectively (see AdamW curves in Figure 7 ). Fig. 1: Validation Loss Comparison between AdamW (8 GPUs, Fully Synchronized) and DiLoCo (8 Groups, 1 GPU per Group) During the Pretraining of GPT-2 XL Model. To reduce communication costs and boost scaling efficiency for LLM pretraining, researchers have proposed algorithms that fall into two categories. The first category lowers the communication volume via gradient sparsity [ 15 , 32 ] and quantization [ 6 ] . And the second category reduces the global communication frequency by exploiting optimizers such as local SGD [ 28 ] . With the hard constraints of model performance (i.e., validation loss and downstream task performance), methods in the first category exhibit modest improvements of 10-30% [ 27 , 32 ] . Although the methods with relaxed global communication show great potential in pretraining acceleration, they suffer from performance degradation. Among these methods, DiLoCo [ 5 ] introduces a two-layer optimization framework that partitions GPUs into groups. It utilizes an inner optimizer that only requires intra-group communication and an outer optimizer that requires inter-group (i.e., global) communication. However, our empirical evaluation in Figure 1 shows degraded validation loss and downstream task performance using the GPT-2 XL model and OpenWebText dataset [ 7 ] . Our analysis of DiLoCo reveals that the gradient oscillation at the switching point from AdamW to DiLoCo is the root cause of performance degradation. This insight motivates the design of Pier, which incorporates two key techniques: momentum warmup and momentum decay for the outer optimizer. The momentum warmup technique tracks the model changes as the outer optimizer’s gradients, for every few iterations during the so-called “lazy start” phase in DiLoCo, where the model is updated by using the same AdamW as baseline. After the “lazy start” phase, the momentum decay technique exploits a fixed schedule to gracefully migrate from AdamW to DiLoCo. To reduce the memory overhead in the outer optimizer, Pier offloads the model and optimizer states to host memory during inner loops. Pier enables the overlay of data parallel over tensor parallel to support larger models, where the outer step parallelizes communication and model updates. Pier is implemented upon the Megatron-LM framework. We evaluate the design effectiveness of Pier using the GPT-2 model family, including small (125M), medium (345M), XL (1.5B), and a 7B model with the OpenWebText dataset. We examine the convergence and downstream task performance of Pier using a suite of thirteen tasks. With the model performance guarantee, we conduct five studies on the scaling patterns of Pier. We first perform a weak scaling study to identify the global batch size boundary of Pier with a fixed token budget and group sizes. The second study explores the convergence boundary when increasing the synchronization interval of the outer optimizer. We then perform a strong scaling study to determine the scaling efficiency boundary of Pier with fixed global batch sizes and the number of groups. In the fourth experiment, we vary the group size (i.e., the number of GPUs in a group) and fix the global batch size to study the impact of the number of groups on training speed. In the fifth study, we perform a mixed parallelism scaling experiment to examine the scalability of Pier for larger models. We run the experiments on two GPU clusters: NERSC Perlmutter and TACC Vista. Pier shows comparable validation loss and downstream task performance with the baseline AdamW optimizer in standard data parallelism for GPT-2 small, medium, and XL. On Perlmutter, Pier speeds up GPT-2 small, medium, and XL training by 1.7x, 2.6x, and 2.7x with the maximum data parallel scale that guarantees model performance. If global communication is further relaxed, the speedup for GPT-2 XL can be as high as 3.7x on 256 A100s. On Vista, Pier is effective on GPT-2 XL with data parallel. The speedup is 1.2-1.9x on 128 GH200s. For GPT-2 7B training with data parallel and tensor parallel, Pier exhibits a speedup of 2.2x on 128 A100s. The contributions of this paper are: • The momentum warmup and momentum decay techniques that allow Pier to achieve comparable model performance as AdamW. • The efficient Pier design that combines data parallel and tensor parallel. • The comprehensive model performance evaluation with thirteen downstream tasks. • The quantified end-to-end LLM pretraining time improvement over two production supercomputers with different GPUs and interconnect architectures. II Background In this section, we introduce the basic concepts of distributed LLM pretraining strategies, the communication bottleneck due to bandwidth hierarchy, and the structure of the DiLoCo optimizer. II-A Distributed Model Training Modern LLM pretraining exploits complex parallelism strategies. 3D parallelism partitions a model across processors by layering data parallelism (DP) over pipeline and tensor parallelism (PP and TP). GPUs are partitioned into groups (Data Parallel Rank), and each group hosts a complete copy of the model. Inside each group, the model is spread across nodes by layers to form the pipeline stages; Inside each node, layers (tensors) are evenly distributed across multiple GPUs, a technique referred to as tensor parallelism. With the data parallel approach, the only communications are the broadcast at the start of training and the gradient exchange in each iteration. In contrast, 3D parallelism requires intra- and inter-node communication for both forward and backward computation. With the need for larger context

Read the complete text