Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

Steven Y. Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, NVIDIA, Stanford University, Boston University [email protected] · 2024

Read Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two-Phase Pretraining by Steven Y. Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, NVIDIA, Stanford University, Boston University [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15285.txt.

Introduction

Steven Feng and co-authors train language models in two phases in a 2024 paper. How training data are picked, mixed, and ordered shapes the end model. Key mix facts stay hidden, since labs share few details of their blends.

The team ran a wide test of mixes for each phase of training. Two phase runs beat both chance order and raw data flow by clear margins. Good blends hinge on source quality and on how often data repeat. Blends tuned on small runs scaled to far larger runs and models. The paper gives builders a step by step path to strong data mixes.

Catalog record

CREATORSteven Y. Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, NVIDIA, Stanford University, Boston University [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Maximize Your Data’s Potential: Enhancing LLM Accuracy with Two-Phase Pretraining Steven Y. Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, NVIDIA, Stanford University, Boston University [email protected] Abstract Pretraining large language models effectively requires strategic data selection, blending and ordering. However, key details about data mixtures especially their scalability to longer token horizons and larger model sizes remain underexplored due to limited disclosure by model developers. To address this, we formalize the concept of two-phase pretraining and conduct an extensive systematic study on how to select and mix data to maximize model accuracies for the two phases. Our findings illustrate that a two-phase approach for pretraining outperforms random data ordering and natural distribution of tokens by 3.4 3.4 % and 17 17 % on average accuracies. We provide in-depth guidance on crafting optimal blends based on quality of the data source and the number of epochs to be seen. We propose to design blends using downsampled data at a smaller scale of 1T tokens and then demonstrate effective scaling of our approach to larger token horizon of 15T tokens and larger model size of 25B model size. These insights provide a series of steps practitioners can follow to design and scale their data blends. 1 Introduction Large language models (LLM) are typically pretrained on large amounts of data in the order of billions (B) or trillions (T) of tokens derived from multiple data sources such as web crawl, books, papers, patents, mathematical and legal documents, and so forth Brown et al. 2020 ; Parmar et al. 2024b ; Team et al. 2024b ; Dubey et al. 2024a ; Nvidia et al. 2024 . To develop a state-of-the-art model, it is critical to understand the nature of these data sources and to make informed decisions about optimal data blending (how different data sources are weighed during pretraining) and training strategies. These decisions typically involve running multiple large-scale experiments to empirically investigate the optimal training data blend(s) and ordering of data. Figure 1: Diagram of our two phase training pipeline. Phase-1 blend encourages data diversity and phase-2 blend is focused on high quality datasets. Most advanced models OpenAI et al. 2024 ; Dubey et al. 2024b do not divulge information on the data blends that are used, nor the ablation studies informing the data mixing and ordering decisions. Recent works Blakeney et al. 2024 ; Groeneveld et al. 2024 ; Dubey et al. 2024b ; Snowflake 2024 provide high-level data blend information about a small portion of pretraining by encouraging the upsampling of certain domains towards the end. In general, there exists a knowledge gap regarding how to craft and choose an optimal data blend(s) for the entire training process, and the generalizability of data blends and ordering strategies to larger token horizons and model sizes. In this work, we address the above knowledge gap by understanding optimal data blends and ordering strategies for training LLM. We formalize and extensively explore a two-phase training approach (Figure 1 ) that balances diversity and quality: phase-1 emphasizes diverse, high-quality web crawl data, while phase-2 focuses on high-quality data sources such as math, code, and wiki data. Specifically, in this work we propose to use downsampled data to prototype and explore multiple blends at a smaller scale of 1T tokens. We craft our blends based on quality of the data source and the number of epochs to be seen during pretraining. We then demonstrate the effectiveness of our approach at a 15T token scale using the full data. We evaluated on a comprehensive set of downstream tasking covering knowledge, reasoning, coding and math benchmarks. Our experiments illustrate that a quality and epoch based blend is better than a blend based on natural distribution by 13.2 % and the two-phase approach is better than random ordering of data (blend is based on quality and epochs) by an average of 3.4 % across downstream tasks. Furthermore, our results on downsampled data generalize across longer 15T token horizons on full data and larger model sizes, demonstrating the scalability and robustness of the two-phase approach. We also provide a fine-grained quality analysis of web crawl data, revealing optimal blending strategies to balance diversity and quality. We share and highlight a series of findings made to create blends and order in our two-phase approach. Our main contributions are: • Formalization and large-scale evaluation of the two-phase training approach for LLMs, with actionable strategies that enable effective LLM pretraining. • Improving the understanding of data selection and blending with quality-based and epoch-based analyses of data, including web crawl. • Demonstration of the scalability of blends using downsampled data at 1T to using full data at 15T tokens and larger model size of 25B. 2 A Two-Phase Approach to Pretraining In this work, we explore a two-phased approach to pretraining: phase-1 ( \mathcal{P}_{1} ) then phase-2 ( \mathcal{P}_{2} ). Figure 1 demonstrates our two-phased approach. In each phase, we explore different data blends based on the quality and number of epochs to be seen of a data source. In phase-1 ( \mathcal{P}_{1} ), we explore a general data distribution which consists of a mix of web crawl data, medium-quality data, and low amounts of high-quality data. In phase-2 ( \mathcal{P}_{2} ), we explore a blend which includes task data and emphasizes high-quality datasets such as math, code, and high-quality web crawl (§ 5.1 ). As seen in Figure 1 , our model sees the first general data blend during \mathcal{P}_{1} for the majority of training, then a different data blend focused on high quality data during the shorter \mathcal{P}_{2} of training. The steps to create blends for \mathcal{P}_{1} and \mathcal{P}_{2} are: 1) Downsample a data source by a factor of f , 2) Estimate the quality of a data source (§ 5.1 ), 3) Estimate the epochs to be seen in the whole pretraining (§ 5.2 ) and finally 4) distribute the epochs appropriately in \mathcal{P}_{1} and \mathcal{P}_{2} (§ 3.2 ). The downsampling factor f is based on the final total token budget which we assume to be 15 T similar to Dubey et al. 2024b . Hence, for us f=1/15 i.e for each data source, the number of tokens available for pretraining is 1/15^{th} of the total token in that dataset. Downsampling helps to observe the impact of epochs of datasets at a smaller scale of 1T tokens and then can be used to scale the blend to a longer token horizon of 15T tokens using the full data. Baselines: Since our blends are based on quality and epoch based analyses of the data as well as the ordering of the data in the two phases, we consider the following two baselines: 1) Natural Distribution Blend (BASE-ND): This blend is based on ratio of the number of tokens available in each data source. The weight for each dataset is equal to the total number of tokens in that dataset divided by the sum of tokens available in all the datasets. This weighting is neither based on quality nor the epochs to be seen for the dataset. 2) Random Order Pretraining (BASE-RO): This blend is based on quality and epochs of each dataset but does not use two

Read the complete text