Mix-LN : Unleashing the Power of Deep Layers by Combining Pre-LN and Post-LN
Pengxiang Li *, Lu Yin, Shiwei Liu · 2024
Read Mix-LN : Unleashing the Power of Deep Layers by Combining Pre-LN and Post-LN by Pengxiang Li *, Lu Yin, Shiwei Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13795.txt.
Introduction
Pengxiang Li, Lu Yin, and Shiwei Liu offer a 2024 paper on how large language models are built. In these models, a step that scales values can sit before or after each layer. The common choice puts it before, but the authors show that this leaves the deep layers with weak training signals, so they add little.
Their fix mixes the two choices in one model. The early layers use the after form, and the later layers use the before form, which evens out the training signal. In tests with models from small to large sizes, the mixed form trained better. The work helps builders use all layers of a model well.
Catalog record
| CREATOR | Pengxiang Li *, Lu Yin, Shiwei Liu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mix-LN : Unleashing the Power of Deep Layers by Combining Pre-LN and Post-LN Pengxiang Li *, Lu Yin, Shiwei Liu Abstract Large Language Models (LLMs) have achieved remarkable success, yet recent findings reveal that their deeper layers often contribute minimally and can be pruned without affecting overall performance. While some view this as an opportunity for model compression, we identify it as a training shortfall rooted in the widespread use of Pre-Layer Normalization (Pre-LN). We demonstrate that Pre-LN, commonly employed in models like GPT and LLaMA, leads to diminished gradient norms in its deeper layers, reducing their effectiveness. In contrast, Post-Layer Normalization (Post-LN) preserves larger gradient norms in deeper layers but suffers from vanishing gradients in earlier layers. To address this, we introduce Mix-LN , a novel normalization technique that combines the strengths of Pre-LN and Post-LN within the same model. Mix-LN applies Post-LN to the earlier layers and Pre-LN to the deeper layers, ensuring more uniform gradients across layers. This allows all parts of the network—both shallow and deep layers—to contribute effectively to training. Extensive experiments with various model sizes from 70M to 7B demonstrate that Mix-LN consistently outperforms both Pre-LN and Post-LN, promoting more balanced, healthier gradient norms throughout the network, and enhancing the overall quality of LLM pre-training. Furthermore, we demonstrate that models pre-trained with Mix-LN learn better compared to those using Pre-LN or Post-LN during supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), highlighting the critical importance of high-quality deep layers. By effectively addressing the inefficiencies of deep layers in current LLMs, Mix-LN unlocks their potential, enhancing model capacity without increasing model size. Our code is available at https://github.com/pixeli99/MixLN . 1 Introduction Large Language Models (LLMs) have ushered in a new era of artificial intelligence by demonstrating unprecedented capabilities in understanding and generating human-like text ( Brown, 2020 ; Achiam et al., 2023 ; Touvron et al., 2023 ; Dubey et al., 2024 ) . Trained on vast datasets that span multiple languages and topics, LLMs are driving advancements across industries and academia, enhancing human-computer interactions, and fostering innovation in previously unimaginable ways. Recent studies reveal a critical observation regarding the effectiveness of deeper layers in LLMs, particularly those beyond the middle layers ( Jaiswal et al., 2024 ; Sun et al., 2025 ; Liu et al., 2022b ) . It has been shown that these deeper layers can often be pruned significantly ( Yin et al., 2023 ) , or even removed entirely ( Gromov et al., 2024 ; Men et al., 2024 ) , without notably affecting the model’s overall capabilities. Moreover, Li et al. (2024) demonstrated that deeper layers contribute minimally to performance during fine-tuning, further questioning their importance. Unfortunately, this finding has been largely overlooked by the research community, where many see it primarily as an opportunity for model compression ( Siddiqui et al., 2024 ; Zhong et al., 2024 ; Sreenivas et al., 2024 ) , rather than recognizing it as a potential shortfall in the training process. In this paper, we seek to challenge the prevailing notion that deeper layers in LLMs are of lesser significance. The training of LLMs is extraordinarily resource-intensive, often requiring thousands of GPUs or TPUs and several months of computation on vast datasets. For example, the training of GPT-3 reportedly incurred millions of dollars in computational costs. The underutilization of deeper layers leads to inefficiencies, squandering resources that could otherwise be leveraged to enhance model performance. Ideally, all layers in a model should be well-trained, with sufficient diversity in features from layer to layer, to maximize the utility of the network’s parameters ( Yang et al., 2023 ) . This makes it crucial to investigate the root causes of this underutilization and to develop strategies that fully capitalize on the potential of deeper layers, ensuring that the overall architecture is optimized for both performance and efficiency. Figure 1: (a) Post-LN layer; (b) Pre-LN layer; (c) Mix-LN layer. We hypothesize that the inefficiency of deeper layers in LLMs primarily stems from the choice of Layer Normalization. Specifically, Pre-Layer Normalization (Pre-LN) ( Dai, 2019 ; Baevski & Auli, 2018 ) tends to produce smaller gradients in deeper layers, thereby diminishing their effectiveness, while Post-Layer Normalization (Post-LN) ( Ba, 2016 ) results in larger gradients in deeper layers but leads to gradient vanishing in earlier ones. Most state-of-the-art LLMs, like GPT, LLaMA, and Mistral, employ Pre-LN, which contributes to the widespread assumption that deeper layers are inherently less effective. To validate this conjecture, we conduct experiments with the following two categories of LLMs and compare the effectiveness of layers across different depths in Pre-LN models and Post-LN models. • Open-weight large-scale LLMs: We select LLaMa2-7B ( Touvron et al., 2023 ) as a representative Pre-LN model and BERT-large ( Devlin, 2018 ) as a Post-LN model to evaluate the quality of their layers. Our findings confirm that the deeper layers of LLaMa2-7B exhibit high similarity, with their removal leading to minimal impact compared to the early layers. In stark contrast, BERT shows higher similarity among its first half, which contributes less to the model’s output. • In-house small-scale LLMs: To control for irrelevant confounding variables, we conduct a second set of experiments by training small-scale LLMs ourselves, ensuring that the only difference between the models is the choice of layer normalization. Consistent trends are observed in these experiments, reinforcing our earlier observations. Building on these insights, we propose a novel normalization technique, dubbed Mix-LN , which synergizes Pre-LN and Post-LN to achieve more balanced and healthier gradient norms across the network. Mix-LN applies Post-LN to the earlier layers and Pre-LN to the deeper layers. The rationale behind this is that Post-LN enhances gradient flow in the deeper layers, while Pre-LN stabilizes gradients in the earlier layers. By employing Post-LN in the initial layers and Pre-LN in the later layers, Mix-LN promotes healthier gradient norms in the middle and deeper layers, fostering more balanced training across the entire network and ultimately improving the model’s overall performance. Our extensive experiments, spanning models from 70M to 7B parameters, demonstrate that Mix-LN consistently outperforms Pre-LN, Post-LN, and their variants. Mix-LN not only avoids the training instability associated with Post-LN but also significantly improves the quality of deeper layers compared to Pre-LN, leading to better pre-training performance. Additionally, models pre-trained with Mix-LN demonstrate superior learning during Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) compared to those trained with Pre-LN or Post-LN, underscoring the importance of high-quality deep layers in LLMs. 2 Hypothesis Evaluation In this section, we will evaluate our hypothesis that the inefficiency of deeper layers in LLMs stems from the choice of Pre-LN. The evaluation details are described as follows. 2.1 Layer Normalization and Its Gradient Figure 1 (a) and (b) illustrate Post-LN and Pre-LN Transformer architectures, respectively. Formally, let us define x as the input, \mathcal{F}(x) as either a FFN layer or a multi-head attention layer, and \mathrm{LN}(\cdot) as the layer normalization. Post-LN applies \mathrm{LN}(\cdot) after the residual addition: \displaystyle\text{Post-LN}(x)=\mathrm{LN}(x+\mathcal{F}(x)). (1) In contrast, Pre-LN applies \mathrm{LN}(\cdot) before the residual addition: \displaystyle\text{Pre-LN}(x)=x+\mathcal{F}(\mathrm{LN}(x)). (2) We can calculate the derivatives of Equations ( 1 ) and ( 2 ), as follows: \displaystyle\frac{\partial\text{Post-LN}(x)}{\partial x} \displaystyle=\frac{\partial\mathrm{LN}(x+\mathcal{F}(x))}{\partial(x+\mathcal{F}(x))}\left(I+\frac{\partial\mathcal{F}(x)}{\partial x}\right), (3) \displaystyle\frac{\partial\text{Pre-LN}(x)}{\partial x}