GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu · 2026
Read GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization by Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22095.txt.
Introduction
Chuanyang Zheng and co-authors rethink where normalization layers belong in a Transformer. Designers must choose between normalizing before each block or after it, the Pre-Norm and Post-Norm layouts. Each has known quirks, and the choice has stayed an open question in architecture design.
The authors view the question through manifold optimization. The outputs of the attention and feed-forward layers are treated as update directions in an optimization problem living on a curved space. Their 2026 preprint, GeoNorm, uses geodesic optimization to unify the two layouts in one scheme. A principled settlement of this small design question could stabilize training across the huge models built on the Transformer pattern.
Catalog record
| CREATOR | Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu Abstract The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost. 1 Introduction Recent years have witnessed remarkable progress in Large Language Models (LLMs) ( 8 ; 59 ; 78 ) , driven primarily by the exponential growth of training data and model parameters. This advancement is reflected in substantial improvements across language modeling ( 20 ; 63 ; 31 ; 55 ; 43 ; 77 ) and computer vision ( 64 ; 42 ) . Within the Transformer architecture, the choice of normalization function, such as Pre-Norm and Post-Norm, is critical, as it influences training stability, model performance, and gradient flow. Currently, Pre-Norm has become the predominant choice in modern LLMs due to its favorable stability properties ( 43 ; 86 ; 31 ) . In contrast, Post-Norm ( 62 ; 69 ) , which applies normalization after the addition of the residual connection, was originally introduced in the Transformer and facilitates the training of deeper networks. However, it is often prone to training instability and loss spikes ( 80 ) . Pre-Norm was subsequently proposed to improve stability by applying layer normalization to the residual branch before the addition operation ( 58 ) . Nonetheless, Pre-Norm introduces imbalanced gradients, with lower layers tending to receive larger updates than higher ones ( 80 ) . To address these limitations, recent methods have sought to enhance stability and enable extreme model scaling. DeepNorm ( 80 ) stabilizes training and improves performance through better parameter initialization and residual scaling. Meanwhile, SandwichNorm ( 17 ; 87 ) introduces additional normalization layers at the network’s input and output. While these approaches demonstrate strong empirical results, they generally lack a unified theoretical framework for analyzing normalization functions in Transformers. In this work, we interpret the operations \bm{x}+\mathrm{FFN}(\bm{x}) and x+\mathrm{Attention}(\bm{x}) as optimization steps on a sphere, where \mathrm{FFN}(\bm{x}) and \mathrm{Attention}(\bm{x}) serve as estimated gradient directions produced by the respective modules. From this perspective, each Transformer layer can be viewed as performing an iterative optimization step within a dynamical system: token embeddings act as the initial state, while the attention and feed-forward modules generate update directions based on the current iterate. The normalization operator then projects the updated vector back onto a constraint set, specifically, the sphere defined by the normalization radius. Building on this formulation, our contributions are threefold: • Theoretical Framework: We introduce a theoretical framework for analyzing normalization in Transformers. Here, \bm{x} is treated as the current point, while \mathrm{FFN}(\bm{x}) and \mathrm{Attention}(\bm{x}) are interpreted as the update direction in optimization. • GeoNorm Normalization: We propose GeoNorm, a novel normalization function that replaces conventional schemes by leveraging geodesic and Riemannian optimization on the sphere, leading to improved performance. • Extensive Empirical Validation: Through comprehensive experiments across varying training lengths, datasets, and model sizes, we validate the effectiveness of GeoNorm. Additionally, we demonstrate its scalability and generalization on large-scale datasets with downstream task evaluation. 2 Related Work Normalization Function The original Post-Norm architecture, first introduced in the Transformer model ( 62 ; 69 ) , applies layer normalization to the sum of the identity term and the residual output. While Post-Norm facilitates the training of deep networks, it can suffer from instability and exhibit sharp loss spikes ( 80 ) . In response, the Pre-Norm variant ( 58 ) modifies the residual connection by applying layer normalization directly to the identity term before the addition, leading to improved training stability. However, this often results in imbalanced gradient magnitudes, with lower layers receiving larger gradients compared to higher layers ( 80 ) . To further enhance stability and enable scaling to significantly greater depths, DeepNorm ( 80 ) introduces refined parameter initialization and residual scaling. Another approach, StandwichNorm ( 17 ; 87 ) , incorporates normalization layers at both the network input and output alongside the identity connection. Although these methods have empirically demonstrated strong performance, they currently lack a comprehensive theoretical framework for analyzing the underlying normalization mechanisms. Gradient Descent. The stochastic gradient descent (SGD) ( 7 ) is an iterative optimization method with the gradient. Based on the SGD, there is SGD with momentum ( 65 ) which use the momentum of previous gradient. Adagrad (adaptive gradient algorithm) ( 18 ) is a modified stochastic gradient descent algorithm with per-parameter learning rate. The RMSProp ( 22 ) divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight. Adam ( 34 ) run averages with exponential forgetting of both the gradients and the second moments of the gradients are used. Optimization Method. The Geodesic optimization is related to the optimization on the manifold ( 34 ) . Riemannian Steepest Descent Method extends the classic gradient descent to optimize functions on curved spaces (manifolds) ( 15 ) by following the path of steepest decrease, using the Riemannian gradient (the projection of the standard gradient onto the manifold) and a retraction to map steps back onto the manifold. Riemannian Newton’s method ( 14 ) adapts classical Newton’s method for optimization problems on curved spaces (Riemannian manifolds). The Riemannian Conjugate Gradient method ( 26 ) extends the conjugate gradient method to a manifold. For the retraction relation method, they will calculate the gradient and map the result back to the manifold. The geodesic-based methods ( 91 ) use the gradient descent method on the geodesic of the manifold. 3 Method 3.1 Post-Normalization in LLMs Beyond the well-recognized attention and feed-forward network modules, modern LLMs rely critically on normalization to stabilize training and inference as depth increases. Let \bm{x}_{k} denote the token representation after the k -th Transformer layer. A Transformer layer with post-normalization can be written as \begin{split}&\tilde{\bm{x}}_{k}=\text{Norm}(\bm{x}_{k}+\text{Attn}(\bm{x}_{k})),\\ &\bm{x}_{k+1}=\text{Norm}(\tilde{\bm{x}}_{k}+\text{FFN}(\tilde{\bm{x}}_{k})),\end{split} (1) where \text{Attn}(\cdot) and \text{FFN}(\cdot) denote the attention and feed-forward modules, and \text{Norm}(\cdot) is the normalization operator (typically RMSNorm followed by rescaling to preserve the vector norm). From this formulation, we may view the Transformer layer as performing an iterative optimization step in a dynamical system. The token embeddings serve as the initial state, while the attention and feed-forward modules produce update directions based on the current iterate \bm{x}_{k} . The normalization operator then projects the updated vector back to a constraint set, specifically, the sphere defined by the normalization radius \|\bm{x}_{k}\| . This interpretation closely mirrors the projected optimization scheme: \bm{x}_{k+1}=\text{Proj}_{\bm{\Omega}}(\bm{x}_{k}+\alpha_{k}\bm{s}_{k}), (2) where \bm{s}_{k} is an update step determined by past iterates \{\bm{x}_{0},\bm{x}_{1},\cdots,\bm{x}_{k}\} , \alpha_{k} represents the