IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning
Xiandong Zou, Pan Zhou · 2025
Read IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning by Xiandong Zou, Pan Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25262.txt.
Introduction
Xiandong Zou, Jia Li, Xiaotong Yuan, and Pan Zhou propose IBNorm. Their 2025 paper offers a new normalization method. Normalization rescales values inside a neural network. It keeps training stable and smooth. Common methods include BatchNorm, LayerNorm, and RMSNorm. Those methods focus mainly on spread of values. They do not control what information representations keep.
IBNorm draws on the information bottleneck principle. That principle favors keeping facts that predict answers. It also favors dropping noise and useless variation. IBNorm adds bounded compression steps toward that goal. The authors prove stronger theory results for it. They claim tighter generalization bounds than rival methods. Tests cover language models like LLaMA and GPT-2. Tests also cover vision models like ResNet and ViT. In those tests, IBNorm beat the standard methods.
Catalog record
| CREATOR | Xiandong Zou, Pan Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25262
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning Xiandong Zou, Pan Zhou Abstract Normalization is fundamental to deep learning, but existing approaches such as BatchNorm, LayerNorm, and RMSNorm are variance-centric by enforcing zero mean and unit variance, stabilizing training without controlling how representations capture task-relevant information. We propose IB-Inspired Normalization (IBNorm), a simple yet powerful family of methods grounded in the Information Bottleneck principle. IBNorm introduces bounded compression operations that encourage embeddings to preserve predictive information while suppressing nuisance variability, yielding more informative representations while retaining the stability and compatibility of standard normalization. Theoretically, we prove that IBNorm achieves a higher IB value and tighter generalization bounds than variance-centric methods. Empirically, IBNorm consistently outperforms BatchNorm, LayerNorm, and RMSNorm across large-scale language models (LLaMA, GPT-2) and vision models (ResNet, ViT), with mutual information analysis confirming superior information bottleneck behavior. Code will be released publicly. 1 Introduction Normalization has become a cornerstone of deep learning, credited for stabilizing and accelerating training across domains. Techniques such as Batch Normalization (BN) ( Ioffe & Szegedy 2015 ) , Layer Normalization (LN) ( Ba et al. 2016 ) , and RMSNorm ( Zhang & Sennrich 2019 ) are now standard in architectures spanning vision and language ( Vaswani et al. 2017 ; Dosovitskiy et al. 2020 ; He et al. 2016a ) , underscoring normalization as a key driver of training stability and acceleration. Despite this success, existing normalization methods share a fundamental limitation: they are inherently variance-centric . By enforcing zero mean and unit variance, followed by rescaling and shifting, they improve the conditioning of the optimization problem ( Huang et al. 2023 ; Huang et al. 2018 ; He et al. 2016b ; Desjardins et al. 2015 ) . However, this process operates purely on first- and second-order statistics, without offering any guidance on (intermediate) representation learning. Indeed, two representations may share identical mean and variance, but can encode drastically different amounts of task-relevant information. This raises a key question: can normalization be re-designed not only to stabilize training but also to shape representations toward sufficiency and generalization? Early attempts have explored this direction. For instance, Eftekhari & Papyan 2025 encouraged Gaussian-like features via power transforms and additive noise, arguing that Gaussianity aids representation and noise acts as implicit regularization. However, such approaches lack a principled information-theoretic foundation and fail to explicitly balance preserving predictive information against suppressing nuisance factors, often yielding suboptimal embeddings for downstream tasks. Contributions. In this work, we move beyond variance-centric normalization and introduce IBNorm , a simple yet powerful normalization method grounded by the Information Bottleneck (IB) principle ( Shamir et al. 2010 ; Tishby & Zaslavsky 2015 ; Shwartz-Ziv & Tishby 2017 ) . Moreover, we also theoretically justify the superiority of IBNorm and its better generalization performance. First, we introduce an IB perspective to normalization, and propose IBNorm for generalization improvement. The IB principle seeks representations that preserve task-relevant information while discarding nuisances, thereby connecting normalization to generalization. Within this framework, we decompose existing normalization methods into three steps: (i) grouping features (e.g., across batch, channel, or dimension), (ii) standardization, and (iii) recovery via re-scaling and shifting. Our analysis reveals that the normalization operation itself is the key bottleneck governing information flow. Building on this insight, IBNorm (IBNorm-S, IBNorm-L, IBNorm-T) introduces compression operators that selectively compress activations toward their means, providing explicit control over sufficiency–redundancy tradeoffs while remaining drop-in compatible with existing architectures. Second, we provide theoretical guarantees. Under the IB framework, we prove that IBNorm achieves a strictly larger IB value than variance-centric methods like LN and BN, thereby better balancing predictive sufficiency with nuisance suppression. Moreover, we establish a provably tighter generalization bound, explaining why IBNorm consistently outperforms standard normalization in practice. Extensive experiments show the effectiveness of IBNorm across architectures and modalities. In language modeling, integrating IBNorm into LLaMA ( Touvron et al. 2023b ) and GPT-2 ( Radford et al. 2019 ) consistently outperforms LN, RMSNorm, and NormalNorm on LLM Leaderboard I/II by up to 8.75%. In vision, applying IBNorm to ResNet ( He et al. 2016a ) and ViT ( Dosovitskiy et al. 2020 ) yields substantial gains, e.g., 3.98% improvement on ImageNet ( Deng et al. 2009 ) (ResNet-50), and up to 8.17% improvement on ImageNet (ViT). 2 Related Work Normalization plays a central role in modern deep learning, enabling faster convergence and improved generalization across natural language processing (NLP) ( Devlin et al. 2019 ; Vaswani et al. 2017 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Dubey et al. 2024 ; Bai et al. 2023 ; Liu et al. 2024 ) and computer vision (CV) ( He et al. 2016a ; Dosovitskiy et al. 2020 ; Liu et al. 2022 ) . Batch Normalization (BN) ( Ioffe & Szegedy 2015 ) normalizes activations across mini-batches to stabilize training, with numerous extensions such as Mean-only BN ( Salimans & Kingma 2016 ) , L^{p} -Norm BN ( Liao et al. 2016 ) , Conditional BN ( De Vries et al. 2017 ) , and Whitening BN ( Siarohin et al. 2018 ) . Layer Normalization (LN) ( Ba et al. 2016 ) was proposed for sequential and small-batch settings and has inspired variants including Dynamic LN ( Kim et al. 2017 ) , RMSNorm ( Zhang & Sennrich 2019 ) , and Adaptive LN ( Xu et al. 2019 ) , which are now standard in Transformers ( Vaswani et al. 2017 ; Dosovitskiy et al. 2020 ) . Group Normalization (GN) ( Wu & He 2018 ) and Batch Group Normalization (BGN) ( Summers & Dinneen 2019 ) extend this idea by combining grouping across features, channels, or batches. Collectively, these methods underscore normalization as a structural tool for scaling depth and stabilizing optimization. Beyond variance-centric approaches, some works attempt to explicitly shape activation distributions. For example, Eftekhari & Papyan 2025 encourage Gaussian-like features via power transforms with additive noise, arguing that Gaussianity enhances representational capacity while noise acts as implicit regularization. However, this approach faces key limitations: it does not optimize mutual information with targets, incurs overhead in estimating power parameters, and introduces stochasticity that can destabilize training (see Appendix B ). Broadly, most normalization methods rely on re-centering and rescaling to mitigate internal covariate shift, yet they overlook the information-theoretic properties of representations, such as sufficiency and redundancy. This gap motivates our work. We propose IBNorm , which augments conventional normalization with compression operation designed to regulate information flow, guiding activations toward IB-optimal representations that enhance both generalization and robustness. 3 Preliminary Here we introduce information bottleneck (IB) framework ( Tishby et al. 2000 ; Shamir et al. 2010 ; Tishby & Zaslavsky 2015 ) which forms the basis of our approach in Sec. 4 . Consider two continuous random variables X and Y with joint probability densities p(x,y) over the space \mathcal{X}\times\mathcal{Y} . Then, we can define the mutual information between X and Y : I(X;Y)=\int_{x\in\mathcal{X}}\int_{y\in\mathcal{Y}}p(x,y)\,\log\frac{p(x,y)}{p(x)\,p(y)}\,dx\,dy. (1) Mutual information measures dependencies between random variables, and can be understood as how much knowing X reduces the uncertainty in Y or vice versa. Building on this, the IB framework formulates representation learning as a trade-off between two goals: 1) minimality which compresses X by reducing