Why Self-Supervised Encoders Want to Be Normal
Yuval Domb · 2026
Read Why Self-Supervised Encoders Want to Be Normal by Yuval Domb (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27743.txt.
Introduction
Domb shows that self-taught nets prefer normal curves, following the Information Bottleneck rule. Framing the Info Bottleneck goal as a rate-distortion problem gives a basis for optimal, target-neutral latent codes. They tend toward bell-shaped Gaussian states. Latent codes group inputs sharing forecast curves in a simplex structure. The framework joins taught and less-taught goals and explains smoothing. It derives usable loss goals matching this structure and offers a lens on collapse and codes.
Catalog record
| CREATOR | Yuval Domb |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83682 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27743
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Self-Supervised Encoders Want to Be Normal Yuval Domb Abstract Self-supervised learning has achieved remarkable empirical success in learning robust representations without explicit labels, most recently demonstrated within the framework of Joint-Embedding Predictive Architectures (JEPA). However, a fundamental question remains: what analytical principles drive these encoders toward specific distributional states? In this paper, we demonstrate that the preference for normal distributions in self-supervised encoders is a direct consequence of the Information Bottleneck (IB) principle. By recasting the IB objective as a rate-distortion problem over the predictive manifold, we provide a theoretical basis for why optimal, target-neutral, latent representations should tend towards isotropic Gaussian states. Under this framework, we show that latent representations correspond to soft clustering of inputs sharing similar predictive distributions, organized within a natural simplex structure. This perspective unifies a wide range of existing supervised and less-supervised objectives and provides a principled explanation for commonly used regularization schemes. Furthermore, we derive practical loss objectives that approximate this structure and demonstrate their effectiveness on standard benchmarks. Ultimately, our framework offers a geometric lens to understanding representation collapse and it establishes a mathematical system for regularization strategies to be used to ensure high-entropy, informative embeddings in modern self-supervised models. 1 Introduction Learning compact representations that preserve predictive information is a central problem in machine learning. Given observations X and targets Y , the goal is to learn a latent variable W that compresses X while retaining enough information to predict Y . The IB method [ 40 ] formalized this tradeoff by minimizing the mutual information I(X;W) subject to a constraint on the predictive information I(W;Y) . This paper develops a geometric and information-theoretic framework for encoder-decoder learning, built on the IB principle. Starting from the rate-distortion equivalence of Harremöes and Tishby [ 26 ] , with Kullback-Leibler (KL) divergence as the distortion measure, we characterize the optimal representation geometrically as a soft clustering of the predictive manifold \mathcal{M}=\{p(Y|x):x\in\mathcal{X}\} 1 1 1 We use the term “manifold” loosely. \mathcal{M} is the image of x\mapsto p(Y|x) and may have singularities, self-intersections, or varying dimension. . A key outcome is a theoretical justification of the Sketched Isotropic Gaussian Regularization (SIGReg) method [ 6 ] as a relaxation of the maximum entropy principle on the predictive manifold, mediated by a chain of transformations from the flat Dirichlet distribution to the isotropic Gaussian. SIGReg, together with the Conditional Entropy Bottleneck (CEB) decomposition of Fischer [ 20 ] , yields concrete encoder losses for supervised and semi-supervised settings. In the self-supervised setting, SIGReg serves as the distributional regularizer alongside a view-prediction proxy. Y X W enc min I(X;W) max I(W;Y) W\!\in\!\mathcal{P}_{K} W\!\sim\!\mathcal{N}(0,I) Figure 1: Overview from left to right. (a) The IB optimal encoder. (b) Soft clustering in the probability simplex. (c) Max entropy for less supervised settings. The framework applies to both discrete and continuous variables. For discrete Y with K outcomes, the predictive manifold lives in the probability simplex \mathcal{P}_{K} . For continuous Y , we introduce the effective predictive dimension and show that the manifold has finite effective dimension bounded by the intrinsic dimension of X . 1.1 Contributions 1. Geometric framework for optimal IB representations. Building on the rate-distortion equivalence of Harremöes and Tishby [ 26 ] and the tutorial treatment of Goldfeld and Polyanskiy [ 23 ] , we assemble known ingredients, including the optimal encoder form, the canonical simplex representation, invariance under prediction-preserving transformations, and the dimensionality bound \dim(W^{*})\leq K{-}1 (see Corollary 3 ), into a unified geometric picture: the optimal encoder at any distortion level performs a soft clustering of the predictive manifold \mathcal{M}\subset\mathcal{P}_{K} , with a linear decoder in the canonical parameterization (see Section 3.4 ). We extend the dimensionality statement to continuous Y via covering numbers (see Proposition 4 ). The individual results are not new. The contribution is the synthesis, which makes contributions 2 and 3 below concrete. 2. Dirichlet-to-Gaussian chain and characterization of SIGReg. We derive a chain of exact transformations, \mathrm{Dir}(1,\ldots,1)\to K i.i.d. \mathrm{Exp}(1/2)\to 2K i.i.d. \mathcal{N}(0,1) , that connects the maximum entropy prior on \mathcal{P}_{K} to an isotropic Gaussian in Euclidean space, with quantified entropy overhead at each step (see Section 5.3 ). This identifies SIGReg [ 6 ] as a Gaussian relaxation of the flat Dirichlet via the Cramér-Wold theorem, with a non-diminishing phase entropy overhead of K\log 2\pi nats that affects rate accounting but not achievable prediction (see Section 5.4 ). 3. Non-parametric encoder losses across supervision regimes. Using the CEB decomposition of Fischer [ 20 ] , which rewrites the IB Lagrangian as conditional rate I(X;W|Y) plus total rate I(X;W) (eq. 49 ), we derive encoder losses for the supervised and semi-supervised settings, estimated via minibatch marginals rather than variational bounds (Sections 6.1 – 6.2 ). In the self-supervised setting, where no labels are available, the CEB conditional rate is replaced by a view-prediction proxy and serves as the distributional regularizer (see Section 6.3 ). 1.2 Prior Work The Information Bottleneck. The optimization problem underlying the IB method was first formulated by Witsenhausen and Wyner [ 43 ] , who studied \inf_{p(w|x):\,H(X|W)\geq\alpha}H(Y|W) , in the context of common information [ 22 ] for discrete random variables. Tishby, Pereira, and Bialek [ 40 ] independently introduced the same problem as a principled framework for extracting relevant information, coining the term “information bottleneck.” The optimal encoder satisfies self-consistent equations solvable by the Blahut-Arimoto algorithm [ 10 , 5 ] , originally developed for rate-distortion computation. The connection between IB and rate-distortion theory was made explicit by Harremöes and Tishby [ 26 ] , who showed that the IB is a rate-distortion function with KL divergence as distortion. We build on this equivalence and develop its geometric consequences. IB and deep learning. Goldfeld and Polyanskiy [ 23 ] provide a comprehensive tutorial on IB theory and its applications to machine learning, covering the discrete-variable formulation in detail. Shwartz-Ziv and Tishby [ 38 ] studied the information plane dynamics of deep networks during training. Saxe et al. [ 35 ] showed that the reported compression phase does not hold in general. Achille and Soatto [ 2 ] develop an IB-flavoured account of representation learning in which invariance to nuisances emerges from a variational IB objective. Our treatment is complementary: non-variational, and with an explicit geometric characterization of the optimum on the probability simplex. Alemi et al. [ 3 ] introduced VIB, using variational bounds (a fixed prior to upper-bound I(X;W) and a learned decoder to lower-bound I(W;Y) ) to make the IB objective tractable for neural networks. Fischer [ 20 ] introduced CEB, which rewrites the IB Lagrangian as I(X;W|Y)-\gamma\,I(W;Y) via the chain rule of mutual information (equivalent to our eq. ( 49 ) with \gamma=\beta-1 ), and demonstrated benefits for calibration and adversarial robustness. Fischer used variational bounds in practice. We estimate both CEB terms via minibatch marginals rather than variational surrogates. Subsequent work extended the variational IB approach to nonlinear settings [ 30 ] and connections to PAC-Bayes generalization bounds [ 45 ] . Strouse and Schwab [ 39 ] studied the deterministic IB, which replaces the rate I(X;W) with the marginal entropy H(W) and thereby admits deterministic encoders by construction. The resulting frontier is in the (H(W),I(W;Y)) plane rather than