Thermodynamically Optimal Regularization under Information-Geometric Constraints

Laurent Caraffa · 2026

Read Thermodynamically Optimal Regularization under Information-Geometric Constraints by Laurent Caraffa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17330.txt.

Introduction

Laurent Caraffa asks if common training tricks share one deep rule. Machine learning uses weight decay, dropout, and moving averages to keep models in check. Each trick works, yet each rests on its own story. At the same time, training big models burns more and more energy. Caraffa asks if learning nears a hard efficiency bound.

He frames best regularization under information-geometric constraints. That frame ties the tricks to limits from thermodynamics. If such a bound holds, it would guide greener training. It would also unite split tricks in one theory. The work is a step toward learning rules set by physics, not just trial and error.

Catalog record

CREATORLaurent Caraffa
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thermodynamically Optimal Regularization under Information-Geometric Constraints Laurent Caraffa Abstract Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques, such as weight decay, dropout, and exponential moving averages. At the same time, the rapidly increasing energetic cost of training large models raises the question of whether learning algorithms approach any fundamental efficiency bound. In this work, we propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization. Under three explicit assumptions — (A1) that optimality requires an intrinsic, parametrization-invariant measure of information, (A2) that belief states are modeled by maximum-entropy distributions under known constraints, and (A3) that optimal processes are quasi-static — we prove a conditional optimality theorem. Specifically, the Fisher–Rao metric is the unique admissible geometry on belief space, and thermodynamically optimal regularization corresponds to minimizing squared Fisher–Rao distance to a reference state. We derive the induced geometries for Gaussian and circular belief models, yielding hyperbolic and von Mises manifolds, respectively, and show that classical regularization schemes are structurally incapable of guaranteeing thermodynamic optimality. We introduce a notion of thermodynamic efficiency of learning and propose experimentally testable predictions. This work provides a principled geometric and thermodynamic foundation for regularization in machine learning. 1 Introduction Machine learning has achieved remarkable empirical success over the past decades, yet its theoretical foundations remain fragmented. Regularization techniques such as weight decay, dropout, batch normalization, early stopping, and exponential moving averages are widely used, but their theoretical justifications are partial, domain-specific, and often post hoc. There is no unifying principle explaining why these methods work together, nor any guarantee that they are optimal with respect to a fundamental criterion. In parallel, the energetic cost of training modern models has grown dramatically, with large-scale systems requiring millions to billions of dollars in compute expenditure. This trend raises a fundamental question: Is there a theoretical lower bound on the energy required to learn, and how close are current methods to that bound? In physics, the Landauer principle establishes a universal lower bound on the energy required to erase one bit of information, namely k_{B}T\ln 2 joules per bit. This result is independent of implementation and applies to any physical system performing irreversible information processing. If learning can be understood as a process of information acquisition and elimination, then its energetic cost should be constrained by such thermodynamic limits. At the same time, information geometry provides a unique intrinsic geometry on spaces of probability distributions. The Fisher–Rao metric is, up to scale, the only Riemannian metric invariant under sufficient statistics and reparametrizations, as established by Čencov’s theorem. This metric underlies natural gradient methods and plays a central role in statistical inference. This work unifies these perspectives. We ask: If one demands thermodynamic optimality of learning under explicit and reasonable assumptions, what geometric and algorithmic structure is implied? Our main contribution is to show that, under three assumptions, thermodynamically optimal regularization is uniquely characterized by squared Fisher–Rao distance to a reference belief state. Classical regularization schemes, such as Euclidean weight decay, are shown to be structurally incompatible with this optimality requirement. Contributions. Our main contributions are: 1. We formalize three explicit assumptions connecting thermodynamic optimality, intrinsic information measures, and quasi-static processes. 2. We prove a conditional optimality theorem establishing Fisher–Rao geometry and Fisher–Rao regularization as necessary under these assumptions. 3. We derive the induced geometries for Gaussian and circular belief models, yielding hyperbolic and von Mises manifolds. 4. We introduce a notion of thermodynamic efficiency of learning and show that classical regularization schemes cannot guarantee optimality. 5. We propose experimentally testable predictions and diagnostic quantities. 2 Background 2.1 The Landauer Principle The Landauer principle states that any logically irreversible operation that erases one bit of information must dissipate at least k_{B}T\ln 2 joules of energy, where k_{B} is Boltzmann’s constant and T is the temperature of the environment. Formally, E_{\text{erase}}\geq k_{B}T\ln 2. (1) This bound is universal and independent of the physical substrate. It has been experimentally verified, notably by Bérut et al. (2012), and forms a cornerstone of the thermodynamics of computation. 2.2 Information Geometry and the Fisher–Rao Metric Information geometry studies the differential-geometric structure of families of probability distributions. Given a parametric family \{p(x|\theta)\} , the Fisher information matrix is defined as I_{ij}(\theta)=\mathbb{E}_{x\sim p(\cdot|\theta)}\left[\frac{\partial\log p(x|\theta)}{\partial\theta_{i}}\frac{\partial\log p(x|\theta)}{\partial\theta_{j}}\right]. (2) This matrix defines a Riemannian metric on parameter space known as the Fisher–Rao metric. Čencov’s theorem states that, up to a multiplicative constant, the Fisher–Rao metric is the unique Riemannian metric on the space of probability distributions that is invariant under sufficient statistics and Markov morphisms. 2.3 Maximum Entropy Principle The maximum entropy principle, introduced by Jaynes, asserts that among all distributions satisfying given constraints, one should choose the distribution with maximal entropy, as it makes the least additional assumptions beyond the known information. For example, under constraints on mean and variance, the maximum entropy distribution is Gaussian. 2.4 Local Relation between KL Divergence and Fisher–Rao Distance For two nearby distributions p and p+dp , the Kullback–Leibler divergence admits the expansion D_{\mathrm{KL}}(p\|p+dp)=\frac{1}{2}d_{F}^{2}(p,p+dp)+O(\|dp\|^{3}), (3) where d_{F} denotes Fisher–Rao distance. Thus, Fisher–Rao distance locally measures information divergence. 3 Assumptions We make explicit the assumptions underlying our framework. Assumption 1 (Intrinsic Measure of Information) . Thermodynamic optimality requires an intrinsic, parametrization-invariant measure of information. Fundamental quantities such as minimal energy cost should not depend on arbitrary choices of coordinates or parameterizations. Assumption 2 (Maximum Entropy Belief States) . Belief states of a learning system are modeled by maximum-entropy distributions subject to known constraints. Assumption 3 (Quasi-Static Processes) . Thermodynamic optimality corresponds to quasi-static (infinitesimally slow) processes. Real learning dynamics may deviate from this idealization, but the quasi-static regime defines the fundamental lower bound on dissipation. These assumptions are not theorems; they are explicit modeling choices. Our results are conditional upon their acceptance. 4 Main Result We now state and prove our central theorem. Theorem 1 (Conditional Optimality of Fisher–Rao Regularization) . Under Assumptions 1–3, thermodynamically optimal regularization is uniquely characterized by minimizing squared Fisher–Rao distance to a reference belief state. In particular: 1. The metric on belief space must be the Fisher–Rao metric. 2. The optimal regularization functional is \mathcal{L}_{\mathrm{reg}}=d_{F}^{2}(q,q^{*}) . 3. The minimal energy dissipated by regularization is E_{\min}=k_{B}T\ln 2\cdot D_{\mathrm{KL}}(q\|q^{*}) . Proof. Step 1: Uniqueness of Fisher–Rao. By Assumption 1, the metric measuring information change must be intrinsic and invariant under reparametrization. By Čencov’s theorem, the Fisher–Rao metric is the unique (up to scale) Riemannian metric with this property. Therefore, the metric must be Fisher–Rao. Step 2: Geometry of belief space. By Assumption 2, belief states are modeled by maximum-entropy distributions under constraints. For fixed mean and variance, this yields Gaussian distributions; other constraints yield other exponential families. In all cases, the intrinsic geometry is induced by the Fisher–Rao metric. Step 3: Quasi-static optimality. By Assumption 3, thermodynamic optimality corresponds to quasi-static processes. The infinitesimal information erased along a path \gamma in belief space is proportional to the Fisher–Rao line element ds_{F} . The total information erased is therefore I_{\text{total}}=\int_{\gamma}ds_{F}, (4) which is minimized by geodesics in the Fisher–Rao geometry. Thus, the optimal regularization path minimizes Fisher–Rao distance. Step 4: Energy bound. By the Landauer principle, the minimal energy dissipated is E_{\min}=k_{B}T\ln 2\cdot I_{\text{total}}. (5) Locally, I_{\text{total}} corresponds to KL divergence, yielding the

Read the complete text