Information Geometry of Variational Bayes
Mohammad Emtiyaz Khan · 2025
Read Information Geometry of Variational Bayes by Mohammad Emtiyaz Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15641.txt.
Introduction
Mohammad Emtiyaz Khan, writing alone, links two fields that grew up apart. Variational Bayes is a fast way to guess a probability spread when the exact one is too hard. Information geometry studies curved spaces of such spreads. Khan shows that, under set conditions, a variational answer always needs a special kind of slope, called a natural gradient.
He draws out what this link means through one learning rule. Bayes' rule itself becomes a sum of these slopes. A class of curved stand ins used in slope based training gets a wider form. The rule also scales to variational training of large language models. Khan stresses the shared roots of the two fields to spur more joint work.
Catalog record
| CREATOR | Mohammad Emtiyaz Khan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Information Geometry of Variational Bayes Mohammad Emtiyaz Khan Abstract We highlight a fundamental connection between information geometry and variational Bayes (VB) and discuss its consequences for machine learning. Under certain conditions, a VB solution always requires estimation or computation of natural gradients. We show several consequences of this fact by using the natural-gradient descent algorithm of Khan and Rue 2023 called the Bayesian Learning Rule (BLR). These include (i) a simplification of Bayes’ rule as addition of natural gradients, (ii) a generalization of quadratic surrogates used in gradient-based methods, and (iii) a large-scale implementation of VB algorithms for large language models. Neither the connection nor its consequences are new but we further emphasize the common origins of the two fields of information geometry and Bayes with a hope to facilitate more work at the intersection of the two fields. keywords Bayesian Inference, Variational Inference, Natural Gradient Descent 1 Introduction The information geometry of variational Bayes (VB) ( Saul et al. 1996 ; Bishop 2006 ) is a relatively less studied area compared to maximum-likelihood estimation ( Amari 1998 ) . Some of the early works on natural gradients for VB were proposed as early as the 90s ( Amari 1995 ; Sato 1999 ; Sato 2001 ) but they have only been popularized recently, for instance, by Honkela et al. 2010 and Hoffman et al. 2013 , among others. These works clearly show that the two fields of VB and information geometry are inseparable. Our goal in this paper is to highlight this fundamental connection and show its impact on machine learning. Both the connection and its consequences are based on existing works. The goal of this work is to further emphasize the common origins of the two fields of information geometry and Bayes. Our hope is that the connection will facilitate more work at the intersection of the two fields. We start in Section 2 with some background on VB and then, in Section 3 , illustrate the special role of information geometry and natural gradients for VB. In Section 4 , we present the Bayesian Learning Rule (BLR) ( Khan and Rue 2023 ) which exploits the power of natural gradients to optimize the VB objective. Subsequently, we discuss three consequences due to the connections between information geometry and VB. 1. In Section 5 , we show that Bayes’ rule for conjugate models can be equivalently written as addition of natural gradients. 2. In Section 6 , we show that the natural gradients used in the BLR generalize the quadratic surrogates used in optimization methods. 3. In Section 7 , we highlight the impact of the above results on scaling VB to large deep networks (such as GPT-2) which was not possible until recently ( Shen et al. 2024 ) . Throughout, we emphasize the role played by information geometry in VB. Our hope is that these results will facilitate future research at the intersection of the two fields of information geometry and Bayesian learning. 2 Variational Bayesian Learning Variational Bayesian (VB) inference aims to approximate the posterior distribution by using an optimization problem over the space of distributions. For example, consider a Bayesian model with likelihood p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$}) over the data {\cal D} with parameter vector \mbox{$\mbox{$\boldsymbol{\theta}$}$}\in\real^{P} and a prior p_{0}(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) . For such models, Zellner 1988 shows that the posterior distribution p(\mbox{$\mbox{$\boldsymbol{\theta}$}$}|\mbox{${\cal D}$}) obtained by using Bayes’ rule can also be recovered by solving an optimization problem over the space \mathcal{P} of all possible distributions over \boldsymbol{\theta} , p(\mbox{$\mbox{$\boldsymbol{\theta}$}$}|\mbox{${\cal D}$})=\frac{p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$})p_{0}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})}{p(\mbox{${\cal D}$})}=\argmin_{q\in\mathcal{P}}\,\,\mathbb{E}_{q}\left[-\log p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$})\right]+\mathbb{D}_{\text{KL}}[q\,\|\,p_{0}]. (1) The optimization in the last expression is over distributions q(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\in\mathcal{P} , and the second term there denotes the Kullback-Leibler (KL) divergence between q(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) and p_{0}(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) . VB restricts the optimization problem to approximate the posterior within a subset \mathcal{Q}\subset\mathcal{P} of candidate distributions, for example, \mathcal{Q} can be the set of all Gaussian distributions. Below, we will write the VB formulation in a general form using a loss function, which is suited for machine learning and deep learning problems. Essentially, we rearrange the objective by expanding the KL term and clubbing the likelihood and prior together as shown below: \mathbb{E}_{q}\left[-\log p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$})\right]+\mathbb{D}_{\text{KL}}[q\,\|\,p_{0}]=\mathbb{E}_{q}\left[-\log\left(p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$})p_{0}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\right)\right]-\mathcal{H}(q). The second term on the right is the entropy denoted by \mathcal{H}(q) . Then, by defining the loss as \bar{\ell}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})=-\log\left(p(\mbox{${\cal D}$}|\mbox{$\mbox{$\boldsymbol{\theta}$}$})p_{0}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\right) , the optimization problem in Eq. 1 can be rewritten as the following VB problem over the subset of distributions \mathcal{Q} , q_{*}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})=\argmin_{q\in\mathcal{Q}}\,\,\mathbb{E}_{q}\left[\bar{\ell}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\right]-\mathcal{H}(q). (2) We refer to this as the VB learning problem (as opposed to VB inference). The form applies more generally than VB inference, extending to cases when \bar{\ell} does not correspond to a ‘proper’ Bayesian model. Such cases are often covered under a larger Bayesian framework that targets a generalized posterior which is proportional to \exp(-\bar{\ell}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})) with an arbitrary loss ( Zhang 1999 ; Catoni 2007 ; Bissiri et al. 2016 ) . 3 Information Geometry of VB Solutions The goal of this paper is to highlight a fundamental connection between variational Bayes and information geometry. We will now show that computation of any q_{*} must involve computation of natural gradients, which are gradients scaled by a Fisher matrix. We believe that this result holds for general posterior forms, but here we will focus on cases where \mathcal{Q} is restricted to regular and minimal exponential-family members with respect to a fixed sufficient statistic \mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) as shown below: q_{\boldsymbol{\lambda}}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})=h(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\exp\left[\langle\boldsymbol{\lambda},\mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\rangle-A(\boldsymbol{\lambda})\right]. (3) The candidates are parameterized through \boldsymbol{\lambda}\in\Omega which is called the natural parameter and takes values in a non-empty open set \Omega . The cumulant function A(\boldsymbol{\lambda}) is finite, strictly convex and differentiable over \Omega . Further, \langle\cdot,\cdot\rangle is an inner product and h(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) is a function of \boldsymbol{\theta} . The family is log-linear, that is, the log of q_{\boldsymbol{\lambda}}(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) is linear in \mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$}) . Natural gradients naturally arise for VB with exponential-family posterior distributions. This is because the derivative of the Shannon entropy \mathcal{H}(q_{\boldsymbol{\lambda}}) with respect to natural parameters \boldsymbol{\lambda} necessarily involves the Fisher matrix of q_{\boldsymbol{\lambda}} . This is derived below (by assuming h(\mbox{$\mbox{$\boldsymbol{\theta}$}$})=1 for simplicity), \begin{split}\nabla_{\boldsymbol{\lambda}}\mathcal{H}(q_{\boldsymbol{\lambda}})&=-\nabla_{\boldsymbol{\lambda}}\mathbb{E}_{q_{\boldsymbol{\lambda}}}[\log q_{\boldsymbol{\lambda}}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})]\\ &=-\nabla_{\boldsymbol{\lambda}}\mathbb{E}_{q_{\boldsymbol{\lambda}}}[\langle\boldsymbol{\lambda},\mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})\rangle-A(\boldsymbol{\lambda})]\\ &=-\nabla_{\boldsymbol{\lambda}}\left[\langle\boldsymbol{\lambda},\nabla_{\boldsymbol{\lambda}}A(\boldsymbol{\lambda})\rangle-A(\boldsymbol{\lambda})\right]\\ &=-[\nabla_{\boldsymbol{\lambda}}^{2}A(\mbox{$\mbox{$\boldsymbol{\lambda}$}$})]\,\mbox{$\mbox{$\boldsymbol{\lambda}$}$}-\nabla_{\boldsymbol{\lambda}}A(\boldsymbol{\lambda})+\nabla_{\boldsymbol{\lambda}}A(\boldsymbol{\lambda})\\ &=-[\nabla_{\boldsymbol{\lambda}}^{2}A(\mbox{$\mbox{$\boldsymbol{\lambda}$}$})]\,\mbox{$\mbox{$\boldsymbol{\lambda}$}$}.\end{split} (4) The second equality is obtained by simply substituting the definition of q_{\boldsymbol{\lambda}} . The third equality is due to a well known property of exponential-families that the expected value of sufficient statistic is simply the derivative of the cumulant function, that is, \mathbb{E}_{q_{\boldsymbol{\lambda}}}[\mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})]=\nabla_{\boldsymbol{\lambda}}A(\boldsymbol{\lambda}) . The expectation \mathbb{E}_{q_{\boldsymbol{\lambda}}}[\mbox{$\mbox{$\mathbf{T}$}$}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})] is also known as the expectation parameter, as we will see in the next section where we will denote it by \mbox{$\mbox{$\boldsymbol{\mu}$}$}(\mbox{$\mbox{$\boldsymbol{\lambda}$}$}) . The fourth line applies the derivative, and the last line is obtained by cancelling the last two terms. For exponential-families, the Fisher matrix is \mbox{$\mbox{$\mathbf{F}$}$}(\mbox{$\mbox{$\boldsymbol{\lambda}$}$})=\nabla_{\boldsymbol{\lambda}}^{2}A(\boldsymbol{\lambda}) , which shows that the gradient of the entropy always contains this matrix. Using this result, we can obtain the optimality condition of VB by simply setting the derivative of the objective at the optimal q_{*} to zero. Denoting the natural parameter of q_{*} by \mbox{$\mbox{$\boldsymbol{\lambda}$}$}_{*} and rewriting q_{*} as q_{\boldsymbol{\lambda}_{*}} , the optimality condition becomes \nabla_{\boldsymbol{\lambda}_{*}}\mathbb{E}_{q_{\boldsymbol{\lambda}_{*}}}[\bar{\ell}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})]=\nabla_{\boldsymbol{\lambda}_{*}}\mathcal{H}(q_{\boldsymbol{\lambda}_{*}}) . Then we substitute Eq. 4 in the right hand side to get \boldsymbol{\lambda}_{*}=\mbox{$\mbox{$\mathbf{F}$}$}(\boldsymbol{\lambda}_{*})^{-1}\nabla_{\boldsymbol{\lambda}_{*}}\mathbb{E}_{q_{*}}[-\bar{\ell}(\mbox{$\mbox{$\boldsymbol{\theta}$}$})]. (5) The quantity \mbox{$\mbox{$\mathbf{F}$}$}(\mbox{$\mbox{$\boldsymbol{\lambda}$}$})^{-1}\nabla_{\boldsymbol{\lambda}} on the right is a natural gradient due to the preconditioning by the inverse Fisher. The equation shows that the optimal natural