AutoBayes: A Compositional Framework for Generalized Variational Inference
Toby St Clere Smithe, Marco Perin · 2025
Read AutoBayes: A Compositional Framework for Generalized Variational Inference by Toby St Clere Smithe, Marco Perin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18608.txt.
Introduction
Toby St Clere Smithe and Marco Perin present AutoBayes, a framework for building statistical models piece by piece, in a 2025 paper. Bayesian inference works out what hidden causes best explain data, but large models are hard to assemble and fit. The pair show that both exact inference and the shortcut methods used in practice obey a chain rule, much like the rule behind automatic gradient tools.
Building on that likeness, they offer a kit of parts. The kit holds tools to build a model, flip it around, attach a local score to each part, and expose settings. Even the final tuning game can be optimized locally, part by part. Worked examples from classic problems show how the kit fits together and where it can grow.
Catalog record
| CREATOR | Toby St Clere Smithe, Marco Perin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18608
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AutoBayes: A Compositional Framework for Generalized Variational Inference Toby St Clere Smithe, Marco Perin Abstract We introduce a new compositional framework for generalized variational inference, clarifying the different parts of a model, how they interact, and how they compose. We explain that both exact Bayesian inference and the loss functions typical of variational inference (such as variational free energy and its generalizations) satisfy chain rules akin to that of reverse-mode automatic differentiation, and we advocate for exploiting this to build and optimize models accordingly. To this end, we construct a series of compositional tools: for building models; for constructing their inversions; for attaching local loss functions; and for exposing parameters. Finally, we explain how the resulting parameterized statistical games may be optimized locally, too. We illustrate our framework with a number of classic examples, pointing to new areas of extensibility that are revealed. † † proceedings: : 1 Introduction Key to the success and vibrancy of deep learning is automatic differentiation ( autodiff ) and particularly its reverse mode , which allows the backpropagation of loss through arbitrary differentiable programs. In turn, key to automatic differentiation is the chain rule from calculus (and, in reverse mode, its transpose), by which the derivatives of composite functions may be computed from the derivatives of their parts. Mathematically, the chain rule follows from the functoriality of the (co)tangent bundle structure; in reverse mode, for differentiable maps X\xrightarrow{f}Y\xrightarrow{g}Z , we have \mathrm{d}_{x}(g\circ f)(z)=\mathrm{d}_{x}f\circ\mathrm{d}_{f(x)}g(z) , for any cotangent vector z to Z and point x in X . This story is of course well known. Less well known is that there is an analogous chain rule for Bayesian inference ( Braithwaite et al., 2023 ) , related to (but different from) the better known chain rule for entropy. The existence of this rule means that, to invert a complex model, it suffices to compose the inversions of its factors. And just as reverse mode automatic differentiation yielded a renaissance in differential machine learning, we believe that this chain rule for inference heralds a new era for Bayesian methods. Moreover, this new era can be built upon the advances in optimization due to deep learning, by focusing our efforts on variational methods. To this end, we show that the loss functions adopted in variational inference exhibit a compositional structure that is compatible with the Bayesian chain rule. Our work is therefore not only of theoretical interest, specifying as it does the computational structure of a powerful new framework for generalized variational inference. In order to carve nature cleanly and precisely at its joints, we make use of the mathematical language of category theory. This allows us: to separate the specification of a statistical model from that of its inversion; to enable models to make use of dependent types ; to define and compose loss functions locally while ensuring correctness; to distinguish the different roles that likelihood and regularization play; to annotate models with the families of distribution that they may yield; to separate syntax (model specification) from semantics (optimization); and to ensure that all of these components interact well with each other. All the same, we have done our best to minimize the demands of novel mathematics in the main text of this paper. Their authors being unaware of all this structure, existing frameworks are only able to approximate it. For example, various probabilistic programming languages (PPLs) allow the user to separate the specification of the model from its inversion (typically called a ‘guide’ ( Ritchie et al., 2016 ; Bingham et al., 2019 ) ), but there is typically only a weak structural coupling between these two ( Pham et al., 2024 ) . Moreover, because PPLs are typically aimed at describing and sampling from arbitrary distributions, the machinery involved in computing inversions is typically the same sampling procedure used for computing the models themselves: the weak coupling between the two means that the compositional structure cannot be fully exploited. In the variational setting, two conceptual frameworks get closer to the mark: the “generalized variational inference” of Knoblauch et al ( Knoblauch et al., 2019 ) , and the “Bayesian learning rule” of Khan et al ( Khan and Rue, 2023 ) . These works are notable for considering explicitly the structure of the loss functions involved in variational inference, but due to their focus on optimizing single (parametric families of) models rather than models in general, they miss the gains to be had from considering models themselves as compositional. This single-model focus of much of the statistical machine learning literature produces an excess of manual work, as researchers derive loss functions for complex models by hand. We advocate instead using modern mathematical and computational tools to automate this work, and thus improve modularity and reusability. Contributions 1. We describe a compositional framework for specifying probabilistic models, generalizing Bayesian networks and enabling models with dependent types. Our framework does not supplant PPLs: models can be composed of probabilitistic programs. 2. We make explicit the relationship between the specification of a model and its inversion, the compositional structure of which is given by the Bayesian chain rule. 3. We explain precisely how loss functions for complex models are obtained from simple parts, separating the likelihood terms from the differently-behaved regularizers. 4. We clearly separate the ‘syntax’ of model specification from the ‘semantics’ of optimization, as there are often multiple algorithms applicable to each model type. 5. To that end, we clarify how to annotate models with the specific families of distributions that they yield, observing that this expressivity is traded for compositionality. 6. Finally, optimization algorithms need parameters to act on, so we explain how these parameters may be exposed coherently with respect to the rest of the structure. Although we leave the implementation of the framework for future work, we demonstrate its utility by exhibiting a number of examples in Appendix § A . Notation We write f:X\to Y to denote deterministic maps (functions), and c:X\rightsquigarrow Y to denote stochastic maps (measure kernels). The latter yield measures for each element of their domain, which we write in conditional probability style, as c(\mathrm{d}y|x) . Thus, we denote sets and spaces with upper-case letters and elements of those spaces with corresponding lower-case letters. We will assume that each measurable space X is equipped with a canonical measure, denoted \mathrm{d}x . When a kernel c:X\rightsquigarrow Y is associated with a density function with respect to this canonical measure, we will denote it by p_{c}(y|x) , so that c(\mathrm{d}y|x)=p_{c}(y|x)\,\mathrm{d}y . We use \circ and \bullet to denote the composition of functions and kernels respectively, with the latter defined by the Chapman-Kolmogorov equation: X\overset{c}{\rightsquigarrow}Y\overset{d}{\rightsquigarrow}Z is given by (d\bullet c)(\mathrm{d}z|x)=\int_{y:Y}d(\mathrm{d}z|y)\,c(\mathrm{d}y|x) . We use 1 to denote a singleton set, and note that measures are equivalent to kernels with domain 1 . The pushforward of a measure \pi along a kernel c is written c_{*}\pi and defined by composition: c_{*}\pi=c\bullet\pi . The Bayesian inversion of a kernel c with respect to a prior \pi is a kernel (almost surely unique) in the opposite direction, which we denote c^{\dagger}_{\pi}:Y\rightsquigarrow X . We write [x=y] to denote the indicator function defined as 1 when x=y and 0 otherwise. 2 Beyond Bayesian Networks A Bayesian