Gradient Descent as Implicit EM in Distance-Based Neural Models
Alan Oursland · 2025
Read Gradient Descent as Implicit EM in Distance-Based Neural Models by Alan Oursland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24780.txt.
Introduction
Alan Oursland gives a direct explanation for why distance-based neural models act like chance-based inference systems, showing soft clustering, model focus, and uncertainty tracking. Loose analogies to mixture models were the usual account. For any objective with a log-sum-exp form over spans or energies, he derives an exact identity.
The gradient for each distance equals the negative posterior responsibility of its part, the same quantity an expectation-maximization step would use. Gradient descent therefore acts as implicit expectation-maximization. Readers get a firm math bridge between neural training and classic data-based inference, rather than a suggestive resemblance between the two.
Catalog record
| CREATOR | Alan Oursland |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient Descent as Implicit EM in Distance-Based Neural Models Alan Oursland Abstract Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures—in attention mechanisms, classification heads, and energy-based models—yet existing explanations rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct derivation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component: ∂ L / ∂ d j = − r j \partial L/\partial d_{j}=-r_{j} . This is an algebraic identity, not an approximation. The immediate consequence is that gradient descent on such objectives performs expectation-maximization implicitly—responsibilities are not auxiliary variables to be computed but gradients to be applied. No explicit inference algorithm is required because inference is embedded in optimization. This result unifies three regimes of learning under a single mechanism: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. The Bayesian structure recently observed in trained transformers is not an emergent property but a necessary consequence of the objective geometry. Optimization and inference are the same process. 1 Introduction Neural networks trained with standard objectives repeatedly exhibit behaviors associated with probabilistic inference: soft clustering, prototype specialization, uncertainty tracking, and mixture-model dynamics. These phenomena appear across architectures—in attention mechanisms, classification heads, and energy-based models—yet their origin has remained unclear. Are they emergent properties of scale? Architectural accidents? Or something more fundamental? This paper argues that such behaviors are none of these. They are necessary consequences of the geometry of common objective functions. 1.1 The Puzzle Consider the range of phenomena that appear, unbidden, in trained neural networks. Attention heads in transformers learn to specialize, each routing information for distinct semantic roles. Classification networks partition representation space into regions that behave like mixture components. Deep networks trained on noisy data exhibit robustness patterns reminiscent of Bayesian inference, down-weighting outliers and tracking uncertainty across inputs. These behaviors emerge without explicit probabilistic modeling, without mixture-model architectures, and without any algorithm resembling expectation-maximization. The standard explanations are unsatisfying. One view holds that these are emergent properties of scale—that sufficient parameters and data somehow give rise to statistical structure. Another treats them as architectural coincidences, artifacts of specific design choices like softmax normalization or residual connections. A third offers loose analogies: attention is “like” soft clustering; cross-entropy “resembles” a mixture model. None of these explanations answers the deeper question: why do these specific behaviors appear, rather than others? And why do they appear so reliably, across such different architectures and tasks? 1.2 Recent Evidence Recent work by 2 ; 1 has sharpened this puzzle considerably. In controlled experimental settings—“Bayesian wind tunnels” where the true posterior is analytically known—small transformers reproduce Bayesian posteriors with sub-bit precision. This is not approximate Bayesian behavior; it is exact, verifiable, position-by-position agreement with the analytic solution. Capacity-matched MLPs trained under identical conditions fail completely, suggesting that the phenomenon depends on the inductive biases of attention rather than optimization alone. More striking still is what Aggarwal et al. find in the gradient dynamics. Attention weights stabilize early in training while value vectors continue to refine—a two-timescale structure that mirrors the E-step and M-step of classical expectation-maximization. Values receive updates weighted by attention, precisely as prototypes receive updates weighted by responsibilities in mixture models. The authors provide a complete first-order analysis showing that this structure is not incidental but systematic. Yet Aggarwal et al. explicitly characterize this EM connection as “structural rather than variational”—an analogy, not a derivation. They observe that attention behaves like EM but do not claim that EM is a necessary consequence of the objective. The question of why cross-entropy training produces Bayesian geometry remains, in their framing, open. 1.3 This Paper This paper closes that gap. We show that for objectives with log-sum-exp structure over distances or energies, the gradient of the loss with respect to each distance is exactly the posterior responsibility of the corresponding component. This is not an approximation, not a resemblance, not an analogy. It is an algebraic identity: \frac{\partial L}{\partial d_{j}}=-r_{j} (1) The immediate consequence is that gradient descent on such objectives is expectation-maximization, performed continuously rather than in discrete alternating steps. The forward pass computes unnormalized likelihoods; normalization yields responsibilities; backpropagation delivers responsibility-weighted updates to parameters. No auxiliary latent variables are introduced. No inference algorithm is invoked. The architecture does not approximate EM—it implements EM, because the gradient is the responsibility. This reframes the relationship between optimization and inference. Inference is not a separate algorithmic layer added on top of learning. It is not a post-hoc interpretation of learned representations. Under the objectives we analyze, inference and optimization are the same computation viewed at different levels of abstraction. The Bayesian structure observed by Aggarwal et al. is not an emergent property that happens to appear; it is forced by the geometry of the loss. 1.4 Contributions The contributions of this paper are deliberately narrow and can be summarized as three claims at increasing levels of abstraction. One theorem. For any objective of the form L=\log\sum_{j}\exp(-d_{j}) , the gradient with respect to each distance satisfies \partial L/\partial d_{j}=-r_{j} , where r_{j} is the normalized exponential—the responsibility of component j . This result is derived, not assumed, and requires no conditions beyond differentiability. One interpretation. This identity implies that gradient descent on distance-based log-sum-exp objectives performs implicit expectation-maximization. The E-step is the forward pass; the M-step is the parameter update; responsibilities are never explicitly computed because they are the gradients. EM is not approximated by neural training—it is realized by it. One unification. The same mechanism manifests across three regimes depending on the constraints imposed. In the unsupervised regime, responsibilities are fully latent and prototypes compete freely. In the conditional regime—attention—responsibilities are recomputed per query over a shared prototype family. In the constrained regime—cross-entropy classification—supervision clamps one responsibility while competition persists among alternatives. These are not three different phenomena. They are one phenomenon under different boundary conditions. 2 The Geometric Substrate The main result of this paper—that responsibilities emerge as gradients—rests on a specific geometric foundation. This section establishes that foundation. We first summarize the interpretation of neural network outputs as distances or energies rather than confidences, importing this result from prior work. We then define the class of log-sum-exp objectives to which our analysis applies. Finally, we briefly recall the structure of classical expectation-maximization, not because our method requires it, but to establish the reference point against which implicit EM will be compared. 2.1 Distance-Based Representations The standard interpretation of neural network outputs treats them as confidences or scores indicating the strength of evidence for a hypothesis. A logit is high when the network “believes” in a class; an attention score is high when a query “matches” a key. This interpretation, while intuitive, obscures the geometric structure of what neural networks actually compute. In prior work ( 6 ) , we established a different interpretation: the outputs of standard neural networks are better understood as distances or energies relative to learned prototypes. A linear layer followed by an absolute value