Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
Haobo Yang · 2026
Read Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework by Haobo Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30440.txt.
Introduction
Haobo Yang presents a formal proof that transformer networks, under a stated condition, perform exact Bayesian inference. Transformers power modern language and vision models. Their attention steps resemble Bayesian updating in informal accounts. Such claims have mostly stayed at the level of analogy. Yang works in a measure-theoretic kernel framework instead. The condition is a Bayes joint-distribution requirement on internal updates.
A hierarchy of abstractions structures the claimed proof. It starts with a core Bayesian transformer of prior and likelihood. Semantic transformers with explicit update kernels come next. Full blocks with attention, residual, and feed-forward parts follow. Multilayer stacks close the hierarchy at the top. At each level the update kernel matches the posterior almost surely. A Radon–Nikodym argument yields the Bayes formula for blocks. Softmax weighting is shown to induce a valid probability distribution. Architectural assumptions stay minimal throughout the argument.
Catalog record
| CREATOR | Haobo Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30440
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework Thanks: This preprint has not undergone professional peer review. If you find any errors or have concerns, please contact the corresponding author. Haobo Yang Abstract We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions—from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks—and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update. Area of research: Probabilistic Machine Learning, Formal Methods, Bayesian Inference, Transformer Theory. 1 Introduction The transformer architecture 1 has become the dominant paradigm in modern machine learning, powering large language models, vision models, and multimodal systems. A growing body of theoretical work has explored connections between attention mechanisms and probabilistic inference 2 ; 3 ; 4 , suggesting that transformers may implement forms of Bayesian updating or posterior inference. However, these claims have largely remained at the level of intuition, analogy, or empirical observation, without formal rigorous proof. We close this gap by constructing a complete formal proof that transformer architectures implement exact Bayesian posterior inference, under a clearly stated joint-distribution condition. Our framework has three distinguishing features. First, the proofs are based on first principles and depend only on the kernel composition structure and the Bayes joint semantics condition—no architectural assumptions about depth, width, parameterization, or training procedure are required. Second, we provide a layered hierarchy of abstractions that mirrors the structural complexity of real transformers: (i) the core Bayesian transformer with only prior and likelihood, (ii) the semantic transformer with an explicit update kernel, (iii) the full transformer block with QKV/attention/residual/MLP data flow, and (iv) the multilayer transformer stack. At each level, we prove that the Bayes joint semantics implies the update is the posterior almost everywhere. Third, we expose the explicit block-level Bayes formula through Radon-Nikodym differentiation and prove its normalization, providing a concrete bridge from the abstract kernel composition to practical density-based inference. We additionally prove that softmax attention defines a valid probability distribution, connecting the kernel framework to the standard attention mechanism. The central result is that when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update. The paper is organized as follows. Section 2 presents the mathematical framework and defines the kernel-based transformer abstractions. Section 3 states and proves the core theorems at each level of the hierarchy. Section 4 derives the explicit Bayes formula for the block-level architecture. Section 5 proves softmax normalization and establishes the bridge to concrete attention. Section 6 discusses the scope, assumptions, and limitations of the framework. 2 Mathematical Framework 2.1 Probability Kernels and Bayesian Inference We work in the category of measurable spaces with Markov kernels as morphisms. All spaces are standard Borel, and all measures are \sigma -finite. We adopt standard kernel notation: \kappa:\operatorname{Kernel}XY is a probability kernel from X to Y , \kappa\circ_{\mu}\mu is measure-kernel composition, and \kappa_{1}\otimes_{\kappa}\kappa_{2} is the product kernel. Definition 2.1 (Bayesian Transformer) . A Bayesian transformer is a tuple (\Theta,\mathfrak{X},\pi,\ell) where: • \Theta is the latent/parameter space (standard Borel, nonempty); • \mathfrak{X} is the observation/data space; • \pi is a finite prior measure on \Theta ; • \ell:\operatorname{Kernel}\Theta\mathfrak{X} is a finite Markov kernel. Its Bayes method is the posterior kernel: \operatorname{posterior}(\ell,\pi):\operatorname{Kernel}\mathfrak{X}\Theta . Proposition 2.2 (Bayes Joint Law) . For any Bayesian transformer T=(\Theta,\mathfrak{X},\pi,\ell) , the Bayes method satisfies the joint-distribution identity: (\ell\circ_{\mu}\pi)\otimes_{\kappa}\operatorname{posterior}(\ell,\pi)=(\pi\otimes_{\kappa}\ell).\operatorname{map}\operatorname{swap}. (1) This proposition—that the posterior kernel, when composed with the prior, recovers the symmetrized joint law—is the fundamental identity of Bayesian inference and serves as the anchor for all subsequent transformer-level proofs. 2.2 Semantic Transformers: Adding an Update Kernel The core Bayesian transformer defines the posterior abstractly. A semantic transformer equips this structure with an explicit internal update kernel, and the key question becomes: when does this update equal the posterior? Definition 2.3 (Semantic Transformer) . A semantic transformer extends a Bayesian transformer T=(\Theta,\mathfrak{X},\pi,\ell) with an internal update kernel u:\operatorname{Kernel}\mathfrak{X}\Theta . Its Bayes joint semantics condition is: (\ell\circ_{\mu}\pi)\otimes_{\kappa}u=(\pi\otimes_{\kappa}\ell).\operatorname{map}\operatorname{swap}. (2) The Bayes joint semantics condition states that the transformer’s internal update, when composed with the prior and likelihood, reconstructs the correct symmetrized joint law. This is the sole structural condition required for all subsequent theorems. 2.3 Transformer Block with QKV/Attention/Residual/MLP A practical transformer block has a concrete data-flow pipeline: input embedding \to QKV projection \to attention \to residual/MLP \to readout. We formalize this as a composition of Markov kernels. Definition 2.4 (QKV State and Attention Trace) . Let Q,K,V be the query, key, and value type parameters. • \operatorname{QKVState}\;Q\;K\;V:=(Q\times K)\times V ; • \operatorname{AttentionTrace}\;X\;Q\;K\;V\;A:=(X\times\operatorname{QKVState}\;Q\;K\;V)\times A . The attention trace stores the input state, the QKV state, and the attention state, enabling the residual/MLP kernel to condition on all intermediate representations. This is the natural kernel-level encoding of the residual connection. Definition 2.5 (Block Likelihood) . For type parameters \Theta,X,Q,K,V,A,H,O , the block likelihood is the kernel composition: \displaystyle\operatorname{blockLikelihood} \displaystyle\;(\text{input}:\operatorname{Kernel}\Theta X)(\text{qkv}:\operatorname{Kernel}X(\operatorname{QKVState}\;Q\;K\;V)) \displaystyle\;(\text{attention}:\operatorname{Kernel}(X\times\operatorname{QKVState}\;Q\;K\;V)A) \displaystyle\;(\text{residualMLP}:\operatorname{Kernel}(\operatorname{AttentionTrace}\;X\;Q\;K\;V\;A)H) \displaystyle\;(\text{readout}:\operatorname{Kernel}HO):\operatorname{Kernel}\Theta O \displaystyle:=\text{readout}\circ_{\kappa}\text{residualMLP}\circ_{\kappa} \displaystyle\qquad((\text{input}\otimes_{\kappa}\operatorname{prodMkLeft}\;\Theta\;\text{qkv})\otimes_{\kappa}\operatorname{prodMkLeft}\;\Theta\;\text{attention}). (3) The composition reflects the exact data flow: the input kernel maps latent parameters to input states; the QKV kernel maps input states to query-key-value triples; the attention kernel sees both input and QKV states; the residual/MLP kernel sees the full attention trace; and the readout kernel produces the final output. 2.4 Multilayer Transformer Architecture Real transformers stack multiple blocks. We formalize this as a list of hidden-state kernels composed in reverse order (first layer first, last layer last). Definition 2.6 (Layer Stack) . For hidden-state type H , the layer stack \operatorname{layerStack} maps a list of kernels (\operatorname{Kernel}HH) to a single kernel: \displaystyle\operatorname{layerStack}\;[] \displaystyle:=\operatorname{id}, (4) \displaystyle\operatorname{layerStack}\;(\ell::\ell s) \displaystyle:=\operatorname{layerStack}\;\ell s\circ_{\kappa}\ell. (5) The architecture likelihood composes embedding, layer stack, and readout: \operatorname{architectureLikelihood}\;\text{emb}\;\text{layers}\;\text{readout}:=\text{readout}\circ_{\kappa}\operatorname{layerStack}\;\text{layers}\circ_{\kappa}\text{emb}. (6) 2.5 Softmax Attention The abstract kernel framework treats attention as a black-box Markov kernel. To bridge to concrete implementations, we formally define softmax and prove it produces a valid probability distribution. Definition 2.7 (Softmax) . For a nonempty finite index type \iota and score vector x:\iota\to\mathbb{R} , the softmax of index i is: \operatorname{softmax}\;x\;i:=\frac{\exp(x_{i})}{\sum_{j}\exp(x_{j})}. (7) 3 Core Theorems 3.1 Bayes Kernel Level: The Posterior Joint Law Theorem 3.1 (Posterior Joint Law) . For any Bayesian transformer T=(\Theta,\mathfrak{X},\pi,\ell) with finite prior and finite likelihood, (\ell\circ_{\mu}\pi)\otimes_{\kappa}\operatorname{posterior}(\ell,\pi)=(\pi\otimes_{\kappa}\ell).\operatorname{map}\operatorname{swap}. (8) Proof. This follows from the disintegration theorem for finite kernels, which constructs the posterior kernel via disintegration and proves the identity in full measure-theoretic generality. The proof requires only that \Theta is standard Borel, \pi is \sigma -finite, and \ell