Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion
Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara · 2026
Read Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion by Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13470.txt.
Introduction
Ishtiaque and three colleagues prove that conditional diffusion models with finite Gaussian-mixture reverse steps can approximate well-behaved target distributions as closely as desired. The error splits into a terminal mismatch, which usually vanishes as the diffusion horizon grows, plus per-step kernel errors handled by ReLU-network bounds. Under exact terminal matching, their neural reverse-kernel class can approach any such distribution in conditional KL divergence.
Catalog record
| CREATOR | Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107600 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13470
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara Abstract We prove that conditional diffusion models whose reverse kernels are finite Gaussian mixtures with ReLU-network logits can approximate suitably regular target distributions arbitrarily well in context-averaged conditional KL divergence, up to an irreducible terminal mismatch that typically vanishes with increasing diffusion horizon. A path-space decomposition reduces the output error to this mismatch plus per-step reverse-kernel errors; assuming each reverse kernel factors through a finite-dimensional feature map, each step becomes a static conditional density approximation problem, solved by composing Norets’ Gaussian-mixture theory with quantitative ReLU bounds. Under exact terminal matching the resulting neural reverse-kernel class is dense in conditional KL. 1 Introduction A universality theorem asks whether an architecture is expressive enough, in principle, to approximate the objects it is meant to learn. In classical approximation theory, universal-approximation results for neural networks played exactly this role, while later refinements quantified the dependence on depth, width, and smoothness [ 1 , 2 , 3 , 4 ] . This paper asks the analogous question for a class of conditional diffusion models . Diffusion methods now form a central paradigm in generative modeling [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] . Yet in standard DDPM-style discretizations each reverse step is typically a single Gaussian with learned mean and sometimes learned variance [ 6 , 9 ] . Recent work argues that this ansatz can be too restrictive, especially in few-step regimes, and proposes richer reverse transitions based on Gaussian mixtures or related soft-mixture denoisers [ 15 , 16 , 17 ] . Our aim is to give a population-level expressivity theorem for that idea. The target is a conditional law c\mapsto\pi^{\star}(\cdot\mid c) , i.e. a Markov kernel from a context space \mathcal{C} to \mathcal{X}=\mathbb{R}^{D} , with error measured by \mathcal{E}_{\operatorname{KL}}(\pi):=\mathbb{E}_{C\sim\mathsf{P}_{\mathcal{C}}}\left[\operatorname{KL}(\pi^{\star}(\cdot\mid C)\,\|\,\pi(\cdot\mid C))\right]. (1.1) Thus the paper concerns approximation of conditional distributions, or equivalently Markov kernels, rather than approximation of a single unconditional density or score field. Our answer is affirmative for discrete-time conditional diffusions whose reverse kernels are finite Gaussian mixtures with fixed Gaussian components and feature-dependent logits realized by ReLU networks. The proof reduces the output conditional-KL to a terminal mismatch plus a sum of per-step reverse-kernel errors . Under a feature-sufficiency assumption, each step factors through feature space into a static conditional density approximation problem. At each reverse step we then use a two-level hierarchy: first approximate the true feature-space reverse density by a Gaussian mixture whose weights are target dependent and a priori unknowable (referred to as “infeasible” for that reason), then approximate the resulting logits by ReLU networks. This yields the error decomposition \text{terminal mismatch}+\textstyle\sum_{t=1}^{T}\left(\text{infeasible mixture error}_{t}+\text{neural logit error}_{t}\right). The mixture error is controlled through Norets’ conditional Gaussian-mixture theory [ 18 ] , and the logit error through the ReLU approximation bounds of [ 4 ] . Under a smooth-logit assumption, the resulting bounds are fully explicit. The main conclusions are: (i) a quantitative approximation result up to terminal mismatch (Proposition 4.1 ); (ii) under exact terminal matching, a universality theorem showing mean-KL denseness of the neural predictor class (Thm. 4.2 ); and (iii) high-probability and almost-everywhere consequences (Corollary 4.4 ). Scope of the result. The theorem proved here is an expressivity result for a class of conditional diffusion architectures. It does not address statistical estimation, optimization, or score approximation. Its role is to isolate representation error at the level of reverse-kernel parameterization, in the same spirit as classical universal-approximation theorems for feedforward networks [ 1 , 2 ] . The analysis relies on a feature-sufficiency assumption requiring each true reverse kernel to depend on its inputs only through a known finite-dimensional feature map; this is a modeling hypothesis, not derived from the forward process. The resulting rates are correspondingly structural rather than practically sharp, since in the present fully general ambient-space setting they inherit a substantial response-side curse of dimensionality. Relation to prior work. The paper sits at the intersection of three literatures. First, it is adjacent to the theory of score-based diffusion, which studies approximation and estimation of score fields, including recovery guarantees, minimax rates, low-dimensional adaptation, and conditional score estimation [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 14 ] . Our theorem concerns not the score class but the reverse-kernel class itself in discrete time. Second, it complements recent empirical and algorithmic work advocating richer-than-Gaussian reverse transitions [ 15 , 16 , 17 ] by supplying a population-level expressivity theorem. Third, the construction draws on the older literature on conditional density estimation and mixtures of experts: adaptive mixtures of local experts, mixture density networks, and hierarchical mixtures of experts provided early neural mechanisms for modeling conditional output laws [ 27 , 28 , 29 ] ; approximation-theoretic results appeared already in work of Jiang and Tanner [ 30 ] , and Norets later established strong Kullback–Leibler (KL) approximation theorems for smooth mixtures of regressions [ 18 ] . Our stepwise reverse model is a diffusion-era version of that viewpoint. In short, the paper imports the representation-theoretic viewpoint of universal approximation into conditional diffusion, replaces the usual score-centric perspective by a kernel-class perspective, and shows that a natural neural Gaussian-mixture reverse architecture is universal in conditional KL once the terminal mismatch is isolated. Organization. § 2 formulates conditional prediction as kernel approximation and proves the path-space KL decomposition. § 3 develops the stepwise Gaussian-mixture and neural approximation theory, including the general explicit bounds and the sharper C^{1} -branch rates. § 4 states the main universality theorem and its consequences. § 5 concludes. § A contains some technical proofs. § B exemplifies the general formalism by approximating a conditional Gaussian kernel with Ornstein-Uhlenbeck noising, with explicit parameters and error rates. 2 Conditional prediction and diffusion kernels 2.1 Prediction as kernel approximation For measurable spaces (\mathcal{A},\Sigma_{\mathcal{A}}) and (\mathcal{B},\Sigma_{\mathcal{B}}) , write \Delta(\mathcal{A}) and \mathsf{Ker}(\mathcal{A}\to\mathcal{B}) for the sets of probability measures on \mathcal{A} and Markov kernels K:\mathcal{A}\to\Delta(\mathcal{B}) respectively. Definition 2.1 (Probabilistic prediction problem) . Let (\mathcal{C},\Sigma_{\mathcal{C}}) and (\mathcal{X},\Sigma_{\mathcal{X}}) be standard Borel spaces, and let \mathsf{P}\in\Delta(\mathcal{C}\times\mathcal{X}) with marginal \mathsf{P}_{\mathcal{C}} on \mathcal{C} . By disintegration there exists a true conditional \pi^{\star}\in\mathsf{Ker}(\mathcal{C}\to\mathcal{X}) with \mathsf{P}(\mathrm{d}c,\mathrm{d}x)=\mathsf{P}_{\mathcal{C}}(\mathrm{d}c)\,\pi^{\star}(\mathrm{d}x\mid c) . The prediction problem is to approximate \pi^{\star} , \mathsf{P}_{\mathcal{C}} -a.e., in KL divergence. We take \mathcal{X}=\mathbb{R}^{D} with Lebesgue measure \operatorname{Leb} , assume \pi^{\star}(\cdot\mid c)\ll\operatorname{Leb} for \mathsf{P}_{\mathcal{C}} -a.e. c (writing the density as \pi^{\star}(x\mid c) ), and measure error by \mathcal{E}_{\operatorname{KL}}(\pi):=\mathbb{E}_{C\sim\mathsf{P}_{\mathcal{C}}}\left[\operatorname{KL}(\pi^{\star}(\cdot\mid C)\,\|\,\pi(\cdot\mid C))\right]. (2.1) 2.2 Forward and backward diffusion paths Fix diffusion horizon T\in\mathbb{N} and forward kernels q_{t}\in\mathsf{Ker}(\mathcal{X}\to\mathcal{X}) , t=0,\dots,T-1 , with strictly positive densities q_{t}(x_{t+1}\mid x_{t}) . Definition 2.2 (Forward path kernel) . Define q^{\pi^{\star}}\in\mathsf{Ker}(\mathcal{C}\to\mathcal{X}^{T+1}) by q^{\pi^{\star}}(\mathrm{d}x_{0:T}\mid c):=\pi^{\star}(\mathrm{d}x_{0}\mid c)\prod_{t=0}^{T-1}q_{t}(\mathrm{d}x_{t+1}\mid x_{t}), (2.2) \text{with density}\qquad q^{\pi^{\star}}(x_{0:T}\mid c)=\pi^{\star}(x_{0}\mid c)\prod_{t=0}^{T-1}q_{t}(x_{t+1}\mid x_{t}). (2.3) Write q_{t}^{\pi^{\star}}(\cdot\mid c) for the X_{t} -marginal of q^{\pi^{\star}}(\cdot\mid c) . Fix a reference law \nu\in\Delta(\mathcal{X}) with density \nu(x) ; this is the initialization of the reverse chain. In practice \nu is chosen so that q_{T}^{\pi^{\star}}(\cdot\mid c)\approx\nu for large T ; we track the mismatch explicitly. Definition 2.3 (Backward path kernel and induced predictor) . A reverse model is a family p_{t}\in\mathsf{Ker}(\mathcal{C}\times\mathcal{X}\to\mathcal{X}) , t=1,\dots,T , with positive