Thermodynamic Diffusion Inference with Minimal Digital Conditioning
Aditi De · 2026
Read Thermodynamic Diffusion Inference with Minimal Digital Conditioning by Aditi De (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14332.txt.
Introduction
De equates diffusion-model inference with overdamped Langevin motion, so a substrate holding the score function settles to the right output by thermodynamics alone. Two barriers blocked scale-up: non-local skip connections, which analog bases cannot show, and input conditioning with far too little signal. Hierarchical bilinear coupling turns U-Net skip connections into rank-k interactions from Gram matrices, using far fewer links. A 2,560-parameter digital interface overcomes conditioning, and the full system reaches 0.9906 decoder cosine similarity with roughly ten-million-fold energy savings over GPU inference.
Catalog record
| CREATOR | Aditi De |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21890 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Thermodynamic Diffusion Inference with Minimal Digital Conditioning Aditi De Abstract Diffusion-model inference and overdamped Langevin dynamics are formally identical. A physical substrate that encodes the score function therefore equilibrates to the correct output by thermodynamics alone, requiring no digital arithmetic during inference and potentially achieving a 10,000 × 10{,}000\times reduction in energy relative to a GPU. Two fundamental barriers have until now prevented this equivalence from being realized at production scale: non-local skip connections, which locally coupled analog substrates cannot represent, and input conditioning, in which the coupling constants carry roughly 2,600 × 2{,}600\times too little signal to anchor the system to a specific input. We resolve both obstacles. Hierarchical bilinear coupling encodes U-Net skip connections as rank- k k inter-module interactions derived directly from the singular structure of the encoder and decoder Gram matrices, requiring only O ( D k ) O(Dk) physical connections instead of O ( D 2 ) O(D^{2}) . A minimal digital interface —a 4-dimensional bottleneck encoder together with a 16-unit transfer network, totalling 2,560 parameters —overcomes the conditioning barrier. When evaluated on activations drawn from a trained denoising U-Net, the complete system attains a decoder cosine similarity of 0.9906 against an oracle upper bound of 1.0000, while preserving theoretical net energy savings of approximately 10 7 × 10^{7}\times over GPU inference. These results constitute the first demonstration of trained-weight, production-scale thermodynamic diffusion inference. 1 Introduction Artificial-intelligence inference is a structural, not merely an optimization, problem. Data-center electricity demand is projected to reach the total output of Japan by 2030. Although quantization, sparsity, and specialized silicon deliver roughly a 10\times improvement, the thermodynamic lower bound lies nine orders of magnitude lower. These are qualitatively distinct limits. For diffusion models the route to the thermodynamic limit is explicit. The reverse score-matching stochastic differential equation is formally identical to the overdamped Langevin equation once the potential is identified with the negative log-probability of the data distribution at time t . Every matrix multiplication performed on a GPU is therefore simulating a physical process that a suitably engineered analog substrate can execute at zero marginal energetic cost. Prior work has established this principle for simple cases: Whitelam demonstrated that a continuous oscillator network can generate MNIST digits directly from thermal noise, and Jelinčič et al. (Extropic) fabricated CMOS hardware that achieves a 10{,}000\times efficiency gain over GPU inference on binarized Fashion-MNIST [ 7 ] . Both works identified non-local information routing as the principal obstacle to scaling but proposed no solution; the problem has remained open. Production U-Nets introduce precisely this difficulty. Skip connections couple physically separated encoder and decoder modules; ablating them increases FID by more than 30 points. A locally connected substrate cannot accommodate such long-range interactions without O(D^{2}) wiring, which is intractable for the feature dimensions ( D\gtrsim 10^{4} ) typical of modern models. A second, equally fundamental barrier is input conditioning: when a Langevin substrate is driven solely through its fixed coupling constants, the bias signal is structurally 2{,}600\times too weak to distinguish one input from another, a direct consequence of the extreme eigenvalue concentration of the Gram matrices. Jelinčič et al. explicitly flag both barriers—non-local routing and input-dependent conditioning—as blocking problems for scaling thermodynamic inference beyond toy distributions. The present work resolves both. This paper presents complete, validated solutions to both problems and demonstrates them on activations extracted from a fully trained denoising U-Net—the first such results reported for production-scale thermodynamic diffusion inference. Figure 1 : System architecture. Left: Conventional GPU inference executes the full score network digitally at \sim 1–10 J per image. Right: A 2,560-parameter digital conditioning interface (0.032% of U-Net cost) computes the bias vectors \mathbf{b}_{\rm enc} and \mathbf{b}_{\rm dec} ; hierarchical bilinear skip coupling routes information physically; the Langevin substrate equilibrates under thermal noise to produce the denoised output (cosine similarity 0.9906 versus digital baseline). Theoretical net energy gain: \sim\!10^{7}\times . 2 Background The diffusion–Langevin identity. Score-based diffusion defines a forward SDE d\mathbf{x}=f(\mathbf{x},t)\,dt+g(t)\,d\mathbf{W} and learns to reverse it via d\mathbf{x}=\bigl[f(\mathbf{x},t)-g^{2}(t)\nabla_{\mathbf{x}}\!\log p_{t}(\mathbf{x})\bigr]dt+g(t)\,d\bar{\mathbf{W}}. (1) The overdamped Langevin dynamics of a particle in potential V_{\theta} read \dot{x}_{i}=-\mu\,\partial_{i}V_{\theta}+\sqrt{2\mu k_{\rm B}T}\,\eta_{i}(t) , with equilibrium distribution p(\mathbf{x})\propto\exp(-V_{\theta}/k_{\rm B}T) . Setting V_{\theta}\equiv-k_{\rm B}T\log p_{t}(\mathbf{x}) renders Eq. ( 1 ) exact. A substrate that physically realizes V_{\theta} therefore equilibrates to the correct output without any arithmetic during inference. Prior thermodynamic computing. Landauer and Bennett established the thermodynamic lower bound on computation. Contemporary realizations include probabilistic spin logic, thermodynamic linear algebra, and the two generative demonstrations most relevant here: Whitelam’s continuous oscillator network for MNIST and the CMOS hardware of Jelinčič et al. that attained a 10{,}000\times energy reduction on Fashion-MNIST. The non-local coupling problem. Production U-Nets connect encoder level \ell directly to decoder level \ell : h_{\ell}^{\rm dec}=\mathcal{F}_{\ell}(\mathcal{U}(h_{\ell+1}^{\rm dec}),h_{\ell}^{\rm enc}) . A locally coupled analog substrate cannot represent these non-local interactions without incurring O(D^{2}) wiring—an insurmountable cost for realistic feature dimensions. This is exactly the blocking problem identified by Jelinčič et al., who note that scaling beyond simple distributions requires architectural solutions that their hardware does not provide [ 7 ] . 3 Hierarchical Bilinear Skip Coupling Consider an encoder module with state \mathbf{x}\in\mathbb{R}^{D} and a decoder with state \mathbf{y}\in\mathbb{R}^{D} evolving under the joint potential V(\mathbf{x},\mathbf{y})=V_{\rm enc}(\mathbf{x})+V_{\rm dec}(\mathbf{y})+\mathbf{x}^{\top}\mathbf{J}_{\rm skip}\,\mathbf{y}. (2) The skip matrix \mathbf{J}_{\rm skip} must encode the U-Net skip connection using far fewer than D^{2} physical conductances. We exploit the low-rank singular structure already latent in the trained weights. Definition 1 (Hierarchical Bilinear Skip Coupling) . Let \mathbf{J}_{\rm enc}=\mathbf{W}_{\rm enc}^{\top}\mathbf{W}_{\rm enc}/(4k_{\rm B}T) and \mathbf{J}_{\rm dec}=\mathbf{W}_{\rm dec}^{\top}\mathbf{W}_{\rm dec}/(4k_{\rm B}T) , with SVDs U_{e}\Sigma_{e}V_{e}^{\top} and U_{d}\Sigma_{d}V_{d}^{\top} . The rank- k skip coupling is \mathbf{J}_{\rm skip}^{(k)}=\frac{1}{4k_{\rm B}T}\cdot\frac{\bigl(U_{e}^{(k)}(\Sigma_{e}^{(k)})^{1/2}\bigr)\bigl(U_{d}^{(k)}(\Sigma_{d}^{(k)})^{1/2}\bigr)^{\top}}{\|\cdot\|_{F}}. (3) This construction requires only O(Dk) memristor conductances (a simple coupling bus). In the linear regime ( J_{4}=0 ) the coupled equilibrium satisfies the block system \underbrace{\begin{pmatrix}\mathbf{A}&\mathbf{J}_{\rm skip}\\ \mathbf{J}_{\rm skip}^{\top}&\mathbf{B}\end{pmatrix}}_{\mathbf{M}}\begin{pmatrix}\mathbf{x}^{*}\\ \mathbf{y}^{*}\end{pmatrix}=\begin{pmatrix}\mathbf{b}_{\rm enc}\\ \mathbf{b}_{\rm dec}\end{pmatrix}, (4) where \mathbf{A}=2J_{2}\mathbf{I}+\mathbf{J}_{\rm enc}+\mathbf{J}_{\rm enc}^{\top} and \mathbf{B}=2J_{2}\mathbf{I}+\mathbf{J}_{\rm dec}+\mathbf{J}_{\rm dec}^{\top} . The unique solution \mathbf{M}^{-1}(\mathbf{b}_{\rm enc},\mathbf{b}_{\rm dec})^{\top} permits exact quantification of information flow across the skip channel. Proposition 1 . The relative decoder shift induced by \mathbf{J}_{\rm skip}^{(k)} is \rho_{\rm skip}=\mathbb{E}[\|\mathbf{y}^{*}-\mathbf{y}^{*}_{0}\|/\|\mathbf{y}^{*}_{0}\|] , where \mathbf{y}^{*}_{0} is the equilibrium obtained with \mathbf{J}_{\rm skip}=\mathbf{0} . At rank k=16 we measure \rho_{\rm skip}=7.48\% (analytical regime, D=128 , random weights) and \rho_{\rm skip}=12.74\% (trained U-Net, D=64 , real activations), with coefficient of variation <1.5\% across 64 test samples in both regimes. The remarkably low variance demonstrates that the skip coupling encodes an input-independent structural bias. Fixed analog hardware can therefore be fabricated once and reused across all inputs. The larger shift observed with trained weights confirms that training sharpens the encoder–decoder singular alignment that the construction exploits. 4 Input Conditioning: Signal-Deficit Barrier and Resolution The barrier. To place the equilibrium at a target activation \mathbf{x}^{*} the oracle bias \mathbf{b}_{\rm enc}=\mathbf{A}\mathbf{x}^{*} must be supplied. The most natural candidate constructed from the coupling constants alone is the naive bias \mathbf{b}_{\rm enc}^{\rm naive}=\mathbf{x}\mathbf{J}_{\rm enc}^{\top} . Figure 2 : Skip-coupling analysis (trained regime). Trained U-Net, D=64 , real correlated activations. Panel (a): coupling matrix \mathbf{J}_{\rm skip} . Panel (b): per-dimension decoder shift \Delta y_{j}^{*} (mean \pm std, 64 samples). Panel (c): \rho_{\rm skip} versus rank k . \rho_{\rm skip}=12.74\% , CV 1.4\% . The larger shift confirms that training strengthens encoder–decoder singular alignment. Proposition 2