Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
Injin Kong, Hyoungjoon Lee, Yohan Jo · 2026
Read Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement by Injin Kong, Hyoungjoon Lee, Yohan Jo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14368.txt.
Introduction
Kong, Lee and Jo ask where diffusion should operate inside a pretrained model, treating hidden-state interface choice as a design question. They score interfaces with three training-free geometric proxies: local compactness, global stiffness and effective rank. The score predicts diffusion bridgeability beyond layer depth alone. Their method, DiHAL, replaces the transformer prefix below an interface with diffusion, keeping the suffix and head. Geometry-chosen interfaces stay near validation-loss oracles and beat worst-layer controls. The bridge also beats a fixed replacement.
Catalog record
| CREATOR | Injin Kong, Hyoungjoon Lee, Yohan Jo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84340 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14368
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement Injin Kong, Hyoungjoon Lee, Yohan Jo Abstract Continuous diffusion language models lag behind autoregressive transformers, partly because diffusion is applied in spaces poorly suited to language denoising and token recovery. We propose DiHAL , a geometry-guided diffusion–transformer hybrid that asks where diffusion should enter a pretrained transformer. DiHAL scores layers with geometry-based proxies, selects a diffusion-friendly hidden-state interface, and replaces the lower transformer prefix with a diffusion bridge while retaining the upper layers and original LM head. By reconstructing the selected-layer hidden state rather than tokens, DiHAL avoids direct continuous-to-discrete recovery. Experiments on 8B-scale backbones show that the geometry score predicts effective shallow insertion layers under a fixed bridge-training protocol and that hidden-state recovery improves over continuous diffusion baselines in a diagnostic comparison matching the diffusion/recovery training budget. These results suggest that hidden–state geometry helps identify where diffusion–based replacement is feasible inside pretrained language models. 2 2 footnotetext: Corresponding author. 1 Introduction Large language models have achieved remarkable progress across a wide range of language generation tasks, but this progress has come with increasing size and computational cost ( 3 ; 7 ; 28 ) . Diffusion models offer a different generative paradigm based on iterative denoising and have become a dominant approach in image generation ( 23 ) . Their success has motivated growing interest in diffusion-based language generation ( 13 ; 24 ; 16 ) . However, transferring diffusion from images to text is difficult because text generation must ultimately handle discrete tokens. A natural response is to adapt diffusion to the discreteness of text. Prior work has explored discrete token corruption, masked diffusion, continuous-to-discrete recovery, and continuous diffusion over token embeddings, self-conditioned embeddings, or learned text latents ( 13 ; 24 ; 14 ; 5 ; 30 ) . Despite these efforts, diffusion-based language models still lag behind autoregressive Transformers, particularly in continuous diffusion settings ( 10 ) . A common explanation is that denoised continuous vectors must eventually be mapped back to discrete tokens, so small errors in representation space can change the recovered token ( 13 ) . Why does this gap remain? We start from a complementary hypothesis: discreteness is important but may not fully explain the gap. Transformer language models also use discrete tokens, yet most computation occurs in continuous hidden states later mapped to vocabulary logits ( 25 ) . This suggests that the difficulty may arise not from continuity itself, but from applying diffusion in continuous spaces with unsuitable geometry. If the choice of continuous space matters, then the central question becomes: what makes a representation space suitable for diffusion? We call such a space diffusion-friendly : a space that is easy to denoise, stable under imperfect score estimates, and simple enough for diffusion to learn. We later motivate these requirements using tools from Langevin dynamics and concentration theory ( 26 ; 2 ; 11 ) . Where can such a space be found in a language model? A pretrained Transformer already contains many continuous hidden spaces between the token embedding layer and the LM head. These hidden states are not decoded directly into tokens; they are consumed by the remaining Transformer layers before the LM head produces the final token distribution ( 25 ) . Diffusion at an internal layer can therefore target hidden-state recovery rather than direct token recovery ( 14 ; 18 ) . Since hidden-state geometry varies across depth, we ask: which transformer layer provides the most diffusion-friendly representation space? To answer this question, we propose DiHAL ( Di ffusion-Transformer H ybrid A rchitecture for L anguage Generation), a hybrid architecture based on a Locate-and-Replace strategy. As illustrated in Figure 1 , DiHAL locates diffusion-friendly layers using geometry-based criteria, then replaces the lower transformer layers with a diffusion bridge that reconstructs the selected-layer hidden state while retaining the upper layers and original LM head for token prediction. This reduces continuous-to-discrete recovery error and reframes continuous diffusion for language as a problem of choosing the right internal representation space for denoising. Our contributions are threefold. • We formulate diffusion insertion in pretrained transformer language models as a geometry-guided interface-selection problem and propose practical layer-wise proxies—local compactness, global stiffness, and effective rank—for identifying diffusion-friendly hidden spaces. • We introduce a fixed geometry score that narrows the search for effective insertion layers without exhaustive layer-wise bridge training and correlates strongly with hidden-state reconstruction quality under a one-epoch bridge-training protocol across 8B-scale backbones. • We introduce DiHAL , a Locate-and-Replace hybrid that replaces lower transformer layers with a conditional diffusion bridge and reuses the upper layers and LM head. Under a diagnostic diffusion/recovery budget, DiHAL shows that hidden-state recovery can improve generative perplexity and diversity over embedding-, latent-, and continuous-to-discrete interfaces. Figure 1: Locate-and-Replace framework. Layer-wise geometric proxies score transformer layers, select an insertion point, and guide replacement with a diffusion block. 2 Background Transformer language models take discrete tokens as input, but most computation occurs in continuous hidden spaces. Given x_{1:T} , an autoregressive model factorizes p(x_{1:T})=\prod_{t=1}^{T}p(x_{t}\mid x_{<t}) . Each token is mapped to an embedding h_{t}^{(0)}=E(x_{t})+p_{t} , hidden states are updated as H^{(\ell)}=F_{\ell}(H^{(\ell-1)}) , and the final state is projected to vocabulary logits z_{t}=W_{\mathrm{LM}}h_{t}^{(L)}+b . Thus, discreteness appears at the input and output interfaces, while intermediate computation is continuous ( 25 ) . Diffusion models generate samples by gradually adding noise to data and then learning to reverse this noising process ( 23 ) . In continuous space, this process can be written as a stochastic differential equation dX_{t}=f(X_{t},t)\,dt+g(t)\,dW_{t} , where X_{t} is the noisy representation at time t and W_{t} is Brownian motion. The reverse process depends on the score \nabla_{x}\log p_{t}(x) , which is approximated by a neural network. Applying this idea to language requires choosing what representation the diffusion model should denoise. Discrete diffusion models corrupt tokens directly ( 1 ; 9 ) , while continuous diffusion language models denoise token embeddings or learned latent vectors ( 13 ; 24 ; 14 ) . Continuous token-level diffusion suffers from recovery errors: small denoising deviations can flip the recovered token ( 30 ; 21 ) . Learned latent diffusion reduces this issue but still requires an interface for converting latents to text. We instead target internal transformer hidden states, where recovery becomes hidden-state reconstruction rather than direct token decoding. 3 Method DiHAL is a diffusion–transformer hybrid architecture that replaces part of a pretrained transformer, rather than a standalone diffusion language model. Figure 1 illustrates our Locate-and-Replace procedure. This section develops DiHAL in three steps: we first motivate diffusion-friendly representations using geometric principles from Langevin dynamics and concentration theory, then instantiate them as layer-wise proxies for locating a suitable hidden-state interface, and finally replace the lower transformer layers with a conditional diffusion bridge while retaining the upper layers and original LM head. Rather than modeling token probabilities or recovering discrete tokens directly, the bridge reconstructs an internal boundary representation that the retained upper layers can already process. 3.1 Geometric Principles for Diffusion-Friendly Layer Selection We now make the notion of a diffusion-friendly representation space more concrete. Intuitively, a good diffusion space should satisfy three properties: denoising should contract quickly toward the target representation distribution, remain stable under score-estimation