When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation
De Shuai Zhang · 2026
Read When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation by De Shuai Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15557.txt.
Introduction
De Shuai Zhang studies non-autoregressive text generation from continuous latent states, where good geometry in latent space does not guarantee readable output. His draft-conditioned refinement model starts from a frozen encoder and a parallel decoder, with a denoising prior, a local flow, and a learned metric. On a story dataset, full-size latents recover tokens far better than compressed ones, and refinement gives only modest gains. The diagnostic conclusion is that latent geometry alone is not enough: decoder recoverability must be judged directly.
Catalog record
| CREATOR | De Shuai Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39698 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation De Shuai Zhang Abstract Continuous diffusion and flow models are attractive for non-autoregressive text generation because they can update all positions in parallel. A major difficulty is the interface between continuous latent states and discrete tokens. This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, a denoising DraftPrior, a local FlowNet, and a learned diagonal MetricNet. Early Gaussian-start experiments showed that good latent-space metrics, such as scale matching or cosine similarity, do not guarantee good decoding. Generated latents can be close to real encoder latents but still produce high-entropy, biased, or repetitive token distributions. We therefore frame the task as controlled local refinement rather than full generation from noise. On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents. With 768-dimensional latents, DraftPrior target-token probability is 0.938 for clean drafts, 0.613 for 3% token dropout, 0.483 for 5% dropout, and 0.272 for 10% dropout. Local flow refinement and fused decoder-aware readout give modest additional gains, while metric learning and OT-style alignment improve geometry but do not close the decoder gap. The main result is a diagnostic one: latent geometry alone is not enough. Continuous latent text generation should be evaluated by decoder recoverability, the quality of the start distribution, and whether refinement preserves decoder-readable structure. Keywords: flow matching, Riemannian flow, non-autoregressive text generation, latent text generation, draft refinement, BERT autoencoder, decoder recoverability 1 Introduction Autoregressive language models based on Transformer architectures [ 1 ] , including GPT-style systems [ 3 ] , generate one token at a time. This factorization is powerful, but it makes generation sequential. Non-autoregressive generation is attractive because it could produce an entire continuation in parallel and permit global sequence-level refinement. The difficulty is that language is discrete, while diffusion and flow-based methods operate naturally in continuous spaces. A common approach, related to denoising autoencoding models such as BART [ 4 ] , is to encode text into a continuous latent sequence, generate or refine in that latent space, and decode the resulting latents back into tokens. This resembles latent diffusion in vision, but the text setting is less tolerant. In images, a slightly imperfect latent may still decode into a plausible image. In text, a small latent error can change the token identity completely or leave the decoder uncertain across many vocabulary items. This project began as a conditional latent flow-matching system. A frozen BERT encoder [ 2 ] maps an input token sequence x=(x_{1},\ldots,x_{n}) of length n into continuous latent representations through a projection layer P : z=P(E_{\text{BERT}}(x)),\quad z\in\mathbb{R}^{n\times d} (1) where d is the latent dimension ( d=768 in our main experiments, d=256 for the compressed baseline). A parallel decoder p_{\phi} predicts all token distributions simultaneously, forming a stage-1 latent autoencoder trained with: \mathcal{L}_{\text{AE}}=-\sum_{i=1}^{n}\log p_{\phi}(x_{i}\mid z) (2) For conditional generation, the latent sequence is split into prompt and suffix parts at position m : z=(z^{p},z^{s}),\quad z^{p}=z_{1:m},\quad z^{s}=z_{m+1:n} (3) where z^{p}\in\mathbb{R}^{m\times d} is the known prompt latent and z^{s}\in\mathbb{R}^{(n-m)\times d} is the suffix latent to be generated. In our ROCStories experiments, m=32 prompt token slots and n-m=32 suffix token slots. Diagnostics showed that the original Gaussian-start formulation was too optimistic. A flow path interpolates between noise and the real suffix latent: z_{0}\sim\mathcal{N}(0,I),\quad z_{t}=(1-t)z_{0}+tz^{s},\quad t\sim\mathcal{U}(0,1) (4) where I is the identity matrix and t is the interpolation timestep. A FlowNet v_{\theta} is trained to predict the target velocity u_{t}=z^{s}-z_{0} : \mathcal{L}_{\text{FM}}=\mathbb{E}\left[\|v_{\theta}(z_{t},t,z^{p})-(z^{s}-z_{0})\|^{2}\right] (5) The flow could improve latent-space metrics while the decoder still produced high-frequency-token attractors, repetition, and poor target-token probability. Specifically, generated latents \hat{z}^{s} could achieve high cosine similarity with real suffix latents while still decoding poorly: \cos(z^{s},\hat{z}^{s})\approx 1\not\Rightarrow P_{\text{target}}(\hat{z}^{s})\text{ is high} (6) where P_{\text{target}} is the average target-token probability under the frozen decoder, defined formally in Eq. 40 . The current version reframes the system. Instead of asking a flow to create a valid text continuation from pure noise, we use a rough draft as an instance-level guide. The draft is encoded into the same latent space, a DraftPrior maps it toward the decoder-readable basin, and the metric-aware flow applies only a small local correction. This report studies that controlled refinement setting and the failure modes that remain. 1.1 Contributions This report makes four contributions: • We identify a decoder recoverability failure mode: latent similarity does not guarantee token recovery. • We introduce a draft-conditioned refinement setup for ROCStories, using the first two sentences as prompt and the last three as target. • We compare 256-dimensional compressed latents with full 768-dimensional BERT latents and show that stronger compression loses token-level information. • We ablate DraftPrior, local FlowNet refinement, MetricNet, OT-style alignment, fused readout, and bounded residual refinement. The start distribution gives most of the recoverability; geometry-based refinement gives smaller gains. 2 Related Work 2.1 Continuous diffusion and flow models for text Diffusion-LM [ 19 ] , building on diffusion models such as DDPM [ 13 ] and score-based generative modeling [ 14 ] — which learns the score function s_{\theta}(x_{t},t)\approx\nabla_{x_{t}}\log p(x_{t}) and generates samples via Langevin dynamics — demonstrated that continuous diffusion can be adapted to text by mapping words into continuous embeddings and then rounding back to tokens. Its treatment of embedding choice and rounding is directly relevant here: the continuous-to-discrete interface is not an implementation detail, but a central modeling problem. In this report, the rounding operation is replaced by a parallel decoder, but the same challenge remains: generated continuous states must land in regions that the decoder can reliably map to tokens. Diffusion-LM addresses the continuous-to-discrete gap with a clamping trick: during sampling, the predicted clean vector f_{\theta}(\mathbf{x}_{t},t) is mapped to its nearest word-embedding sequence before forming the next denoising state, \mathbf{x}_{t-1}=\sqrt{\bar{\alpha}_{t}}\,\mathrm{Clamp}(f_{\theta}(\mathbf{x}_{t},t))+\sqrt{1-\bar{\alpha}_{t}}\,\epsilon. (7) Our system faces the same interface problem, but uses a different constraint. Rather than snapping intermediate states to discrete word embeddings, we start from a decoder-readable draft latent and restrict the refinement magnitude: \mathbf{z}_{\mathrm{res}}=\mathbf{z}_{\mathrm{ode}}+\lambda\tanh R_{\omega}(\mathbf{z}_{\mathrm{ode}},\mathbf{z}^{p}) (8) where \lambda>0 is a scalar bound controlling the maximum refinement magnitude, ensuring \|\mathbf{z}_{\mathrm{res}}-\mathbf{z}_{\mathrm{ode}}\|_{\infty}\leq\lambda . Thus, Diffusion-LM enforces discreteness by repeated projection to the embedding vocabulary, while our bounded residual preserves local decoder-readability by preventing the refinement trajectory from moving too far from the draft-conditioned latent basin. These are analogous design responses to the same continuous-to-discrete problem, operating at different stages of generation. Latent Diffusion for Language Generation [ 20 ] is especially close to our setting because it trains continuous generative models in the latent space of a language autoencoder and decodes sampled latent states back into natural language. Our system shares this latent-space generative view but differs in three important ways. First, we use a flow-based model rather than diffusion, updating the full suffix latent sequence jointly in parallel. Second, we replace the pretrained autoregressive decoder with a parallel decoder trained from scratch, removing left-to-right token conditioning during decoding. Third, this setting reveals a sharp decoder-readability failure mode: generated latents can be geometrically close to real suffix latents while still decoding poorly under a frozen parallel decoder. In our experiments, direct Gaussian-start latent generation was therefore insufficient, leading us to adopt draft-conditioned local