Compressed and Smooth Latent Space for Text Diffusion Modeling

Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov · 2025

Read Compressed and Smooth Latent Space for Text Diffusion Modeling by Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21170.txt.

Introduction

Viacheslav Meshchaninov and co-authors introduce Cosmos, a text generation method that diffuses in a compressed latent space, in a 2025 paper. Standard language models write one token at a time, which is slow and can lose the global thread. Diffusion models generate in parallel but struggle with the size of token-level data.

Cosmos first learns an autoencoder that squeezes text into a smooth, compact code, guided by a pretrained language encoder. Diffusion then works in that small space. Text compressed eightfold kept generation quality on par with token-level diffusion. Longer latent sequences pushed quality higher still. The result points to faster, more controllable text generation.

Catalog record

CREATORViacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-21170

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compressed and Smooth Latent Space for Text Diffusion Modeling Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov Abstract Autoregressive language models dominate modern text generation, yet their sequential nature introduces fundamental limitations: decoding is slow, and maintaining global coherence remains challenging. Diffusion models offer a promising alternative by enabling parallel generation and flexible control; however, their application to text generation is hindered by the high dimensionality of token-level representations. We introduce Cosmos , a novel approach to text generation that operates entirely in a compressed, smooth latent space tailored specifically for diffusion. This space is learned using an autoencoder trained simultaneously for token-level reconstruction and alignment with frozen activations from a pretrained language encoder, providing robust semantic grounding and enabling effective perturbation - based augmentations. Empirically, we demonstrate that text representations can be compressed by 8 × 8\times while maintaining generation quality comparable to token - level diffusion models. Furthermore, increasing the latent sequence length allows Cosmos to surpass both diffusion - based and autoregressive baselines. We evaluate Cosmos on four diverse generative tasks including story generation, question generation, summarization, and detoxification and compare it with various generative paradigms. Cosmos achieves comparable or superior generation quality while offering more than 2 × 2\times faster inference. 1 Introduction Autoregressive (AR) language models are a de facto gold standard for text generation ( radford2018improving ; radford2019language ; brown2020gpt3 ) . By factorizing the probability of a sequence into a product of conditional token probabilities, they transform the global generation task into a series of local next-token prediction tasks that can be optimized efficiently via teacher forcing ( williams1989learning ) . The same sequential factorization, however, creates several structural bottlenecks. First, decoding is inherently sequential: the time to produce a sequence grows linearly with its length because each token must await the completion of its predecessor. Second, the process suffers from exposure bias: a single early mistake contaminates the context for all subsequent predictions and cannot be corrected without restarting generation ( ranzato2015sequence ; bengio2015scheduled ; he2019exposure ) . Third, maximizing local log-likelihood encourages the model to privilege fluency over factual accuracy, resulting in the well-documented phenomenon of hallucination ( maynez2020faithfulness ; ji2023survey ) . Finally, because decisions are made token-by-token, the model lacks an explicit global plan and therefore struggles to maintain long-range logical or narrative coherence ( ye2024beyond ) . In stark contrast, computer vision has been reshaped by diffusion models ( ho2020ddpm ; nichol2021improved ) , particularly by latent diffusion ( rombach2022ldm ; blattmann2023stable ; andreas2023 ) , which first compresses a high-resolution image or video into a compact latent representation and then performs the diffusion process in that space. Operating in a low-dimensional latent manifold reduces computational cost by orders of magnitude and enables breathtaking advances in image and video synthesis. Yet recent works ( kouzelis2025eq ; skorokhodov2025improving ) also show that diffusion is highly sensitive to the geometry of the latent space: poorly designed latent representations can destabilize training and degrade sample quality, underscoring the need for principled mechanisms to construct diffusable 1 1 1 Diffusable representations refer to latent spaces that permit effective modeling by a diffusion process. representations. Motivated by these insights, we revisit the foundations of text representation. We hypothesize that the token-level encoding used in contemporary language models is heavily overparameterized for the purpose of sequence generation. Building on the successes of latent diffusion in vision, we pose the following question: How far can we compress textual information into a compact latent space while still matching or even exceeding the generative fidelity of conventional token - level representations? To answer this question, we develop an autoencoder that maps text into a lower-dimensional space and then train a diffusion model directly in that space. We show empirically that naively minimizing token-reconstruction loss yields a brittle latent geometry that hampers diffusion, whereas introducing robustness- and smoothness-oriented objectives produces a well-behaved manifold conducive to high-quality diffusion synthesis. Our experiments demonstrate that with the right training regime, latent diffusion not only rivals but in several settings surpasses traditional token-level baselines. These findings challenge the prevailing assumption that token-level autoregression is indispensable for language generation and position latent-space diffusion as a powerful alternative paradigm for future large-scale language models. Our key contributions are as follows: • We propose Cosmos — a training recipe for a CO mpressed and SMO oth latent S pace, that allows to train a diffusion model in more compact latent space while matching the quality of token - level diffusion baselines. • We demonstrate that modestly scaling the number of latent vectors enables latent - space diffusion to surpass both token - level diffusion and autoregressive models on an unconditional text generation task. • On standard text - generation benchmarks, the proposed latent diffusion achieves up to \mathbf{2\times} faster sampling than conventional token - level diffusion models, while matching or slightly exceeding them in quality and diversity. 2 Related work Early efforts to bring latent diffusion models into natural language generation treat the diffusion module mainly as a pre-processor that supplies conditioning vectors for an autoregressive decoder. LD4LG ( lovelace2023latent ) trains a diffusion model on compressed hidden states from BART, then feeds the generated latent vectors into the BART decoder to produce text. PLANNER ( zhang2023planner ) adopts a similar two-stage recipe: a fine-tuned BERT encoder produces a 16-token variational latent code, a diffusion model refines that code, and a GPT-2 decoder generates text from this code. While both systems improve controllability, their heavy reliance on powerful autoregressive decoders makes it difficult to isolate and evaluate the intrinsic generative capacity of the latent-space diffusion itself. A complementary line of work applies diffusion directly to the continuous embeddings of pretrained encoders. TEncDM ( shabalin2025tencdm ) demonstrates that full-length BERT representations can be modeled with Gaussian diffusion and decoded into coherent text without an intermediate autoregressive step. Crucially, however, TEncDM retains the original sequence dimensionality, leaving open a question: How aggressively can such representations be compressed before generation quality collapses? Our study closes this gap. We devise a training procedure that shrinks BERT-level representations by up to \bf{8\times} while preserving, and in some cases enhancing, their suitability for latent-space diffusion. 3 Preliminary Diffusion models ( ho2020ddpm ; nichol2021improved ) learn a data distribution by reversing a progressive noising process. Given a clean sample z_{0}\sim p_{\text{data}} , the forward dynamic corrupts the input with Gaussian noise whose magnitude is controlled by a continuous time index t\in[0,1] : z_{t}\;=\;\sqrt{\alpha_{t}}\,z_{0}\;+\;\sqrt{1-\alpha_{t}}\,\varepsilon,\qquad\varepsilon\sim\mathcal{N}(0,I), (1) where the noise schedule \alpha_{t} is monotonically decreasing with \alpha_{0}=1 and \alpha_{1}\approx 0 . A neural denoiser z_{\theta}(z_{t},t) with learnable parameters \theta is trained to invert this corruption by minimising the denoising–score–matching objective \mathcal{L}_{\text{DM}}\;=\;\mathbb{E}_{z_{0}\sim p_{\text{data}},\,t\sim\mathcal{U}[0,1],\,\varepsilon\sim\mathcal{N}(0,I)}\Bigl[\bigl\lVert z_{0}-z_{\theta}(z_{t},t)\bigr\rVert_{2}^{2}\Bigr]. (2) At inference time the model is applied iteratively from t=1 to t=0 , gradually transforming pure noise into a realistic sample. 3.1 Latent diffusion Latent Diffusion Models (LDMs) ( rombach2022ldm ; andreas2023 ) improve both compute efficiency and quality by learning the diffusion process in a compact latent space rather than in pixel or token space. An autoencoder with encoder E and decoder D is first trained to achieve high-fidelity reconstruction, \mathbf{\hat{w}}~=~D\!\bigl(E(\mathbf{w})\bigr)~\approx~\mathbf{w}, where \mathbf{z}=E(\mathbf{w}) denotes the low-dimensional latent. Eqs. ( 1 )–( 2 ) yield a diffusion model trained

Read the complete text