Generalized Interpolating Discrete Diffusion

Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, Thomas Hofmann · 2025

Read Generalized Interpolating Discrete Diffusion by Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, Thomas Hofmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04482.txt.

Introduction

Dimitri von Rütte and five colleagues present a 2025 machine learning paper on a new way to train language models. Most models write text one token at a time and can never revise what they produced. Diffusion-style models instead start from noise and clean it up, but the popular masked version still cannot fix a revealed word.

The team generalizes masked diffusion into a broader family that mixes masking with random replacement noise. A fresh training formula lets these models match leading results at equal computing cost. The mixed-noise version also produced better samples and could correct its own mistakes mid-generation, a known weakness of standard models. The work widens the design space for text generators beyond left-to-right writing.

Catalog record

CREATORDimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, Thomas Hofmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE123 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04482

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generalized Interpolating Discrete Diffusion Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, Thomas Hofmann Abstract While state-of-the-art language models achieve impressive results through next-token prediction, they have inherent limitations such as the inability to revise already generated tokens. This has prompted exploration of alternative approaches such as discrete diffusion. However, masked diffusion, which has emerged as a popular choice due to its simplicity and effectiveness, reintroduces this inability to revise words. To overcome this, we generalize masked diffusion, deriving a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes. Leveraging a novel diffusion ELBO, we achieve compute-matched state-of-the-art performance in diffusion language modeling. Exploiting GIDD’s flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality and unlocking the ability for the model to correct its own mistakes, an area where autoregressive models notoriously have struggled. Code: https://github.com/dvruette/gidd/ 1 Introduction Figure 1 : Training a diffusion model using GIDD on a combination of masking and uniform noise teaches it to identify and correct its own mistakes. By iteratively replacing bad tokens with better ones (as determined by the model), sample quality (as per generative PPL via Gemma 2 9B) improves by up to 55%. Machine learning are is a field to study of research in artificial intelligence during which and the development […] Republic of Delta World of Warcraft has made some significant improvements to game it in their the most recent improvement update , the “Death in the Vengeance” change “End of the World” update . Mexico City is the largest city in France Mexico . With an estimated population of 22,752,000 22,000,000 […] Suppose Alice has 5 apples. If Alice gives 2 all of her the apples to Bob, she is left with zero apples. Table 1 : Examples of self-correction (green replaces red) by our GIDD+ base model trained with 20% uniform noise. The model is able to correct grammatical mistakes, improve word choices, and even improve factuality without being explicitly trained to do so. For certain data distributions such as natural images or natural language, the information content of any given sample can be overwhelming, making the task of generating realistic samples through generative modeling difficult. A common strategy to ease the burden on the generative model is to break up the task of generating an entire sample into multiple inference steps, each being simpler in isolation, but recovering the full distribution when recombined. The most prevalent example of this, especially for natural language, is autoregressive modeling ( Bengio et al., 2000 ) , where the task of generating a sentence (or sequence) is decomposed into generating one word (or token) at a time, with each new word serving as additional context for the next word. While extraordinarily successful on a wide range of data modalities ( van den Oord et al., 2016a ; van den Oord et al., 2016b ; Radford et al., 2018 ) , there are some inherent challenges to this approach. First and most obviously, generating a sequence of length N necessarily requires N invocations of the model. This is not a problem if N is small but can become expensive as N grows to large numbers. Secondly, long-term dependencies and coherence can pose a challenge, for example if each step has a non-negligible error rate: If a wrong token is sampled or a previous token becomes incompatible with newly sampled tokens, there is no way to correct it. Considerable effort has gone into solving this limitation, most recently by post-training with reinforcement learning (RL) to teach sequential reasoning over multiple autoregressive steps ( Bengio et al., 2015 ; Ranzato et al., 2015 ; Bahdanau et al., 2016 ; OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ) . Denoising diffusion models ( Sohl-Dickstein et al., 2015 ) propose a different way of decomposing the generative task which can address both limitations. Instead of splitting the sample into elements of a sequence, we gradually decrease the information content of the entire sample by degrading it through the addition of some form of noise until, eventually, the information content reaches zero. The generative task then consists of reversing this degradation process, gradually adding information back in until the full sample is restored. This decouples the number of model invocations from the size of the sample since the number of steps we take to fill in the missing information can be chosen freely. For natural images, an obvious and suitable degradation is the progressive addition of per-pixel Gaussian noise. This choice yields a simple training objective that works well in practice and forms the basis of state-of-the-art image generation models ( Ho et al., 2020 ; Kingma et al., 2023 ) . The success of image diffusion models has spurred interest in applications to other domains and modalities, including discrete data like text ( Austin et al., 2023 ) . Unfortunately, Gaussian diffusion cannot be naively applied to discrete data as there is not necessarily a notion of distance or similarity, at least not one that is straightforward to measure. Instead, discrete diffusion models have converged on a degradation process consisting of gradually removing (“masking”) tokens until none are left ( Austin et al., 2023 ; Shi et al., 2024 ; Sahoo et al., 2024 ) . The task of the model then becomes to reconstruct the original sequence by “filling in the blanks” until all masked tokens have been filled in. This can also be thought of as autoregressive generation in a randomized order ( Welleck et al., 2019 ) , and indeed reintroduces one of its inherent limitations: Once a token is filled in, it can no longer be changed, and any intermediate errors will necessarily propagate to the final sample. However, by injecting a small amount of uniform token noise into the diffusion process, we can allow for any token to transition to any other token, therefore resolving this issue. This elicits the realization that the type of noise is an integral part of a diffusion model with potentially fundamental implications on its strengths and limitations. Drawing motivation from this, our work aims to illuminate the design space of discrete diffusion models by opening up the design space and exploring an alternative diffusion process that combines masking and uniform noise. Our contributions are two-fold: On the theoretical side, in Section 3 we extend the framework of masked diffusion to general interpolating discrete diffusion (GIDD) processes. GIDD offers great flexibility in the choice of noising process, encompassing any diffusion process that can be written as a linear combination between the data and some (time-varying) mixing distribution. We derive closed-form solutions for the cumulative state transitions and the diffusion Evidence Lower Bound (ELBO) for this general family, which are needed for sampling and likelihood training respectively. We also show that the derived ELBO has a global minimum that is reached when the model matches the true distribution. On the practical side, in Sections 4 and 5 we apply our theory to the special case of masking noise in combination with varying levels of uniform noise. We conduct an ablation study, showing that our mask-only

Read the complete text