When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models
Danny Wang, Ruihong Qiu, Zi Huang, 1 {}^{\text{1}} The University of Queensland · 2026
Read When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models by Danny Wang, Ruihong Qiu, Zi Huang, 1 {}^{\text{1}} The University of Queensland (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23994.txt.
Introduction
Wang, Qiu, and Huang argue that discrete diffusion language models commit words too early when decoding in fixed blocks without later context. They define a block as self-contained when its predictions stay the same with or without future context. Variable-size blocks are then chosen by how much predictions would shift if the future were revealed. Theory links this test to predictive consistency, and experiments beat fixed and heuristic block decoding.
Catalog record
| CREATOR | Danny Wang, Ruihong Qiu, Zi Huang, 1 {}^{\text{1}} The University of Queensland |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66124 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When to Commit? Towards Variable-Size Self-Contained Blocks for Discrete Diffusion Language Models Danny Wang, Ruihong Qiu, Zi Huang, 1 {}^{\text{1}} The University of Queensland Abstract Discrete diffusion language models (dLLMs) enable parallel token updates with bidirectional attention, yet practical generation typically adopts blockwise semi-autoregressive decoding. This switch creates a training-inference mismatch: training denoises with full-sequence context, while inference commits tokens within a bounded block without future context . Therefore, decoding with fixed-size or heuristic-based blocks can lead to premature token commitments, as decisions are made without full access to future context that could alter those choices. Motivated by this, we propose self-containedness as a principled criterion for block commitment. A block is self-contained if its predictions remain consistent with Future-Aware (FA) or without No-Future (NF) access to future context, reframing block boundary selection as a test of self-containedness rather than a heuristic choice. Based on this principle, we introduce Variable-size Self-contained Blocks (VSB) for dLLMs. VSB scores and selects block boundaries using the divergence between token-level predictive distributions under NF and FA conditioning, which quantifies how predictions would change if future context were revealed. We provide theoretical justification linking self-containedness to predictive consistency, and extensive experiments validate VSB’s efficacy over fixed-size and heuristic blockwise decoding. 1 Introduction Recent progress in discrete diffusion language models (dLLM) enable parallel token generation with global bidirectional attention, breaking the serial dependency of autoregression ( 27 ; 40 ; 19 ; 22 ; 26 ) . In practice, many dLLMs are decoded blockwise in a semi-autoregressive manner: the model denoises a block of B tokens in parallel and then often irreversibly commits the block to the prefix, meaning the committed tokens cannot be changed later when future context becomes available, before moving on to generate the next block. ( 1 ; 7 ; 37 ) . This strategy balances parallel efficiency with autoregressive style coherence, however, it raises a central but under-explored question: When can a block be safely committed such that its tokens remain consistent when future context is revealed? Existing blockwise decoding typically decides when to commit simply by choosing block boundary with either fixed block size or surface heuristics (e.g., punctuation). These rules can commit tokens whose predictions would change once later tokens are generated, yielding future-dependent commitments and unstable conditioning for subsequent blocks. ( 25 ; 18 ; 39 ) . Figure 1 : Future-dependent vs. self-contained commitments in dLLMs. Fixed or heuristic boundaries can commit tokens whose meaning rely on future context, while self-contained commitment preserves semantic coherence and reduces future dependence. This typical blockwise schema exposes two coupled issues: (1) Fundamental training-inference mismatch. Standard dLLMs are typically trained to denoise under full-sequence ( x_{1:\text{full}} ) bidirectional context: when predicting a token, the model can condition on both its left prefix and right suffix ( 15 ; 27 ; 47 ; 29 ) . This training setup is natural for full-length bidirectional decoding, as it encourages the model to use information from the entire sequence for making predictions. During typical blockwise semi-AR decoding, by contrast, committing a block is often irreversible : once tokens are committed, they cannot be revised using later context ( 37 ; 1 ; 25 ) . If the model would substantially change its predictions for tokens in the current block after seeing future tokens, the block is not self-contained and should not be committed. (2) Heuristic block boundaries. Most methods either fix the block size B (e.g., 32 tokens) or align boundaries with surface cues such as punctuation ( 25 ; 38 ; 1 ) . Neither choice ensures that the committed block is semantically complete or independent of future context. As illustrated in Figure 1 , a fixed-size boundary can cut through an important semantic unit, and punctuation cue (e.g., a full stop) does not imply that the preceding phrase is resolved without the following clause. Both cases lead to future-dependent commitments that can propagate errors to subsequent blocks. Recent work has started to reduce this mismatch with blockwise-aligned training objectives, but boundary selection typically remains heuristic, without an explicit principled notion of when a block is ready to be committed ( 32 ; 7 ; 36 ) . In light of these challenges, we propose self-containedness as the criterion for placing block boundaries in semi-AR decoding. A candidate block is self-contained if the model’s predictive distributions for tokens inside the block remain consistent when future context is revealed. This directly matches the irreversibility of commitment: we should commit tokens only when their predictions no longer rely on future information beyond the boundary. However, self-containedness is inherently counterfactual because future tokens are unavailable at inference time. We resolve this by comparing two predictive distributions under the same prefix within a fixed block budget of length W : (i) Future-Aware (FA) predictions, where the model conditions on the entire budgeted block, and (ii) No-Future (NF) predictions, where the model conditions only up to candidate boundary b . If the predictive distributions for the tokens to be committed are consistent between FA and NF, then committing up to b is self-contained ; if the distributions diverge, the boundary is premature. This motivates quantifying self-containedness by measuring the divergence between conditioning with and without future information (NF-FA), enabling self-contained variable-size blocks without fixed lengths or punctuation heuristics. We introduce Variable-Size Self-Contained Blocks (VSB) to realise this principle. VSB (i) aligns training with blockwise inference by training under the same No-Future truncation used at commitment time, and (ii) dynamically selects self-contained block boundaries by scoring candidate cutoffs via NF-FA divergence. Our contributions are: • Principle: We formalise Self-Containedness as a criterion for determining block boundaries in blockwise semi-AR decoding. A block is committed only when revealing future tokens does not largely alter its predictive distributions. • Method: We propose VSB , a self-containedness-governed training and decoding framework that aligns training with inference via No-Future truncation and selects variable-size blocks using Future-Aware & No-Future divergence. • Theoretical: We relate NF-FA divergence to predictive consistency within committed blocks, providing intuition for why enforcing self-containedness mitigates errors from premature commitment. 2 Preliminaries We first define terminology used throughout the paper. Let x_{1:L} denote a length- L token sequence. 1. Prefix and commitment. During block decoding, the model maintains a committed prefix x_{1:p} , ending at position p . Committing means appending the newly generated block of tokens to this prefix. Once committed, these tokens are treated as fixed context and non-revisable. 2. Block budget and candidate boundary. We restrict the size of a block to a fixed block budget of length W (e.g., 64 tokens). We refer to (p:q] as the budgeted block where q=\min(p+W,L) . b\in(p:q] is a candidate block boundary within the budgeted block. 2.1 Discrete Masked Diffusion Discrete masked diffusion training on the full token sequence for text. Let \mathcal{V} be the vocabulary. Given an input token sequence x_{1:L}\in\mathcal{V}^{L} of length L and diffusion step t\in\{1,\dots,T\} , a masked corruption process produces a noisy sequence as: x^{t}\sim q_{t}(\cdot\mid x) , where some positions are replaced by a special [MASK] token \mathbf{M} based on a masking schedule ( 27 ; 29 ) . A discrete dLLM parameterised by \theta outputs per-position logits \ell_{\theta,i}(x^{t},t)\in\mathbb{R}^{|\mathcal{V}|} and categorical distributions: \vskip-8.5359ptp_{\theta}(x_{i}\mid x^{t},t)=\mathrm{softmax}\big(\ell_{\theta,i}(x^{t},t)\big)[x_{i}].\vskip 0.0pt