BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
Shaobin Zhuang 1 , Yuang Ai 2,3 , Jiaming Han 2 , Xiaohui Li 1 , Huaibo Huang 3 Xiangyu Yue 2 , Xuefeng Hu, Kun Xu, Yali Wang 4 ✉, Hao Chen✉ 1 Shanghai Jiao Tong University 2 MMLab, The Chinese University of Hong Kong 3 Institute of Automation, Chinese Academy of Sciences 4 SIAT, Chinese Academy of Sciences · 2026
Read BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion by Shaobin Zhuang 1 , Yuang Ai 2,3 , Jiaming Han 2 , Xiaohui Li 1 , Huaibo Huang 3 Xiangyu Yue 2 , Xuefeng Hu, Kun Xu, Yali Wang 4 ✉, Hao Chen✉ 1 Shanghai Jiao Tong University 2 MMLab, The Chinese University of Hong Kong 3 Institute of Automation, Chinese Academy of Sciences 4 SIAT, Chinese Academy of Sciences (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11577.txt.
Introduction
Zhuang and nine colleagues present BitLM, a language model generating several tokens at once through bitwise continuous diffusion. Autoregressive models emit one token at a time, yet language meaning often sits in multi-token phrases and collocations. BitLM codes each token as a fixed-length binary string and denoises several tokens in parallel within each block. It keeps causal attention running left to right across blocks while deciding jointly within blocks. Swapping the large-vocabulary softmax for bitwise denoising yields faster inference and stronger pre-training without breaking causal structure.
Catalog record
| CREATOR | Shaobin Zhuang 1 , Yuang Ai 2,3 , Jiaming Han 2 , Xiaohui Li 1 , Huaibo Huang 3 Xiangyu Yue 2 , Xuefeng Hu, Kun Xu, Yali Wang 4 ✉, Hao Chen✉ 1 Shanghai Jiao Tong University 2 MMLab, The Chinese University of Hong Kong 3 Institute of Automation, Chinese Academy of Sciences 4 SIAT, Chinese Academy of Sciences |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31516 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11577
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion Shaobin Zhuang 1 , Yuang Ai 2,3 , Jiaming Han 2 , Xiaohui Li 1 , Huaibo Huang 3 Xiangyu Yue 2 , Xuefeng Hu, Kun Xu, Yali Wang 4 ✉, Hao Chen✉ 1 Shanghai Jiao Tong University 2 MMLab, The Chinese University of Hong Kong 3 Institute of Automation, Chinese Academy of Sciences 4 SIAT, Chinese Academy of Sciences Abstract Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units—phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM , a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference—without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures. 1 Introduction Modern large language models are usually described as models of language, but operationally they are models of transitions between vocabulary IDs. Given a prefix, a transformer computes a contextual hidden state and converts it into a normalized distribution over a vocabulary; one token is then sampled, appended to the prefix, and the process repeats. This next-token paradigm has underwritten the success of contemporary LLMs across scales and domains ( Brown et al. 2020 ; Chowdhery et al. 2023 ; Touvron et al. 2023 ; Grattafiori et al. 2024 ) . At the same time, it imposes a very specific interface between hidden-state computation and symbolic output: language must be emitted as a sequence of atomic categorical decisions. That interface has long been recognized as restrictive in its own right ( Yang et al. 2017 ) , and it also makes generation intrinsically sequential at inference time. A large body of recent work has tried to reduce this sequential bottleneck without changing the underlying output interface. Blockwise decoding, speculative decoding, and multi-token prediction accelerate generation by proposing or verifying several future tokens at once ( Stern et al. 2018 ; Leviathan et al. 2023 ; Cai et al. 2024 ; Gloeckle et al. 2024 ) . Non-autoregressive and semi-autoregressive methods go further by relaxing strict left-to-right factorization ( Gu et al. 2017 ; Wang et al. 2018 ; Ghazvininejad et al. 2019 ) . In parallel, diffusion and iterative refinement have emerged as compelling alternatives to one-step categorical prediction for discrete generation ( Austin et al. 2021 ; Li et al. 2022 ; Gong et al. 2022 ; Lou et al. 2024 ) . These directions have produced important insights, but they often leave one assumption untouched: the model still ultimately talks to text through vocabulary-level categorical outputs, whether one token at a time or many. Figure 1: Conceptual comparison between standard AR LLMs and BitLM . By replacing the softmax head with a diffusion head, BitLM reformulates token generation as iterative denoising in a compact binary space, enabling multi-token joint realization. This paper starts from a different premise, as conceptually illustrated in Figure 1 . Instead of asking how to decode the vocabulary softmax more efficiently, we ask whether the vocabulary softmax should be the primary interface at all. A token identity is usually treated as an atomic class, but it can equally be represented as a short binary code. On the surface, this may look like a simple reparameterization. Our claim is that it changes the geometry of generation. A vocabulary softmax places lexical alternatives on a simplex and encourages per-position categorical decisions; a binary representation turns token prediction into denoising over a small set of coordinated binary variables. In that view, generation is no longer only the problem of selecting the next token ID. It becomes the problem of realizing a continuation as a structured discrete object that can be refined jointly across multiple positions. This perspective resonates with several recent developments. Byte-level and token-free models have shown that the basic interface between raw text and sequence modeling is more flexible than wordpiece tokenization might suggest ( Xue et al. 2022 ; Yu et al. 2023 ) . Diffusion-based models have shown that iterative refinement can be a powerful generative principle for discrete data ( Austin et al. 2021 ; Li et al. 2022 ; Gong et al. 2022 ; Lou et al. 2024 ) . In vision and other discrete domains, binary formulations such as Analog Bits demonstrate that diffusion in bit space can provide a simple and effective bridge between discrete symbols and continuous denoising dynamics ( Chen et al. 2022 ) . Yet this perspective remains underexplored in language modeling, where the output layer is still overwhelmingly organized around a large categorical vocabulary distribution. We argue that language is precisely the setting where this question matters most, because the output interface is not only a modeling choice but also a decoding bottleneck. Motivated by this observation, we propose BitLM , which transfers the binary-space denoising perspective of recent visual token generation methods such as BitDance ( Ai et al. 2026 ; Zhuang et al. 2026 ) to text. As shown in Figure 1 , BitLM represents each tokenizer entry as a fixed-length binary code, projects these codes into the hidden dimension of a standard LLM backbone, and performs causal computation in the usual left-to-right manner. Instead of decoding with a vocabulary softmax, the model uses the resulting contextual latent to condition a diffusion head that jointly denoises the binary codes of future tokens. Because the diffusion head operates over multiple positions simultaneously, it can realize a block of tokens in parallel rather than treating each future position as an isolated categorical draw. By adjusting the attention mask from fully causal to block-causal, the model can vary the degree of parallelism while preserving causal dependencies across blocks. The resulting picture is conceptually simple: the backbone is responsible for reasoning about what should happen next , while the diffusion head is responsible for realizing how that continuation should crystallize into discrete symbols . This separation is the central idea of BitLM. It preserves the causal inductive bias that has made autoregressive LLMs so effective, but replaces the final vocabulary-level decision with joint denoising in binary space. Unlike post-hoc acceleration schemes, blockwise parallel generation is therefore not an external add-on; it is a native consequence of the model’s generative interface. More broadly, BitLM suggests that the large-vocabulary softmax is a historical choice rather than a necessary endpoint of language modeling.