Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation
Yuhang Chen, Xianfeng Wu, Jinhao Duan, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Parish Aggarwal, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Tianlong Chen · 2026
Read Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation by Yuhang Chen, Xianfeng Wu, Jinhao Duan, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Parish Aggarwal, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Tianlong Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27732.txt.
Introduction
Diffusion language models unmask tokens in parallel, which is fast. Their design forces a dilemma, though. Bidirectional attention gives quality but blocks the key-value caching that makes serving cheap. Causal attention allows caching but loses all right-side context, hurting quality. Yuhang Chen and thirteen colleagues propose Bifocal diffusion models in 2026 to escape the dilemma. The context design is asymmetric, like bifocal lenses.
Their model, R2LM, pairs two mechanisms. Standard causal attention supplies precise left context with full cache compatibility. A lightweight reverse Mamba sidecar supplies compressed right-side context without breaking cacheability. Experiments continued pretraining of Qwen3-1.7B on sixty billion tokens. Throughput ran from about twofold to nearly thirteenfold higher than bidirectional diffusion models in batch serving. Speedups over autoregressive baselines ranged from about twofold to threefold. Quality beat the causal baseline on most benchmarks. It also surpassed the bidirectional diffusion model on average.
Catalog record
| CREATOR | Yuhang Chen, Xianfeng Wu, Jinhao Duan, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Parish Aggarwal, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Tianlong Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation Yuhang Chen, Xianfeng Wu, Jinhao Duan, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Parish Aggarwal, Frank Shyu, Luke Simon, Sandeep Pandey, Xi Liu, Tianlong Chen Abstract Discrete diffusion language models (dLLMs) recover masked tokens in parallel, offering significant speedups over autoregressive (AR) generation. However, such promising frameworks face a fundamental architectural design dilemma: ❶ Adopting bidirectional attention achieves strong generation quality by allowing each position to access the full context, but is inherently incompatible with KV caching, limiting inference throughput in batch-serving scenarios; ❷ Conversely, causal attention enables efficient cached inference but loses all right-side context, substantially degrading generation quality. This paper introduces Bifocal dLLMs, a new paradigm that resolves this dilemma through asymmetric bidirectional context . Analogous to bifocal lenses, we instantiate the paradigm as R2LM (Right-to-Left Mamba), which combines two complementary mechanisms: a a ) standard causal attention providing precise left-context with full KV cache compatibility, while b b ) a lightweight reverse Mamba SSM sidecar supplying compressed right-side context without breaking cacheability. Comprehensive experiments on continued pretraining of Qwen3-1.7B with 60B tokens demonstrate that R2LM achieves 2.4 × 2.4\times to 12.9 × 12.9\times higher throughput than bidirectional dLLMs and 1.9 × 1.9\times to 2.9 × 2.9\times speedup over AR baselines in batch serving through parallel decoding with KV caching, while exceeding the causal baseline on most benchmarks and surpassing the bidirectional dLLM on average. † † affiliation: Meta AI † † affiliation: University of North Carolina at Chapel Hill † † contribution: Work done during an internship at Meta. † † contribution: Joint corresponding authors. † † correspondence: Xi Liu ( [email protected] ) and Tianlong Chen ( [email protected] ) † † date: August 9, 2026 1 Introduction Discrete diffusion language models (dLLMs) ( 2 ; 29 ; 32 ; 37 ) are a family of generative language models in which text is produced by iteratively denoising masked sequences. They have recently emerged as a promising alternative to the autoregressive (AR) paradigm ( 4 ; 33 ; 35 ) dominant in language modeling, matching AR quality at scale ( 25 ; 36 ; 20 ) . The appeal is throughput: AR generation requires N sequential forward passes to produce N tokens, regardless of available parallelism. dLLMs recover multiple tokens per step and produce the same N tokens in T\ll N passes, delivering 3 to 8\times wall-clock speedups over optimized AR engines. However, the dLLM paradigm itself faces an internal architectural dilemma. ❶ Most high-quality dLLMs ( 25 ; 36 ) employ bidirectional attention , allowing every position to aggregate information from all others. It maximizes the context available, but each token’s key-value (KV) representation depends on all other tokens (including future, yet-to-be-resolved ones), so standard prefix KV caching is impossible and every denoising step must recompute the full attention. ❷ An alternative line of work adopts causal attention ( 1 ; 20 ; 6 ) , preserving the standard left-to-right dependency structure and thereby enabling KV caching. But causal attention loses all right-side context. Figure 1 : Left : the asymmetric bidirectional context idea at a glance, contrasting bidirectional dLLMs (full attention, no KV cache), causal dLLMs (KV cache, no right context), and Bifocal (causal attention plus an R2LM sidecar that supplies right-side context while preserving cacheability). Right : quality versus throughput at P{=}4096 , B{=}8 on a single H100. R2LM sits in the upper-right region (high quality, high throughput). This dilemma has driven a range of compromise paradigms summarized in table 1 . Block-diffusion variants ( 1 ; 6 ; 34 ) restrict bidirectional attention to fixed-size blocks, recovering partial KV caching at the cost of bounding right-side context to a single block. Purely causal variants ( 20 ; 28 ) introduce topological reordering or specialized masking schedules to maximize prefix cacheability, but accept the right-context quality gap. Hybrid AR-diffusion variants ( 17 ; 21 ; 30 ) combine autoregressive and diffusion modes through dual-mode architectures or slot-level reordering, at the cost of architectural complexity. Across all cases, the central tension persists: causal attention restricts each position to left-only context, and no amount of inference optimization can recover the missing right-side information. We thus ask: Can right-side context be supplied through a non-attention pathway, leaving the prefix KV cache intact? We propose Bifocal dLLMs , a new dLLM paradigm that supplies right-side context through a non-attention pathway in parallel with causal attention ( fig. 1 ; rightmost row of table 1 ). The underlying design principle, asymmetric bidirectional context , treats the two directions of context flow through different mechanisms rather than the symmetric handling of bidirectional attention—preserving the prefix KV cache that causal attention enables. We instantiate this paradigm as R2LM : the right-context pathway is a reverse Mamba SSM ( 13 ) sidecar attached to a pretrained autoregressive backbone via forward hooks at every k -th decoder layer, with zero-initialized contribution so that the augmented model is bit-identical to the causal Transformer at initialization. The R2LM layers are position-aware by construction: as a sequential state space model, Mamba processes tokens in order, maintaining the positional structure that distinguishes meaningful right context from mere capacity. The plug-in variant of section 4.4 , in which the R2L pathway is trained on top of a frozen MDLM-adapted backbone, isolates this property: with only 5 B of additional tokens, the plug-in recovers 59 % of the joint model’s long-target gain over the causal baseline. Bifocal is neither causal nor bidirectional , but a distinct paradigm. The attention mechanism remains strictly causal, ensuring KV cache validity. The overall information flow is bidirectional: each position’s representation incorporates both precise left context (via attention) and compressed right context (via the R2LM sidecar). This asymmetry is the source of both quality and efficiency: where bidirectional attention pays the same per-position cost in both directions, Bifocal delegates the right-context pathway to a lower-cost SSM, while preserving cacheable causal attention as the primary information channel. Our main contributions are: • We introduce Bifocal dLLMs , a new diffusion-language-modeling paradigm built on the principle of asymmetric bidirectional context . This is the first design that delivers continuous, position-aware right-side context while preserving standard causal prefix KV caching. • We instantiate the paradigm as R2LM , a reverse state space sidecar attached to a pretrained autoregressive backbone via forward hooks with zero-initialized contribution. The same architecture supports both joint continued pretraining and a frozen-backbone plug-in mode, allowing integration into existing AR checkpoints without disturbing the pretrained model. • We show empirically through controlled three-way continued pretraining of Qwen3- 1.7 B with 60 B tokens that R2LM achieves 2.4 to 12.9\times higher throughput than a bidirectional dLLM while exceeding the causal baseline on most benchmarks and surpassing the bidirectional dLLM on the ALL average. 2 Related work Table 1 : Taxonomy of dLLM paradigms. Bifocal dLLM is a paradigm whose attention pattern is Causal + \underline{X} , where \underline{X} denotes a right-context module attached as a residual to a causal Transformer backbone; the choice of \underline{X} is an open architectural design space. We instantiate \underline{X} as a reverse-direction Mamba SSM ( R2LM , section 3.4 ). Paradigm Method Attention Right Context KV Cache Bidirectional dLLM