MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Experts
Alexandros Christoforos, Chadbourne Davis · 2025
Read MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Experts by Alexandros Christoforos, Chadbourne Davis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20604.txt.
Introduction
Alexandros Christoforos and Chadbourne Davis make diffusion models write long texts without drowning in cost. Text diffusion builds a passage by step-by-step denoising, but dense attention, where every position looks at every other, makes long texts slow and hungry for memory. Their method, MoE-DiffuSeq, trims the attention pattern and adds a mixture of experts, a design that routes each piece of text to a few chosen subnetworks.
Sparse attention cuts the squared cost, while the expert routing adds skill only where it is needed. A soft absorbing state inside the process reshapes how attention behaves during denoising, which speeds up rebuilding and sharpens each token as it is refined. On long-text benchmarks, the system beats prior diffusion models and sparse-attention rivals in training speed, sampling speed, and output quality. Long science texts, big code bases, and extended chats are the intended uses.
Catalog record
| CREATOR | Alexandros Christoforos, Chadbourne Davis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20604
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoE-DiffuSeq: Enhancing Long-Document Diffusion Models with Sparse Attention and Mixture of Experts Alexandros Christoforos, Chadbourne Davis Abstract In this paper, we present MoE-DiffuSeq, an innovative approach leveraging a Mixture of Experts (MoE) to enhance diffusion models for generating long documents. Traditional diffusion models, such as DiffuSeq, face significant computational inefficiencies and memory constraints when processing extended sequences. To overcome these challenges, we integrate a sparse attention mechanism with the MoE framework, facilitating efficient and scalable sequence generation. Our methodology incorporates a bespoke sparse attention framework designed to minimize computational complexity while preserving text quality and coherence. Additionally, we introduce a tailored soft absorbing state within the diffusion process to optimize the attention architecture, which accelerates sequence reconstruction and enhances precision. Empirical evaluations reveal that MoE-DiffuSeq substantially outperforms existing models in terms of training speed and sampling rates. These improvements are particularly beneficial for managing extended sequences, rendering our model apt for generating scientific articles, managing code repositories, and facilitating long dialogues. Benchmark comparisons further affirm that MoE-DiffuSeq enhances efficiency, speed, accuracy, and expressiveness, thereby advancing the capabilities of diffusion models for high-quality long-form text generation. Intorduction Recent advancements in neural network architectures, especially in the field of natural language processing (NLP), have significantly enhanced performance across a diverse array of tasks. The introduction of the Transformer model by 28 ; 47 ; 44 has revolutionized text processing by adeptly capturing contextual relationships within the text. However, traditional transformer architectures encounter substantial computational challenges when processing long sequences, primarily due to their inherent quadratic complexity. To address these challenges, architectures such as the Longformer ( 4 ; 36 ; 35 ) have been developed. These models incorporate sparse attention mechanisms, enabling more efficient management of extended sequences without compromising the model’s ability to handle complex dependencies. Figure 1: Sparse Attention Alongside advancements in transformer architectures, diffusion models have risen as a robust framework for generative tasks across multiple domains ( 7 ; 5 ; 32 ; 27 ; 52 ; 51 ; 45 ; 37 ; 46 ) . Originally developed for image synthesis, these models have been adeptly repurposed for sequence generation, exemplified by DiffuSeq ( 15 ) . DiffuSeq harnesses diffusion principles to facilitate text generation, delivering competitive results while mitigating computational inefficiencies ( 15 ) . Despite these advancements, significant challenges remain, notably in the areas of slow model convergence and the substantial computational demands of training diffusion models. The challenge of efficiently handling long sequences in text generation is further accentuated by the inherent limitations of large language models (LLMs). Typically trained on text segments no longer than 8,000 tokens, these models often falter when tasked with processing significantly larger inputs, leading to a notable decline in performance ( 19 ) . Such limitations hinder their applicability in scenarios that demand a comprehensive understanding of Long Contexts, such as the transcription of scientific articles, generation of source code repositories, or management of extended dialogues. To mitigate these issues, various strategies have been employed, including the adoption of relative positional encodings and sliding-window attention mechanisms ( 4 ; 21 ; 18 ; 43 ; 41 ) . These approaches are designed to curtail memory consumption while preserving model effectiveness. However, they typically necessitate extensive retraining and may still fall short of fully addressing the fundamental challenges. In this paper, we introduce MoE-DiffuSeq, a novel architecture that synergizes the Mixture of Experts (MoE) paradigm with the established DiffuSeq framework, augmented by a sparse attention mechanism (see in Figure 1 ). This fusion is designed to capitalize on the strengths of both systems, enhancing efficiency and scalability for generating lengthy documents. By integrating a bespoke sparse attention framework, MoE-DiffuSeq significantly reduces computational complexity while preserving the quality and coherence of the text. Furthermore, we incorporate a customized soft absorbing state within the diffusion process, which optimizes the attention architecture, thereby accelerating sequence reconstruction and enhancing precision. This methodological innovation not only streamlines the generative process but also ensures high fidelity in the output, making it a robust solution for extensive text generation tasks. Our approach strategically addresses both the maintenance of high generative quality and the reduction of computational overhead, thereby optimizing performance for the efficient management of extensive textual inputs. By integrating sparse attention with the Mixture of Experts (MoE) framework, we significantly reduce the computational load. This reduction is achieved by minimizing the number of attention computations required, which is critical for optimizing the efficiency of iterative diffusion processes. This streamlined computation not only accelerates training convergence but also enhances sampling efficiency during inference, potentially surpassing the capabilities of traditional full-attention mechanisms in terms of generative quality. Such advancements establish our model as a robust solution for complex text generation tasks, ensuring both speed and accuracy without sacrificing the depth and coherence of the generated content. Furthermore, MoE-DiffuSeq integrates cutting-edge advancements, including DPM-solver++ ( 24 ) , to substantially reduce the number of diffusion steps required, thus accelerating the text generation process without sacrificing quality. This optimization proves particularly beneficial for applications such as machine translation and document summarization, where the demand for rapid, high-quality text generation is paramount. By enhancing the efficiency of the diffusion process, MoE-DiffuSeq enables more agile responses in dynamic environments, setting a new standard for performance in complex NLP tasks. The principal contributions of this paper are summarized as follows: • Integration of Mixture of Experts (MoE) with DiffuSeq: We introduce MoE-DiffuSeq, a pioneering architecture that seamlessly integrates the Mixture of Experts paradigm into the DiffuSeq framework. This synthesis significantly enhances the efficiency and scalability of diffusion models tailored for generating extensive documents. By dynamically selecting the most relevant experts for varying segments of the text, our model adeptly manages the computational complexities traditionally associated with processing long sequences. • Incorporation of Sparse Attention Mechanism: Our methodology incorporates a tailored sparse attention mechanism within the MoE-DiffuSeq model, significantly reducing the number of attention computations needed. This reduction not only lowers the overall computational burden but also ensures the efficient processing of extensive textual inputs without sacrificing the quality or coherence of the generated text, which is essential for producing high-quality long documents. • Empirical Validation and Performance Improvements: Through rigorous empirical evaluations, MoE-DiffuSeq has demonstrated superior performance over existing methods across various NLP tasks, particularly in handling long texts with reduced computational demands. The integration of state-of-the-art techniques, such as DPM-solver++, further augments our model’s efficiency by minimizing the necessary diffusion steps, thereby speeding up both the training and sampling phases without compromising the quality of text generation. These contributions underline our model’s innovative approach to combining robust machine learning paradigms and optimizing them for advanced text generation tasks, setting a new benchmark for future research in the field. Related Work In the rapidly evolving field of text generation, diffusion models have become prominent due to their unique ability to handle complex generative tasks. This section provides an overview of the related work, placing our contributions within the broader context of recent advancements in the field. Mixture of Experts The Mixture of Experts (MoE) approach has revolutionized the scalability of neural networks by dynamically allocating computational resources across a diverse set of expert networks ( 14 ; 9 ; 39 ) . This