Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
Shuaiyi Li, Zhisong Zhang, Yan Wang, Lei Zhu, Dongyang Ma, Chenlong Deng, Yang Deng, Wai Lam {sli, wlam}@se.cuhk.edu.hk , [email protected] · 2026
Read Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation by Shuaiyi Li, Zhisong Zhang, Yan Wang, Lei Zhu, Dongyang Ma, Chenlong Deng, Yang Deng, Wai Lam {sli, wlam}@se.cuhk.edu.hk , [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15913.txt.
Introduction
Shuaiyi Li and seven colleagues advance block attention, which splits input into blocks that cannot attend to each other, for long-context RAG. Two hurdles block wider use: cutting text into meaningful stand-alone blocks, and tuning blocks without harming results. The authors build SemanticSeg, a segment dataset of over 30,000 cases across 16 kinds, and train a segmenter making natural blocks at set sizes. Their block distillation method, taught by a frozen full-attention teacher, uses block sink tokens, block dropout and token loss weights to reach near-full-attention results.
Catalog record
| CREATOR | Shuaiyi Li, Zhisong Zhang, Yan Wang, Lei Zhu, Dongyang Ma, Chenlong Deng, Yang Deng, Wai Lam {sli, wlam}@se.cuhk.edu.hk , [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52522 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15913
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation Shuaiyi Li, Zhisong Zhang, Yan Wang, Lei Zhu, Dongyang Ma, Chenlong Deng, Yang Deng, Wai Lam {sli, wlam}@se.cuhk.edu.hk , [email protected] Abstract Block attention, which processes the input as separate blocks that cannot attend to one another, offers significant potential to improve KV cache reuse in long-context scenarios such as Retrieval-Augmented Generation (RAG). However, its broader application is hindered by two key challenges: the difficulty of segmenting input text into meaningful, self-contained blocks, and the inefficiency of existing block fine-tuning methods that risk degrading performance. To address these, we first construct SemanticSeg, a large and diverse semantic segmentation dataset containing over 30 k 30k instances across 16 categories—including books, code, web text, and conversations with text lengths ranging from 2 k 2k to 32 k 32k . Using this dataset, we train a lightweight segmenter to automatically partition text into human-instinct-aligned blocks with controllable granularity. Second, we propose block distillation, a training framework that is more efficient than block fine-tuning, which uses a frozen full-attention teacher model to guide the block-attention student. This framework integrates three novel components: block sink tokens to mitigate information loss at block boundaries, block dropout to leverage training signals from all blocks, and token-level loss weighting to focus learning on block-attention-sensitive tokens. Experiments across multiple models and benchmarks demonstrate that our segmenter outperforms heuristic and statistical baselines, and block distillation achieves near-full-attention performance under block attention, establishing a practical and scalable pathway for deploying block attention. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in processing long-context inputs [ 2 , 1 , 13 ] , enabling applications such as multi-document question answering, coding, etc. However, the standard full-attention mechanism scales quadratically with sequence length, making inference on long inputs computationally expensive and memory-intensive. A significant source of this inefficiency is the context-dependent nature of full attention: when identical context is paired with different prefixes, its key-value (KV) states must be recomputed from scratch. This leads to substantial waste of compute and energy, particularly in retrieval-augmented generation (RAG) scenarios where overlapping document sets are repeatedly processed across queries. To mitigate this, block attention [ 12 ] has emerged as a promising alternative. By restricting self-attention to independent blocks and allowing only a final aggregation block to attend globally, it eliminates cross-block dependencies and facilitates the reuse of pre-computed KV cache. Nevertheless, its practical adoption is hindered by several obstacles. An important barrier in the application of block attention is segmentation, that is, how to divide the input sequence into separate blocks. Existing approaches often rely on heuristic rules, such as splitting with newlines; however, such rules rarely generalize and are highly likely to break the semantic coherence of the inputs. As demonstrated in Section 5.2 , such naive segmentation leads to performance degradation, highlighting the need for a semantic-aware approach. To address this, we propose a robust, data-driven semantic segmenter capable of handling diverse input formats. We take a data-driven approach and construct a segmentation dataset ( SemanticSeg ), where each sample is segmented by the semantic meaning. Using this dataset, we train a neural segmenter that can automatically produce adaptive and context-aware boundaries, overcoming a major obstacle to the generalization of block attention. Another major challenge is effectively integrating block attention into existing LLMs. While training-free strategies such as Prompt Cache [ 6 ] and Superposition prompting [ 14 ] attempt to enable KV state reuse or parallel processing, such direct application of block attention into general domains suffers from severe performance degradation compared to full attention (Table 3 ). This indicates that these approaches are not sufficiently effective for supporting reliable block attention, further highlighting the necessity of specialized training. [ 12 ] addressed this through “block fine-tuning”, which trains models on both block and full attention patterns to balance specialized performance with general capability. However, this approach is computationally expensive and generalizes poorly across diverse domains (Section 5.2 ). To address these limitations, we introduce Block Distillation , a training framework designed for higher efficiency and signal density. It incorporates three novel mechanisms: block sink tokens to counteract information loss at block boundaries, block dropout to exploit training signals from all blocks, and token-level loss weighting to emphasize tokens that are most sensitive to block attention. To validate our approach, we conduct comprehensive experiments across multiple models and benchmarks. To verify the effectiveness of our segmenter, we quantify the impact of segmentation on downstream performance and compare our segmenter against a range of heuristic and statistical segmentation baselines. To evaluate our training framework, we rigorously tested Block Distillation , evaluating its efficiency and performance in general domains and the specific contributions of its individual components. Experimental results demonstrate that our segmenter consistently outperforms all baselines, and Block Distillation pushes block-attention performance close to the full-attention upper bound while preserving or even improving full-attention capability, establishing a practical and scalable pathway for deploying block attention in long-context applications. We discuss the potential application of the block attention in Appendix D . 2 Preliminary We illustrate the idea of block attention with the example of RAG. Consider a pool of r documents and two instances with overlapping retrieved contexts: Inst 1: Document [ i ]; Document [ i+1 ]; …; Document [ i+x ]; …; [Query q_{x} ]. Inst 2: Document [ j ]; Document [ j+1 ]; …; Document [ j+y ]; …; [Query q_{y} ]. where the document sets intersect at \{i,\dots,i+x\}\cap\{j,\dots,j+y\}=\{n,\dots,m\} . In standard full attention, the KV states for this intersection must be recomputed for each query because the attention mechanism is prefix-dependent, resulting in significant computational overhead and energy waste. However, if encoding can be performed independently for each document, the KV states for the intersection \{n,\dots,m\} become prefix-agnostic and can be safely reused across disparate queries. Block attention [ 12 ] formalizes this by partitioning the input into independent blocks. Each block employs self-attention restricted to its own tokens, ensuring that internal representations are decoupled from other blocks. Only the final block (typically the user query) is permitted to utilize full attention, aggregating information from all preceding KV caches. The implementation of this method can be achieved easily via the following steps: 1) Independently encoding each block except the last one; 2) Computing the positional encoding for each token based on their position in the input text; 3) Concatenating all pre-computed KV states of the blocks and using them to compute the KV states for the final block. However, the previous work [ 12 ] has demonstrated that the model cannot accommodate this pattern without training, which is also verified in this work (section 5.2 ). To cope with this challenge, they employ block fine-tuning, which updates the parameters in both ways, one for block attention and one for full attention. They claim this could enhance block-attention performance while maintaining full attention capability. However, despite its heavy updating scheme, it struggles to generalize to other domains ( 5.2 ). 3 Automatic Segmentation A major barrier to the general application of block attention is segmentation, i.e., given the input text, how to cut it into meaningful, self-contained, and human-instinct-aligned blocks (or chunks) for later