ScaleFormer : Span Representation Cumulation for Long-Context Transformer Conference: Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region; December 7–10, 2025; Xi’an, China Proceedings of the 2025 Annual In

Jiangshu Du, Wenpeng Yin, Philip Yu · 2025

Read ScaleFormer : Span Representation Cumulation for Long-Context Transformer Conference: Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region; December 7–10, 2025; Xi’an, China Proceedings of the 2025 Annual In by Jiangshu Du, Wenpeng Yin, Philip Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10029.txt.

Introduction

Jiangshu Du, Wenpeng Yin, and Philip Yu propose ScaleFormer, a plug-and-play way to give pre-trained encoder-decoder transformers a long memory. Standard self-attention grows quadratically with length, which blocks long documents, and efficient variants usually demand new architectures plus costly retraining. ScaleFormer instead cuts a long input into overlapping chunks and builds a compressed, context-aware representation for the decoder.

Its fusion trick is parameter-free: each chunk's boundary representations are enriched with cumulative context vectors gathered from every chunk before and after it. That gives the model a sense of the document's overall flow while keeping complexity linear. On long-document summarization the method proved highly competitive with, and often better than, state-of-the-art systems, without architectural changes or external retrieval.

Catalog record

CREATORJiangshu Du, Wenpeng Yin, Philip Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScaleFormer : Span Representation Cumulation for Long-Context Transformer Conference: Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region; December 7–10, 2025; Xi’an, China Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP 2025), December 7–10, 2025, Xi’an, China DOI: 10.1145/3767695.3769513 ISBN: 979-8-4007-2218-9/2025/12 CCS: Computing methodologies Natural language generation CCS: Computing methodologies CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing Jiangshu Du, Wenpeng Yin, Philip Yu Abstract . The quadratic complexity of standard self-attention severely limits the application of Transformer-based models to long-context tasks. While efficient Transformer variants exist, they often require architectural changes and costly pre-training from scratch. To circumvent this, we propose ScaleFormer ( S pan Representation C umul a tion for L ong-Cont e xt Trans former )—a simple and effective plug-and-play framework that adapts off-the-shelf pre-trained encoder-decoder models to process long sequences without requiring architectural modifications. Our approach segments long inputs into overlapping chunks and generates a compressed, context-aware representation for the decoder. The core of our method is a novel, parameter-free fusion mechanism that endows each chunk’s representation with structural awareness of its position within the document. It achieves this by enriching each chunk’s boundary representations with cumulative context vectors from all preceding and succeeding chunks. This strategy provides the model with a strong signal of the document’s narrative flow, achieves linear complexity, and enables pre-trained models to reason effectively over long-form text. Experiments on long-document summarization show that our method is highly competitive with and often outperforms state-of-the-art approaches without requiring architectural modifications or external retrieval mechanisms. 1. Introduction The advent of Transformer-based pre-trained language models (PLMs) has marked a paradigm shift in Natural Language Processing (NLP) ( 21 ) . However, the Transformer’s foundational self-attention mechanism scales quadratically with input sequence length, creating a substantial barrier to its application in domains that rely on long-form text, such as summarizing scientific articles, analyzing legal documents, or processing entire books ( 1 ; 25 ; 2 ) . To address this long-context barrier, researchers have developed numerous efficient Transformer variants, often based on sparse attention patterns that reduce complexity from quadratic to linear or near-linear. While effective, these models typically introduce custom architectural changes that require expensive pre-training from scratch. As a result, a critical research direction has emerged: adapting existing powerful models for long-context tasks. This raises a central research question: How can we adapt existing PLMs to handle long-form text without incurring the prohibitive cost of pre-training new models from scratch? The landscape of solutions is diverse. One major line of work involves fundamental architectural modifications to create efficient Transformers, using techniques like sparse attention ( 1 ; 25 ) or recurrence ( 6 ) . Another line focuses on hierarchical pipelines that first extract salient information before processing it ( 20 ) . A third—and most relevant to our work—centers on plug-and-play frameworks designed to wrap around existing PLMs ( 10 ; 7 ; 19 ; 4 ) . Prominent examples include SLED ( 10 ) , which employs a "fusion-in-decoder" strategy ( 11 ) by concatenating all encoded chunks, and Unlimiformer ( 2 ) , which augments the model with a retrieval mechanism that accesses relevant parts of the long input from a k-nearest neighbor (kNN) index. While effective, these plug-and-play methods place the full burden of discerning the document’s overall structure and narrative flow on the decoder, which must infer these relationships from a long sequence of concatenated chunk representations. In this work, we introduce ScaleFormer (Span Representation Cumulation for Long-Context Transformer)—a novel framework that offers an alternative approach by explicitly encoding structural information into the input representations before they reach the decoder. Our key innovation lies in an efficient and parameter-free fusion scheme. Instead of passing all encoded tokens from every segment to the decoder, we represent each segment using only its “boundaries”—the hidden states of its leftmost and rightmost tokens. These boundary representations are then dynamically enriched through our novel span representation cumulation mechanism. This technique provides each segment with positional awareness by combining its left boundary with a summary of all preceding text and its right boundary with a summary of all succeeding text. This approach achieves linear complexity while enabling off-the-shelf PLMs to perform effectively on long-context tasks. Our primary contributions are as follows: (1) We propose ScaleFormer , a novel and efficient framework for applying pre-trained encoder-decoder models to long sequences with linear complexity, without requiring any modifications to the model’s internal architecture. (2) We introduce a directional boundary fusion mechanism that provides each segment with cumulative, position-aware context, explicitly encoding the document’s structure into each chunk’s representation. (3) We conduct extensive experiments on long-document summarization tasks, demonstrating that our method achieves highly competitive performance against strong baselines. 2. Related Work Transformer models have been adapted in various ways to handle long sequences by reducing quadratic self-attention costs. Sparse-attention approaches like Longformer ( 1 ) and BigBird ( 25 ) limit each token’s attention to local windows or selected global tokens. Other efficient attention approximations include Reformer’s hashing-based method ( 13 ) , Routing Transformers’ clustering ( 18 ) , and low-rank or kernel methods such as Linformer ( 22 ) and Performer ( 5 ) . More recent work continues to push these boundaries, with novel architectures like Infini-attention proposing methods to scale to theoretically infinite contexts by integrating a compressive memory directly into the attention mechanism ( 16 ) . These models often require architecture changes and expensive pre-training. In contrast, our method leverages existing pre-trained Transformers without modifying their internal attention mechanism, enabling the efficient reuse of these powerful models. Retrieval-based methods like RAG ( 15 ) and REALM ( 8 ) retrieve relevant passages before processing them. The Fusion-in-Decoder (FiD) ( 11 ) strategy processes passages independently and fuses the encoded representations in the decoder, inspiring various approaches in QA and summarization. Recent chunk-based methods, notably SLED ( 10 ) , segment long inputs and encode chunks separately before decoding. Another parallel line of work adapts decoder-only LLMs using similar chunking principles; for instance, CEPE ( 24 ) also processes long inputs in parallel chunks but requires inserting new cross-attention modules into a frozen decoder to integrate the context. While our method shares SLED’s philosophy of leveraging pre-trained Transformers through chunking, we differ fundamentally in how chunk representations are managed. Instead of using entire chunk encodings, we select only boundary tokens to create compressed representations that are then infused with explicit directional context, a feature absent in prior work. Another related approach, Unlimiformer ( 2 ) , extends input lengths by offloading decoder attention to an external kNN index. This allows for extremely long contexts but adds complexity. In contrast, our model operates within the standard Transformer framework without additional indexing, offering a simpler solution. Overall, our method balances efficiency and simplicity by avoiding architectural modifications, extensive retraining, and external retrieval structures. 3. The ScaleFormer Framework We propose ScaleFormer , a simple yet effective framework for applying existing PLMs to long-sequence tasks. The core idea is to segment long inputs into overlapping chunks and

Read the complete text