A Survey on Transformer Context Extension: Approaches and Evaluation

Yijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, Qingfu Zhu · 2025

Read A Survey on Transformer Context Extension: Approaches and Evaluation by Yijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, Qingfu Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13299.txt.

Introduction

Yijun Liu and co-workers surveyed ways to stretch model text models to long documents in a 2025 paper. Models that shine on short text degrade on long inputs. For reasons ranging from position encodings that never saw such lengths to focus costs that explode. The survey first catalogs those failure modes, then organizes the repair literature into a four-part taxonomy for builders choosing a plan.

The groups cover positional coding fixes, context squeezing, lookup augmentation, and redesigned focus patterns. Review gets equal weight, with data sets, tasks, and metrics from current long-context benchmarks organized for check. Open problems close the survey, which serves newcomers as a map and veterans as a checklist of what remains unsolved.

Catalog record

CREATORYijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, Qingfu Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey on Transformer Context Extension: Approaches and Evaluation Yijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, Qingfu Zhu Abstract Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP), demonstrating strong performance, particularly in handling short text tasks. However, when it comes to long context scenarios, the performance of LLMs degrades due to some challenges. To alleviate this phenomenon, there is a number of work proposed recently. In this survey, we first list the challenges of applying pre-trained LLMs to process long contexts. Then systematically review the approaches related to long context and propose our taxonomy categorizing them into four main types: positional encoding, context compression, retrieval augmented, and attention pattern. In addition to the approaches, we focus on the evaluation of long context, organizing relevant data, tasks, and metrics based on existing long context benchmarks. Finally, we summarize unresolved issues in the long context domain and put forward our views on future developments. 1 Introduction In recent years, the Transformer ( Vaswani, 2017 ) architecture has made significant progress in many NLP tasks ( Devlin, 2018 ; Radford, 2018 ; Lewis, 2019 ; Raffel et al., 2020 ; Brown, 2020 ; Chen et al., 2021a ; Cobbe et al., 2021 ) , and has become the foundational model of many applications. Large language models can handle tasks involving short texts, within the pre-trained context length. However, current scenarios always involve longer texts, such as book-/repo- level tasks ( Sharma et al., 2019 ; Liu et al., 2023a ; Zhang et al., 2023a ; Liu et al., 2023b ) , dialogue systems with long contexts ( Dey et al., 2022 ; Li et al., 2024a ) , content-rich in-context learning ( Li et al., 2024c ) and so on. The performance of the pre-trained LLMs degrades and the models often fail to utilize the complete knowledge contained within the long context inputs. This may be caused by three inherent challenges: out-of-distribution (OOD) problem ( Han et al., 2024 ) , "Lost in the Middle" phenomenon ( Liu et al., 2024a ) , and quadratic complexity of attention ( Zhou et al., 2024 ) . Recently, a lot of work has been proposed to improve and evaluate models’ ability to handle long contexts in the community. This survey focuses on approaches and evaluation in the long context field, systematically reviewing existing related work. As illustrated in Figure 1 , we propose a novel taxonomy for approaches, categorizing them into four main groups: positional encoding, context compression, retrieval augmented, and attention pattern. Additionally, we focus on the evaluation aspect and organize work on data, tasks, and metrics based on existing benchmarks. In addition to the two main parts of approaches and evaluation, we present our viewpoints on the current unsolved issues and potential future directions in the long context domain. To illustrate the current status more theoretically, we also list the main challenges in the field of long context before introducing specific work. Although most existing methods and benchmarks have not corresponded to them, these challenges are still instructive for the development of approaches and evaluation. Figure 1: Framework of survey. We first list three inherent challenges in Section 2 . And then we systematically review related approaches and propose a novel taxonomy with four major categories in Section 3 . Next, in Section 4 , we organize the evaluation aspect from three perspectives: data, tasks, and metrics based on existing benchmarks. At last, we show our views on future roadmap and open problems in Section 5 . There are also some surveys that focus on the long context domain. They each have their own emphasis, but there is no systematic and comprehensive survey of approaches and evaluation in the field of long context, which can provide researchers with a quick and efficient guide. Some surveys only include a part of the methods, lacking a comprehensive overview of the approaches related to long context. Zhao et al. (2023) focus on work addressing length extrapolation from the perspective of positional encoding, while there are some surveys from the perspective of KV Cache ( Li et al., 2025 ; Shi et al., 2024 ) . Besides, though some surveys have categorized existing work, their taxonomies are not clear, and there are overlaps between categories. For example, Huang et al. (2023) divide the methods for enhancing Transformer architecture models into five categories, but some existing methods can belong to multiple categories. And ( Pawar et al., 2024 ) also has this problem, which distinguishes existing techniques into two categories: interpolation and extrapolation. Also, some surveys even involve some common methods that not specifically designed for long contexts. Dong et al. (2023b) provide an overview of the text-preprocessing methods, architectures, special characteristics and application for long context, but they cover some general topics. What’s more, these surveys pay little or even no attention to the evaluation aspect. To fill the above gap, our survey proposes a novel and comprehensive taxonomy on both approaches and evaluation aspects. It is worth noting that we focus on work that applies Transformer-based models to long text tasks, but not improving Transformers (nor other architectures) in a universal scenario. That is to say, this survey does not cover fields like long chain-of-thought reasoning ( Chen et al., 2025 ) , multimodal long context ( Song et al., 2024 ; Qiu et al., 2024 ) , efficient Transformer ( Zhou et al., 2024 ) , and State Space Model (SSM) ( Wang et al., 2024c ) . In addition, the long context we focus on is the long input content, rather than the introduction of external knowledge in the Retrieval-Augmented Generation (RAG) scenario ( Yu et al., 2024 ; Zhao et al., 2024 ; Fan et al., 2024 ) . 2 Challenges When applying pre-trained LLMs to the long context scenarios, there are some inherent challenges affecting models’ performance. We list the three most important and common challenges: OOD problem, "Lost in the Middle" phenomenon, and quadratic complexity. OOD Problem When processing sequences that exceed pre-trained context window length, the models face out-of-distribution (OOD) problems. Han et al. (2024) verify theoretically and empirically that three key factors contribute to OOD issues, thereby limiting models’ extrapolation capabilities: 1) unseen inter-token distances, 2) increased number of attended tokens, and 3) implicitly encoded position information of the starting tokens. "Lost in the Middle" Phenomenon Liu et al. (2024a) discover the "Lost in the middle" phenomenon through experiments that when LLMs receive a long input, they tend to focus on the information at both the beginning and end of the input sequence. At the same time, they neglect the content in the middle, thus failing to capitalize on the key information within the long input. Quadratic Complexity Due to the quadratic complexity of attention, directly using pre-trained LLMs for training or inference on long context is time and resource consuming ( Zhou et al., 2024 ) . The above are three inherent challenges in the field of long context, and some existing methods have alleviated them to a certain extent. But it is worth noting that most of the methods do not start from this perspective.

Read the complete text