Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation
Junhao Zhang, Richong Zhang, Fanshuang Kong · 2025
Read Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation by Junhao Zhang, Richong Zhang, Fanshuang Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06868.txt.
Introduction
Junhao Zhang and co-authors tackle long-text writing by language models in a 2025 paper. Most tools write long outputs from short prompts. Writing long text from long input is a distinct, harder task. Models tend to lose facts buried in the middle of long inputs, which breaks coherence.
Their training-free framework splits the job between a planner and a writer. The planner lays out steps, and the writer drafts each step. The writer scores input parts by meaning and position, pulls the key parts, and restates them as it goes. A new benchmark and three scoring metrics test length, coherence, and quality. Results favor their approach.
Catalog record
| CREATOR | Junhao Zhang, Richong Zhang, Fanshuang Kong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06868
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation Junhao Zhang, Richong Zhang, Fanshuang Kong Abstract Existing long-text generation methods primarily concentrate on producing lengthy texts from short inputs, neglecting the long-input and long-output tasks. Such tasks have numerous practical applications while lacking available benchmarks. Moreover, as the input grows in length, existing methods inevitably encounter the “lost-in-the-middle” phenomenon. In this paper, we first introduce a Long Input and Output Benchmark ( LongInOutBench ), including a synthetic dataset and a comprehensive evaluation framework, addressing the challenge of the missing benchmark. We then develop the Retrieval-Augmented Long-Text Writer ( RAL-Writer ), which retrieves and restates important yet overlooked content, mitigating the “lost-in-the-middle” issue by constructing explicit prompts. We finally employ the proposed LongInOutBench to evaluate our RAL-Writer against comparable baselines, and the results demonstrate the effectiveness of our approach. Our code has been released at https://github.com/OnlyAR/RAL-Writer 1 Introduction While long-context large language models (LLMs) demonstrate proficient comprehension of extensive inputs, they often fail to generate sufficiently lengthy outputs as human instructions. For instance, when instructed to produce a 10,000-word paper, these models typically yield responses under 2,000 words. Numerous methods have been proposed to enhance the ability of LLMs to adhere to instructions and generate lengthy texts, including multi-step agent inferencing Quan et al. 2024 , long response supervised fine-tuning (SFT) Bai et al. 2024 , and preference alignment techniques Pham et al. 2024 . However, these works primarily focus on scenarios where LLMs generate long content from short inputs, but we believe that in reality, long-input and long-output tasks are also meaningful. For example, generating analytical reports from extensive system logs, creating summaries based on documents, and continuing to write articles using existing content. To the best of our knowledge, there remains a research gap in addressing long-input and long-output tasks, which require both extensive context comprehension and sustained content generation. In the context of long-input and long-output tasks, there are two main challenges: (1) As illustrated in Table 1 , the existing benchmarks for long-context LLMs do not feature both lengthy inputs and outputs simultaneously; (2) When the input is lengthy, it becomes particularly prone to the “lost-in-the-middle” phenomenon ( He et al. 2024 ; An et al. 2024 ; Zhang et al. 2024b ) , wherein LLMs often overlook the content positioned in the middle of the input. We believe this issue will similarly arise (proved in Figure 7(b) ), requiring targeted solutions to ensure the coherence and completeness of generated long outputs. Method Long Input (> 8K) Long Output (> 1K) Real-world Aligned Consistency Evaluation Quality Evaluation NIAH ( Kamradt 2023 ) ✓ ✗ ✗ ✓ ✗ RULER ( Hsieh et al. 2024 ) ✓ ✗ ✗ ✓ ✗ \infty Bench ( Zhang et al. 2024a ) ✓ ✗ ✓ ✓ ✗ SummHay ( Laban et al. 2024 ) ✓ ✗ ✓ ✓ ✗ LongGenBench 1 ( Wu et al. 2024 ) ✗ ✓ ✗ ✗ ✗ LongGenBench 2 ( Liu et al. 2024b ) ✗ ✓ ✗ ✗ ✗ LongWriter ( Bai et al. 2024 ) ✗ ✓ ✓ ✗ ✓ ProxyQA ( Tan et al. 2024 ) ✗ ✓ ✓ ✓ ✗ LongInOutBench (Ours) ✓ ✓ ✓ ✓ ✓ Table 1: Recent representative benchmarks for evaluating long-context LLMs. “Real-world Aligned” refers to tasks within the benchmark that align with real-world application requirements, “Consistency Evaluation” refers to checking the correctness of knowledge during evaluation, and “Quality Evaluation” refers to assessing the language proficiency and structural quality of output content. Figure 1: Overview of LongInOutBench . To address the challenge of the lack of benchmarks, we propose the Long Input and Output Benchmark ( LongInOutBench ), which includes datasets featuring long-input and long-output, as well as a convincing evaluation framework. Specifically, we first manually collect scientific papers from arXiv 1 1 1 https://arxiv.org/ and design a long-text writing task, which involves generating a comprehensive summary based on multiple papers. We then develop an evaluation framework to assess the summary through three criteria, including length evaluation, consistency evaluation, and quality evaluation. As illustrated in Figure 1 , the benchmark requires LLMs to read three full-length academic papers, each spanning hundreds of thousands of tokens, and subsequently generate a summary of prescribed length that faithfully captures the core contributions and critical conclusions from the source materials. Additionally, to mitigate the “lost-in-the-middle” issue, we propose the Retrieval-Augmented Long-Text Writer ( RAL-Writer ), which explicitly identifies and preserves information that is both essential and susceptible to being lost. In general, RAL-Writer comprises a Writing Step Planner and a Retrieve-and-Restate Writer. The Planner is tasked with creating the overarching framework and defining the writing steps for producing long outputs after analyzing extensive inputs. Subsequently, the Writer sequentially generates content by following the steps established during the planning phase. Specially, during the writing phase, we adapt the retrieval-augmented framework to retrieve important but potentially lost contents and restate them to explicitly prompt the LLM for the “lost-in-the-middle” mitigation. In short, the main contributions of this paper are as follows: • We propose the LongInOutBench , which is the first benchmark specially designed for long-input long-output tasks, including relevant datasets and evaluation framework. • We introduce RAL-Writer , which retrieves and restates crucial yet lost content, forming an explicit prompt for “lost-in-the-middle” mitigation. • Comprehensive experiments on LongInOutBench demonstrate the effectiveness of our RAL-Writer . 2 Related Work Relevant prior work includes methods for extending the context length of LLMs, investigations into LLMs’ comprehension mechanisms for long-context information, and alignment techniques for generating high-quality long-form outputs. Long-context LLMs The extended context window is essential for LLMs as it allows them to process more reference information, understand longer documents, and learn from additional examples in few-shot learning. However, processing excessively long contexts can result in significant computational overhead and substantial memory pressure. Dao et al. 2022 significantly reduced the memory dependency of LLMs using IO-aware attention computation mechanism. Methods based on Rotary Position Embedding ( Su et al. 2024 ; Peng et al. 2023 ; Zhu et al. 2023 ) allow LLMs to process extended contexts in inference, despite not having been trained on equivalently long text sequences. LM-Infinite ( Han et al. 2024 ) , LongLoRA Chen et al. 2023 , and LongQLoRA ( Yang 2023 ) employ specialized attention mechanisms to extend the context size of models. Although LLMs can generate text with lower perplexity in long-context scenarios, there remains uncertainty regarding whether the models adequately attend to and effectively utilize this information. Long output generation In the domain of long-form text generation, recent advancements have aimed at enhancing the capabilities of LLMs to produce coherent and high-quality outputs over extended lengths. Suri-I-ORPO ( Pham et al. 2024 ) is a pioneer in this long-form text generation model, employing the I-ORPO method to train models up to approximately 5k output context length. LongWriter ( Bai et al. 2024 ) utilizes the AgentWrite approach to gather datasets for training long-text output models. LongDPO ( Ping et al. 2025 ) improves upon the length and quality metrics by constructing preference data and utilizing DPO for training, building on LongWriter. 3 LongInOutBench Benchmark To effectively assess the model’s ability for long-text understanding and long-text generation,