ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models
Singon Kim, Gunho Jung, Seong-Whan Lee* · 2025
Read ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models by Singon Kim, Gunho Jung, Seong-Whan Lee* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12673.txt.
Introduction
Kim, Jung and Lee present in 2025 ACoRN, a method that makes context compression in retrieval-augmented generation resistant to noisy documents. Compressors shrink retrieved text to save computation, but retrieved sets often include irrelevant or wrong passages. Those traps make compressors drop key facts, especially in long contexts. Their system sorts documents more finely and trains in two extra steps.
Augmented training data teaches the compressor to withstand two noise types. Fine-tuning centers summaries on answer-supporting facts, fixing positional bias. A T5 model trained this way gained accuracy while keeping answer strings intact, most on noise-heavy datasets.
Catalog record
| CREATOR | Singon Kim, Gunho Jung, Seong-Whan Lee* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12673
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models Thanks: This research was supported by the Institute of Information & Communica- tions Technology Planning & Evaluation (IITP) grant, funded by the Korea government (MSIT) (No. RS-2019-II190079 (Artificial Intelligence Graduate School Program (Korea University)), No. RS-2024-00436857 (Information Technology Research Center (ITRC)), No. RS-2024-00457882 (AI Re- search Hub Project), and No. RS-2024-00336673 (AI Technology for Interactive Communication of Language Impaired Individuals). * Seong-Whan Lee is the corresponding author. Singon Kim, Gunho Jung, Seong-Whan Lee* Abstract Abstractive compression utilizes smaller langauge models to condense query-relevant context, reducing computational costs in retrieval-augmented generation (RAG). However, retrieved documents often include information that is either irrelevant to answering the query or misleading due to factual incorrect content, despite having high relevance scores. This behavior indicates that abstractive compressors are more likely to omit important information essential for the correct answer, especially in long contexts where attention dispersion occurs. To address this issue, we categorize retrieved documents in a more fine-grained manner and propose Abstractive Compression Robust against Noise ( ACoRN ), which introduces two novel training steps. First, we use offline data augmentation on the training dataset to enhance compressor robustness against two distinct types of retrieval noise. Second, since the language model-based compressor cannot fully utilize information from multiple retrieved documents and exhibits positional bias, we perform fine-tuning to generate summaries centered around key information that directly supports the correct answer. Our experiments demonstrate that T5-large, trained with ACoRN as a compressor, improves EM and F1 scores while preserving the answer string, which could serve as direct evidence. ACoRN excels on datasets with many accuracy-reducing documents, making it highly useful in real-world scenarios. I Introduction Retrieval-augmented language models (RALMs) [ 1 , 2 ] have strong capabilities in both academic research and industrial applications within the area of natural language processing (NLP). Retrieval-augmented generation (RAG) process in RALMs was designed to improve the performance of large language models (LLMs) [ 17 , 36 ] by integrating external knowledge. It can be achieved by simply appending supporting documents to the input, without the need to update the LLMs. While this approach helps bridge knowledge gaps in LLMs, it significantly increases computational costs, particularly for transformer-based LLMs due to attention complexity [ 20 , 40 ] . With scaling, this burden becomes even more significant [ 3 ] . To mitigate these issues without compromising critical information, abstractive compression has been proposed [ 12 ] . Fig. 1: An illustrative example of a challenge in retrieving and summarizing information supporting to the correct answer from the documents. The compressor performs well in summarizing content supported to the correct answer when only the document including the correct answer is provided. However, it generates incorrect information or misses the key information when the retrieved documents contain inaccurate or irrelevant information. Abstractive compression methods [ 12 , 13 ] leverage the query-focused summarization (QFS) capabilities of language models [ 21 ] to reduce tokenization overhead effectively. However, since retrievers rank documents based on their relevance to the the query [ 31 ] , retrieved documents may contain information with high relevance scores that is either irrelevant to generating the answer or is related but incorrect. Due to limitations of language models, such as limited relevance discrimination [ 8 ] and attention dispersion in long contexts [ 5 ] , existing abstractive compressors struggle with significant information loss when retrieving multiple documents. Existing abstractive compression methods do not address these issues, leading to hesitation in their adoption for real-world applications. To investigate these issues step by step, we refer to information that hinders the LLM from generating the correct answer as noise and call the documents containing such information noise documents . As shown in Fig. 1 , various types of retrieval noise exist in real-world scenarios, and the responses generated by the compressor also vary due to the interference of retrieval noise. We systematically explore two types of retrieval noise: (i) retrieved documents that are thematically related to the query but contain incorrect information ( Factual error documents ), and (ii) retrieved documents lacking sufficient information to answer the query ( Irrelevant documents ) . Existing open-domain question answering (ODQA) [ 19 ] training datasets do not consider the types of noise documents, resulting in only partial robustness to noise and causing a significant performance gap between the training dataset and the test dataset. In this work, we seek to mitigate noise influence efficiently within the scope of the ODQA task. Then we propose Abstractive Compression Robust against Noise ( ACoRN ), a training method that addresses this problem with the following two objectives: (i) mitigating distraction caused by two types of retrieval noise. (ii) reducing the loss of information that directly supports the correct answers in long contexts . Our method, ACoRN, reconstructs the training dataset through offline data augmentation to ensure robustness against two types of retrieval noise, as described in Section III-C . To reduce the loss of information that directly supports the correct answer in long contexts, positional bias must be addressed. This problem, known as the “lost in the middle” phenomenon, often occurs in abstractive compression [ 5 , 34 ] . To address this issue, inspired by FILM [ 3 ] , we propose fine-tuning that targets information directly supporting the correct answer. To create training dataset labels that capture key information directly supporting the correct answer, we heuristically define evidential documents as positive documents that contain the answer string. Although the presence of the answer string alone does not necessarily mean that evidential documents contain information that directly supports the correct answer, this heuristic definition still demonstrates highly effective performance [ 27 ] . Then, we use data mining to extract and provide only evidential documents, instead of giving all retrieved documents to the LLM. When training with a dataset constructed using these labels, it preserves key information while mitigating positional bias. Through these training steps, ACoRN enhances robustness against various types of retrieval noise and improves its ability to summarize key information within evidential documents. To demonstrate the effectiveness of our compression strategy for retrieving and summarizing evidential documents, we analyze the performance on ODQA benchmarks, including Natural Questions (NQ) [ 14 ] , TriviaQA [ 15 ] , and PopQA [ 6 ] . We also reconstruct benchmark test datasets by applying offline data augmentation to introduce various types of noise into the documents. Each test dataset is designed to demonstrate noise robustness from different perspectives. Our method, ACoRN, has shown improved performance over other compression methods. We also show that distilling LLM summarization abilities using evidential documents from top- k retrievals better preserves the answer string compared to using all top- k documents. Furthermore, in ACoRN we examine the impact of offline data augmentation on noise robustness. Based on the experimental results, ACoRN performs exceptionally well on datasets with numerous accuracy-reducing documents, demonstrating its high applicability in real-world scenarios. The main contributions of our work can be summarized as follows: • A concise noise classification that distinguishes between irrelevant information and misleading yet query-related information, designed to enhance the robustness of abstractive compression training against retrieval noise. • We show that