RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

Hanjun Cho, Jay-Yoon Lee · 2026

Read RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora by Hanjun Cho, Jay-Yoon Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19047.txt.

Introduction

Cho and Lee present RARE, a retrieval benchmark framework for highly similar corpora like finance, legal, and patent documents. Standard benchmarks assume distinct documents, but real systems face redundant texts with overlapping evidence. This mismatch undervalues retrievers returning sufficient evidence and overrates benchmark stars. RARE splits documents into atomic facts to track redundancy, scoring criteria separately and fusing decisions by rank. On finance, legal, and patent corpora, it yields RedQA, where a retriever drops from 66.4 percent PerfRecall at 10 to between 5 and 28 percent at 4-hop depth.

Catalog record

CREATORHanjun Cho, Jay-Yoon Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92923
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora Hanjun Cho, Jay-Yoon Lee Abstract Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity. This mismatch undermines evaluation validity: retrievers can be unfairly undervalued even when they retrieve documents that provide sufficient evidence, because redundancy across documents is not accounted for in evaluation. On the other hand, retrievers that perform well on standard benchmarks often generalize poorly to real-world corpora with highly similar and redundant documents. We present RARE (Redundancy-Aware Retrieval Evaluation), a framework for constructing realistic benchmarks by (i) decomposing documents into atomic facts to enable precise redundancy tracking and (ii) enhancing LLM-based data generation with CRRF . RAG benchmark data usually requires multiple quality criteria, but LLMs often yield trivial outputs. CRRF scores criteria separately and fuses decisions by rank, improving the reliability of generated data. Applying RARE to Finance, Legal, and Patent corpora, we introduce RedQA , where a strong retriever baseline drops from 66.4% PerfRecall@10 on 4-hop General-Wiki to 5.0–27.9% PerfRecall@10 at 4-hop depth, revealing robustness gaps that current benchmarks fail to capture. RARE enables practitioners to build domain-specific RAG evaluations that faithfully reflect real-world deployment conditions. 1 Introduction Large language models (LLMs) require external knowledge retrieval to remain accurate and current ( 14 ; 9 ) . In enterprise settings, RAG operates on specialized corpora—financial disclosures, legal statutes, patent filings—that exhibit a critical property: high information redundancy and inter-document similarity. Figure 1: Similarity (%) and Redundancy (%) across datasets. Similarity : average pairwise cosine similarity of passage embeddings; Redundancy: fraction of facts whose content also appears in a different passage. Standard benchmarks exhibit low values, whereas enterprise corpora show substantially higher levels. Standard benchmarks do not reflect specialized-domain conditions. Figure 1 quantifies this gap: enterprise corpora (orange) exhibit substantially greater within-corpus overlap than standard benchmarks (blue) such as Natural Questions ( 13 ) , HotpotQA ( 25 ) , and FiQA ( 16 ) , with higher similarity and redundancy. Here, similarity is the average cosine similarity between passage embeddings, and redundancy measures the fraction of facts whose content also appears in a different passage in the corpus ( 20 ) ; see Appendix A.3 for metric details. This mismatch results in two evaluation failures. First, in high-redundancy settings, retrievers that surface valid alternative evidence are penalized by single-canonical-passage labeling ( 28 ) . Second, evaluations on low-overlap benchmarks ( i.e. , those with low inter-document similarity and low fact-level redundancy) can overestimate real-world performance on high-overlap enterprise corpora, thereby obscuring robustness gaps. Constructing benchmarks that reflect these conditions requires two capabilities. First, systematic redundancy tracking : the evaluation should recognize semantically equivalent evidence that appears in multiple passages ( 7 ) . This is distinct from training data construction, which typically requires only local control over positive/negative pairs ( 12 ) ; without redundancy-aware labeling, retrievers that surface valid alternatives are unfairly penalized ( 28 ) . Second, high-quality data generation : LLMs frequently produce trivial, unnatural, or contradictory questions ( 6 ) (Appendix F ), which undermines benchmark validity. We introduce RARE (Redundancy-Aware Retrieval Evaluation), a modular framework that addresses both challenges. RARE decomposes documents into atomic facts, tracks redundancy via embedding similarity combined with LLM verification, and stabilizes data generation with CRRF (Criterion-wise Prompting with Reciprocal Rank Fusion), which evaluates each quality criterion independently and aggregates decisions via rank-based fusion to improve the reliability of generated benchmark data ( 4 ) . Experiments reveal severe degradation under realistic conditions. For 4-hop queries across the Finance, Legal, and Patent domains, strong retriever baseline attains only 27.9% PerfRecall@10 on Patent, with Finance and Legal dropping to 8.5% and 5.0% , respectively, whereas low-overlap General-Wiki maintains 66.4% at 4-hop. Contributions. • We propose RARE , a generalizable framework that enables practitioners to construct domain-specific RAG benchmarks on their own corpora. • We introduce CRRF , a simple multi-criteria quality-control recipe for LLM-based data generation that improves the reliability of the resulting benchmark instances. • We quantify the gap between existing benchmarks and enterprise corpora via redundancy and similarity analysis, and introduce RedQA , a benchmark capturing these conditions across Finance, Legal, and Patent. 2 Related Work Open-Domain QA Benchmarks. Existing benchmarks such as HotpotQA ( 25 ) and MS MARCO ( 1 ) have played a central role in the development of open-domain question answering and retrieval models. A common design choice across these datasets is the use of clearly distinct documents with predominantly one-to-one answer–passage associations. While this setting enables clean evaluation, it reflects an idealized scenario that differs from real-world, domain-specific corpora. Toward More Realistic Retrieval Benchmarks. Recent work has sought to increase the realism of retrieval benchmarks along multiple dimensions. MIRACL ( 26 ) extends retrieval evaluation to a multilingual setting, while other benchmarks such as KILT ( 18 ) and TREC-COVID ( 21 ) emphasize domain-specific and knowledge-intensive retrieval scenarios. These efforts enhance realism with respect to language coverage and domain specificity; however, they do not explicitly model settings with high similarity and redundancy, where the same information is repeatedly instantiated across multiple documents in slightly varied forms. General Retrieval Evaluation Suites. Beyond individual QA benchmarks, several evaluation suites aim to provide standardized retrieval assessment across diverse tasks and domains. BEIR ( 20 ) and MTEB ( 17 ) consolidate a wide range of retrieval tasks under unified evaluation protocols, supporting systematic model comparison and reproducibility. Although these suites substantially broaden benchmark coverage, they continue to rely on fixed corpora and static relevance annotations, which limit their ability to reflect deployment-time retrieval dynamics. LLM-based Data Generation. LLMs have recently been explored as a means of synthetic data generation. Approaches such as Promptagator ( 5 ) , InPars ( 2 ; 10 ) , and PAQ ( 15 ) demonstrate the feasibility of automatically generating large-scale query or QA datasets. However, most existing pipelines focus on surface-level generation such as paraphrasing. Effective retrieval questions often require structured reasoning and the satisfaction of multiple quality criteria. Systematically supporting such reasoning in data generation remains an open challenge. Figure 2: Overall workflow of the RARE framework. The pipeline consists of valid information selection, redundancy tracking, and question-answer generation. Multi-criteria judgments are stabilized through CRRF, and redundancy-aware labeling ensures fair evaluation under real-world corpus conditions. During the logical filtering step in Question-Answer Generation, answerability validation additionally performs answer generation. 3 RARE Framework 3.1 Pipeline Overview RARE constructs realistic multi-hop RAG evaluation data under high redundancy and similarity. Figure 2 illustrates the pipeline, which consists of three stages: Valid Information Selection , Redundancy Tracking , and Question–Answer Generation . Each stage follows a common three-step pattern: constructing a candidate pool, filtering invalid candidates, and ranking candidates based on multiple criteria. A central component of RARE is our ranking method, CRRF (detailed in Section 3.5 ), which stabilizes LLM judgments by evaluating each quality criterion independently and fusing results through rank aggregation rather than confidence scores. By decomposing complex multi-criterion reasoning into simple binary or scalar judgments, CRRF yields more stable and higher-quality decisions. Notation. • C : input document

Read the complete text