FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation
Sohyun An, Hayeon Lee, Shuibenyang Yuan, Chun-cheng Jason Chen, Cho-Jui Hsieh, Vijai Mohan, Alexander Min · 2026
Read FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation by Sohyun An, Hayeon Lee, Shuibenyang Yuan, Chun-cheng Jason Chen, Cho-Jui Hsieh, Vijai Mohan, Alexander Min (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14227.txt.
Introduction
An and six colleagues introduce FRESCO, a benchmark that tests re-rankers as facts evolve over time. Pairing recency-seeking queries with past Wikipedia revisions, it checks whether re-rankers favor factually recent evidence while staying semantically relevant. Their evaluation finds a steady failure mode: existing re-rankers lean toward older, semantically rich documents even when the facts are obsolete. Tuning instructions toward Pareto-optimal balances then lifts Evolving Knowledge scores by up to 27 percent while keeping Non-Evolving Knowledge performance competitive.
Catalog record
| CREATOR | Sohyun An, Hayeon Lee, Shuibenyang Yuan, Chun-cheng Jason Chen, Cho-Jui Hsieh, Vijai Mohan, Alexander Min |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79987 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation Sohyun An, Hayeon Lee, Shuibenyang Yuan, Chun-cheng Jason Chen, Cho-Jui Hsieh, Vijai Mohan, Alexander Min Abstract Retrieval-Augmented Generation (RAG) is a key approach to mitigating the temporal staleness of large language models (LLMs) by grounding responses in up-to-date evidence. Within the RAG pipeline, re-rankers play a pivotal role in selecting the most useful documents from retrieved candidates. However, existing benchmarks predominantly evaluate re-rankers in static settings and do not adequately assess performance under evolving information—a critical gap, as real-world systems often must choose among temporally different pieces of evidence. To address this limitation, we introduce FRESCO ( F actual R ecency and E volving S emantic CO nflict), a benchmark for evaluating re-rankers in temporally dynamic contexts. By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance. Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete. We further investigate an instruction optimization framework to mitigate this issue. By identifying Pareto-optimal instructions that balance Evolving and Non-Evolving Knowledge tasks, we obtain gains of up to 27% on Evolving Knowledge tasks while maintaining competitive performance on Non-Evolving Knowledge tasks. † † affiliation: Meta Superintelligence Labs † † affiliation: UCLA † † contribution: Work done at Meta † † contribution: Joint last author † † date: August 10, 2026 † † correspondence: Sohyun An at [email protected] † † Code: https://github.com/facebookresearch/fresco 1 Introduction Large Language Models (LLMs) have achieved strong performance across a wide range of NLP tasks ( 5 ; 34 ; 4 ; 8 ; 38 ) , yet their practical reliability is often constrained by temporal staleness —the mismatch between static pretraining corpora and a rapidly changing world. Retrieval-Augmented Generation (RAG) has therefore become a standard remedy, grounding LLM outputs in external and up-to-date evidence ( 19 ; 15 ; 11 ; 13 ; 1 ) . Within the RAG pipeline, re-rankers act as a key gatekeeper: given a pool of retrieved candidates, they must prioritize the most useful evidence so that downstream generation remains factually correct ( 22 ; 20 ; 35 ; 33 ; 30 ; 24 ; 25 ) . Despite this central role, the prevailing evaluation paradigm for re-ranking largely assumes a non-evolving information environment. Widely used benchmarks ( 2 ; 9 ; 17 ; 31 ) typically operationalize relevance through semantic overlap or topical alignment, implicitly treating the target information as fixed over time . This assumption is frequently violated in real-world RAG deployments: when facts change, retrieval systems can surface multiple snapshots of ostensibly relevant evidence, some outdated and others reflecting the current state. For example, a query asking for the latest stable version of vLLM may retrieve two release notes or changelogs—one reporting version 0.5.0 from several weeks ago and another reporting version 0.6.0 from two days ago. Both documents can appear highly relevant under conventional semantic matching, yet only the latter is chronologically valid for answering the query. We refer to this phenomenon as Evolving Semantic Conflict : the re-ranker must choose among multiple semantically relevant candidates whose content conflicts due to temporal evolution. Handling temporal discrimination—incorporating factual recency in addition to semantic relevance—is a practical requirement for robust RAG, particularly when multiple candidates are similarly relevant to the query. Without this capability, a re-ranker may systematically promote semantically informative but obsolete documents, causing the generator to produce confident yet incorrect answers. However, current evaluation regimes provide limited visibility into this failure mode, leaving it unclear whether existing re-rankers can reliably resolve conflicts induced by evolving knowledge without sacrificing relevance. To bridge this gap, we introduce FRESCO ( F actual R ecency and E volving S emantic CO nflict), a benchmark for evaluating re-rankers in settings where semantic relevance is necessary but not sufficient . FRESCO pairs recency-seeking queries with historical Wikipedia revisions to construct candidate sets in which passages remain topically aligned yet differ in factual recency. This design isolates the key challenge faced by real-world RAG systems: selecting evidence that is both relevant and chronologically valid when semantically similar candidates disagree. Our evaluation identifies a consistent failure mode across existing re-rankers: they exhibit a strong semantic bias , frequently preferring older, contextually dense documents over newer ones even when the older evidence is factually obsolete. We further study a Pareto-based instruction optimization framework for LLM-based re-rankers, which take instructions alongside the query and candidate documents. The framework explicitly captures the trade-off between Evolving Knowledge tasks (where recency is essential) and Non-Evolving Knowledge tasks (where semantic relevance is typically sufficient), yielding a spectrum of Pareto-optimal instructions that practitioners can select based on deployment needs. Empirically, a Pareto-optimal instruction that emphasizes Evolving Knowledge improves performance on such tasks by up to 27% while maintaining competitive results on Non-Evolving Knowledge tasks. In summary, our contributions are as follows: • We formalize Evolving Semantic Conflict and introduce FRESCO , a benchmark that systematically evaluates re-rankers under temporally evolving information. • We identify a consistent failure mode in existing re-rankers, showing a strong bias toward semantically rich but factually obsolete documents. • We investigate a Pareto-based instruction optimization framework for LLM-based re-rankers, enabling controllable trade-offs between Evolving Knowledge and Non-Evolving Knowledge tasks. 2 Related Work Re-ranking and Its Evaluation. Two-stage retrieval pipelines—retrieve followed by re-rank—are a standard approach in information retrieval and are widely used to support downstream applications such as RAG ( 11 ) . Re-rankers have progressed from BERT-style cross-encoders ( 22 ) to more recent LLM-based re-rankers ( 24 ; 37 ) , with performance commonly reported on benchmarks such as MS MARCO ( 2 ) , TREC DL ( 9 ) , Natural Questions ( 17 ) , and BEIR ( 31 ) . However, these benchmarks largely encode a temporally static notion of relevance: once labels are created, they are treated as invariant. As a result, they provide limited insight into whether a re-ranker can select chronologically valid evidence when multiple semantically relevant candidates reflect different points in time. QA Benchmarks under Evolving Knowledge. Evolving, time-sensitive knowledge has motivated benchmarks such as RealtimeQA ( 16 ) and FreshQA ( 32 ) . These resources are valuable for measuring end-to-end QA performance under recency demands, but they are not designed to isolate the re-ranking decision itself. In particular, they typically evaluate final answer accuracy and do not provide controlled candidate pools containing competing, semantically relevant evidence with explicit temporal conflicts. This makes it difficult to perform a granular analysis of a re-ranker’s ability to resolve Evolving Semantic Conflict . In contrast, FRESCO leverages Wikipedia revision histories to construct timestamp-conditioned candidate pools, enabling scalable benchmarking with fine-grained control over temporal conflicts while keeping semantic relevance comparable across candidates. Figure 1 : Illustration of the construction pipeline of FRESCO . Time-annotated facts from Wikidata (left) are aligned with Wikipedia revisions (middle) to create positive and negative (outdated or irrelevant) passages (right) for explicit and implicit recency-seeking queries (top). Incorporating Temporal Signals in Ranking. A growing line of work incorporates temporal signals into ranking and retrieval by combining semantic relevance with recency-aware scoring, explicit time features, or