Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting

Jüri Keller, Maik Fröbe, Gijs Hendriksen, Daria Alexander, Martin Potthast, Philipp Schaer · 2025

Read Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting by Jüri Keller, Maik Fröbe, Gijs Hendriksen, Daria Alexander, Martin Potthast, Philipp Schaer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17440.txt.

Introduction

Jüri Keller and five co-authors ran a small case study on testing search systems over time. Their 2025 arXiv paper notes that classic retrieval tests freeze queries and documents in place, so they miss how needs and texts evolve. Longitudinal tests fix that gap, but they demand custom code, which hurts reproducibility and reuse.

Working from their entries to the LongEval 2024 shared task, the team proposes a declarative extension of the ir_datasets toolkit. In the new style, researchers state which queries, documents, and feedback exist at each point in time instead of coding the logic by hand. They rebuilt their LongEval entries on the extension and found the code grew much simpler. The paper is a software lesson as much as a retrieval result: plain declarations beat tangled scripts for time-aware search tests.

Catalog record

CREATORJüri Keller, Maik Fröbe, Gijs Hendriksen, Daria Alexander, Martin Potthast, Philipp Schaer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting Jüri Keller, Maik Fröbe, Gijs Hendriksen, Daria Alexander, Martin Potthast, Philipp Schaer Abstract The longitudinal evaluation of retrieval systems aims to capture how information needs and documents evolve over time. However, classical Cranfield-style retrieval evaluations only consist of a static set of queries and documents and thereby miss time as an evaluation dimension. Therefore, longitudinal evaluations need to complement retrieval toolkits with custom logic. This custom logic increases the complexity of research software, which might reduce the reproducibility and extensibility of experiments. Based on our submissions to the 2024 edition of LongEval, we propose a custom extension of ir_datasets for longitudinal retrieval experiments. This extension allows for declaratively, instead of imperatively, describing important aspects of longitudinal retrieval experiments, e.g., which queries, documents, and/or relevance feedback are available at which point in time. We reimplement our submissions to LongEval 2024 against our new ir_datasets extension, and find that the declarative access can reduce the complexity of the code. https://github.com/clef-longeval/ir-datasets-longeval 1 Introduction Information Retrieval (IR) is centered on the task of finding relevant information to meet users’ information needs. Therefore, systems must be able to cope with the ongoing flood of information. Current IR test collections abstract this complex task to enable a well-proven evaluation framework. This abstraction excludes certain aspects, such as temporal dynamics, that are inherent to real-world retrieval environments and tasks. The LongEval lab reintroduces some of these dynamics into offline evaluations with dynamic test collections [ 3 , 4 , 8 ] . That is, a changing document corpus, evolving information needs, and updated relevance judgments are considered. This makes the evaluations more realistic in these regards and allows new directions for retrieval approaches. For example, if previous relevance judgments are available, based on user interactions, as in the LongEval test collections, they can be used as a relevance signal. Our previous works submitted to the lab have shown that such signals can strongly improve retrieval effectiveness at low cost [ 2 , 17 , 18 ] . While IR systems are exposed to changing data, users expect consistent, good effectiveness. To provide and maintain this, it is essential to regularly assess the system. Such longitudinal evaluations substantially increase the complexity compared to conventional Cranfield-style offline evaluations. With each new snapshot of a search setting, new versions of documents, queries, and qrels are introduced that need to be stored and maintained. Additionally, the retrieval approach and adequate baselines must take care to process only past data, while it would be helpful if they can easily process new and modified versions of the data. The number of necessary experiments increases with each variation of the parameters and snapshots. That means that the complexity, the demand for resources, and the propensity to errors of the experiments drastically increase. These challenges of longitudinal evaluations make software submissions, as enabled by TIRA [ 12 ] , difficult, as the submitted software becomes more complex and, as a result, the maintenance effort for organizers would increase (e.g., because every submission would come with different custom code to load the data). Still, in longitudinal settings, software submissions are especially interesting because they allow the application of the exact same approach to different snapshots. To simplify longitudinal retrieval experiments and thereby facilitate software submissions in longitudinal scenarios, a standardized interface to dynamic test collections is necessary. We extend the ir_datasets framework [ 19 ] with methods tailored to dynamic test collections. We re-implement some of the proposed methods from last year’s iteration of the LongEval lab and compare the results in terms of code complexity and retrieval effectiveness. Our ir_datasets extension provides a valuable benefit for software submissions, as the submitted software can focus on the main ideas of their approach and does not have to do dedicated data wrangling, thereby reducing the maintenance efforts of organizers. Even bugs in the ir_datasets extension can be handled without modifying the code of participants, e.g., by re-installing the extension in the submitted software. We hope that our ir_datasets extension also simplifies software submissions for longitudinal experiments in the future, as submitted software is less complex. In summary, our core contributions are: • Re-implementation of our original approaches submitted to LongEval 2024 [ 18 , 2 , 17 ] , as PyTerrier transformers [ 20 ] . • An extension of ir_datasets for longitudinal evaluations. • We added the LongEval Sci and LongEval Web datasets to the extension as the first dynamic test collections. • A preliminary analysis of the re-implemented systems in terms of code complexity and retrieval effectiveness. The ir_datasets extension 1 1 1 https://github.com/clef-longeval/ir-datasets-longeval and our re-implemented approaches from LongEval 2024 2 2 2 https://github.com/clef-longeval/longeval-code/tree/main/clef25 are publicly available on GitHub. 2 Related Work The observation that real-world search engines must operate in an evolving environment motivates longitudinal evaluations. Compared to traditional Cranfield-style evaluations, which use a static set of documents and relevance judgments, longitudinal retrieval evaluations must capture varying versions of relevance judgments and documents, thereby increasing the complexity of the experiments. Streamlining access to dynamic test beds should support researchers in conducting longitudinal evaluations. In this context, we discuss the related work. Evolving Search Settings. While in traditional Cranfield-style evaluations most dynamics are abstracted, in real-world settings each component may change over time and thereby influence the retrieval results [ 16 ] . The document corpus or the content of the documents change [ 1 , 22 ] . Simultaneously, information needs evolve [ 9 , 28 , 22 ] , ultimately affecting what users perceive as relevant [ 5 ] . Longitudinal Evaluations. Assessing how well retrieval systems work for users at different points in time requires repeated experiments. This significantly increases the complexity of IR experiments, as every point in time might have different restrictions on what information can be accessed. This evolving setting is investigated by the LongEval CLEF lab [ 3 , 4 ] that provides two dynamic test collections with up to 15 snapshots [ 8 , 13 ] . Beyond that, not many dynamic evolving test collections are available, so that studies often simulate dynamics or rely on other versioned datasets such as TREC-COVID, with five snapshots [ 29 ] . In many longitudinal experiments, each snapshot requires unique runs. This increases the complexity and computational resources of experiments. For instance, Keller et al. tested five different retrieval approaches on TREC-COVID, LongEval 2023, and a simulated dynamic test collection based on TripClick [ 23 , 16 ] . In total, 45 retrieval runs were created. Depending on the simulation strategy, many more snapshots can be created, resulting in even more retrieval runs [ 10 ] . Accessing Datasets. Thakur et al. propose the BEIR benchmark to investigate the out-of-distribution generalization of IR systems [ 27 ] . They enable straight forward evaluations over 18 datasets from various domains. MacAveney et al. introduce ir_datasets , a Python toolkit that provides a standardized interface to datasets typically used in IR research [ 19 ] , which got also integrated into TIRA [ 12 , 11 ] to promote more reproducible shared tasks. The community can contribute to ir_datasets to extend the interface which is essentially what we

Read the complete text