Overview of the TREC 2024 NeuCLIR Track
Dawn Lawrie, † Sean MacAvaney, ‡ James Mayfield, † Paul McNamee, † Douglas W. Oard, o † {{}^{o}}^{\dagger} Luca Soldaini, ∗ Eugene Yang † · 2025
Read Overview of the TREC 2024 NeuCLIR Track by Dawn Lawrie, † Sean MacAvaney, ‡ James Mayfield, † Paul McNamee, † Douglas W. Oard, o † {{}^{o}}^{\dagger} Luca Soldaini, ∗ Eugene Yang † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14355.txt.
Introduction
Dawn Lawrie and six co-authors overview the TREC 2024 NeuCLIR track in 2025, a shared test of search across languages. Neural methods may change how well users find news and technical writing written in another language. The track builds the collections to find out.
Collections cover Chinese, Persian and Russian news, plus Chinese academic abstracts. Four task types run from cross language search to multilingual search and report generation. Five teams plus the coordinators submitted several hundred runs across eight tasks. Results and task designs are reported for each type. Shared collections matter because cross language search cannot improve on private data alone. Later teams can rerun the same tests and compare fairly.
Catalog record
| CREATOR | Dawn Lawrie, † Sean MacAvaney, ‡ James Mayfield, † Paul McNamee, † Douglas W. Oard, o † {{}^{o}}^{\dagger} Luca Soldaini, ∗ Eugene Yang † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Overview of the TREC 2024 NeuCLIR Track Dawn Lawrie, † Sean MacAvaney, ‡ James Mayfield, † Paul McNamee, † Douglas W. Oard, o † {{}^{o}}^{\dagger} Luca Soldaini, ∗ Eugene Yang † Abstract . The principal goal of the TREC Neural Cross-Language Information Retrieval (NeuCLIR) track is to study the effect of neural approaches on cross-language information access. The track has created test collections containing Chinese, Persian, and Russian news stories and Chinese academic abstracts. NeuCLIR includes four task types: Cross-Language Information Retrieval (CLIR) from news, Multilingual Information Retrieval (MLIR) from news, Report Generation from news, and CLIR from technical documents. A total of 274 runs were submitted by five participating teams (and as baselines by the track coordinators) for eight tasks across these four task types. Task descriptions and the available results are presented. 1. Introduction This is the third and final year of the TREC Neural Cross-Language Information Retrieval (NeuCLIR) track. 1 1 1 https://neuclir.github.io The first year of the track included News CLIR tasks ( Lawrie et al. 2022a ) . The second year of the task added News MLIR and Technical Documents CLIR tasks ( Lawrie et al. 2023 ) . In this third and final year of NeuCLIR those three task types continued, and we added a new Report Generation pilot task. This overview describes NeuCLIR’s each of these four task types. There are three News CLIR tasks, each of which has topics in English and news documents in one other language (Chinese, Persian or Russian). 2 2 2 The news test collections also provide topics in Chinese, Persian and Russian, which can be used to create monolingual baselines or for other purposes. CLIR is the most mature of the track’s task types, and the capabilities that CLIR provides are foundational to the other three task types. Current CLIR systems face two broad challenges that distinguish CLIR from monolingual retrieval: (1) there is less robust training data than is available for monolingual ranked retrieval tasks; and (2) there is misalignment of term representations for different languages in multilingual embeddings. The news test collections in Chinese, Persian and Russian are the same as in the TREC 2022 NeuCLIR track. In 2024, the CLIR tasks provide relevance judgments for 56 new topics in Chinese, 68 new topics in Persian, and 65 new topics in Russian. Three of the five participating teams submitted CLIR runs in 2024, and the track coordinators also created baseline runs. There is one News MLIR task, in which the topics are in English and the documents to be searched are news stories from the union of the Chinese, Russian and Persian news collections. This task requires generating a single ranked list for each topic over the unified collection. The principal additional challenge in this task is that scores computed independently for documents in different languages might not be as easy to compare as are scores for documents in a single language. This is the second year of the News MLIR task. In 2024, the news MLIR task has relevance judgments for 51 topics, each of which is also present in the 2024 topic set for two or more of the news CLIR tasks. Four of the five participating teams submitted results for the news MLIR task in 2024, and the track coordinators also created baseline runs. There is one Technical Documents CLIR task in which the topics are in English and the documents to be searched are Chinese academic abstracts. The principal additional challenge in this task is that some standard tools such as multilingual embeddings, pretrained language models such as BERT ( Devlin et al. 2019 ) , or generative large language models may be less well suited to the highly-technical language of these abstracts than to more general news text. In 2024, the Technical Documents CLIR task has relevance judgments for 71 new topics. Three of the five participating teams submitted results for the Technical Documents CLIR task, and the track coordinators also created baseline runs. Report Generation is a new task type this year. In the Report Generation task, systems receive a report request in English and are asked to respond with an English report that meets the requirements specified in the request, and in which substantive statements in the report are supported by a citation to one or more documents in one of the three news collections. There are three report generation tasks, one each for reports with citations to Chinese documents, Russian documents, and Persian documents. Evaluation used a recently-proposed evaluation framework for such reports ( Mayfield et al. 2024 ) . Four of the five participating teams submitted results for the Report Generation task. The remainder of this paper is organized as follows. We begin with a brief summary of the News CLIR and News MLIR tasks, emphasizing changes since TREC 2023. This is followed by results for those tasks. Next, we present a brief summary of the Technical Documents CLIR task and the results for that task. Additional details for the News CLIR tasks can be found in the TREC 2022 and TREC 2023 NeuCLIR track overview papers ( Lawrie et al. 2022a ; Lawrie et al. 2023 ) ; additional details on the News MLIR and Technical Documents CLIR tasks can be found in the TREC 2023 track overview paper. Following that, we provide details for our new Report Generation task. This will be the final year of the TREC NeuCLIR track. Report generation will continue to be evaluated in a new TREC 2025 RAGTIME track, so we conclude with a brief overview of our plans for RAGTIME. 2. News Retrieval Tasks In this section we describe the CLIR and MLIR tasks. 2.1. Task Definitions We have two news retrieval task types, CLIR and MLIR. The News CLIR task includes two “settings” (i.e., sub-tasks): ad hoc CLIR and (for pool enrichment) monolingual retrieval. The two settings use the same document collections, topics, and relevance assessments. Monolingual runs use topics manually translated into the language of the documents; ad hoc runs use the original English topics and rank documents from the entire collection. 2.1.1. Ad Hoc CLIR For the ad hoc CLIR setting, systems receive a document collection in Chinese, Persian, or Russian, and a set of topics written in English. For each topic, the system must return a ranked list of 1,000 documents drawn from the entire target language document collection, ordered by likelihood and degree of relevance to the topic. Runs that use a human in the loop for ad hoc retrieval (or had design decisions influenced by human review of the topics) are indicated as “manual” runs; all others are considered “automatic.” 2.1.2. Monolingual Retrieval While monolingual retrieval is not a focus of the NeuCLIR track, monolingual runs can improve assessment pools and serve as points of reference for cross-language runs. The monolingual retrieval setting is identical to the ad-hoc setting, but it uses topic files that are human translations of the English topics into a target language in a way that would be expressed by native speakers of the language. This setting is suitable for teams looking to explore monolingual ranking in languages other than English. It also has a lower barrier to