Investigating Task Arithmetic for Zero-Shot Information Retrieval Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Dev
Marco Braga, Pranav Kasela, Alessandro Raganato, Gabriella Pasi · 2025
Read Investigating Task Arithmetic for Zero-Shot Information Retrieval Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Dev by Marco Braga, Pranav Kasela, Alessandro Raganato, Gabriella Pasi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00649.txt.
Introduction
Marco Braga, Pranav Kasela, Alessandro Raganato, and Gabriella Pasi adapt search rankers to new domains by adding and subtracting model weights. Language models re-rank documents well until vocabulary and style shift under them. Task arithmetic merges models trained on different tasks through simple weight operations. No extra fine-tuning is required.
The team synthesizes diverse task and domain knowledge into a single retriever this way. Experiments span scientific, biomedical, and multilingual collections. Re-ranking quality rose by up to eighteen percent on one standard measure and fifteen on another. Both gains came at zero-shot, on tasks the merged model never trained for. The analysis also maps where the trick strains. Code is public. For search teams facing a new domain with no labeled data, weight arithmetic offers a cheap first adaptation.
Catalog record
| CREATOR | Marco Braga, Pranav Kasela, Alessandro Raganato, Gabriella Pasi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating Task Arithmetic for Zero-Shot Information Retrieval Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730216 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Language models Marco Braga, Pranav Kasela, Alessandro Raganato, Gabriella Pasi Abstract . Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR . † † cc-license: by 1. Introduction Large Language Models (LLMs) have achieved state-of-the-art performance in a wide range of tasks in several research fields ( Min et al. 2023 ; Rogers and Luccioni 2024 ) , including Information Retrieval (IR) ( Zhu et al. 2023 ) . By learning representations from massive unlabeled corpora, LLMs can be employed in document re-ranking ( Nogueira et al. 2020 ; Qin et al. 2024 ) , query expansion ( Chen et al. 2024 ; Li et al. 2024 ; Rizzo et al. 2024 ) , and synthetic data generation ( Almeida and Matos 2024 ; Braga et al. 2024a ) . Notably, these models often excel in zero-shot scenarios ( Brown et al. 2020 ; Agrawal et al. 2022 ; Wang et al. 2024 ) , effectively handling unseen tasks or domains without additional supervised fine-tuning. This zero-shot capability has driven their widespread use in document re-ranking ( Zhuang et al. 2024 ; Borges et al. 2024 ; Lin et al. 2022 ) , enabling robust retrieval even in the absence of domain-specific training data. Despite these advantages, domain mismatch remains a critical challenge that can significantly hamper the effectiveness of the model ( Wang et al. 2023 ) . The BEIR benchmark ( Thakur et al. 2021 ; Kamalloo et al. 2024 ) , which spans diverse tasks and domains, provides a heterogeneous framework for evaluating zero-shot IR performance. A common strategy is to pre-train a model on a large-scale IR dataset (e.g., MS-MARCO ( Nguyen et al. 2016 ) ) and then apply it zero-shot to unseen domains ( Kamalloo et al. 2024 ) . Although this approach often achieves strong performance, developing a single LLM that robustly generalizes to every domain remains very challenging ( Thakur et al. 2021 ; Kasela et al. 2024b ) , as domain-specific adaptation typically requires large labeled datasets and considerable computational resources ( Xia et al. 2024 ) . To mitigate these costs, Parameter-Efficient Fine-Tuning (PEFT) approaches have been proposed ( Houlsby et al. 2019 ; Hu et al. 2021 ; Braga et al. 2024b ) , which adapt a small subset of parameters while leaving most of the model frozen. Although PEFT significantly reduces the scale of updates and can be effective under limited data, labeled instances are still needed for each target domain, which makes it less suitable for frequent domain shifts or truly zero-shot scenarios. Meanwhile, the proliferation of publicly available LLMs, fine-tuned for diverse tasks, domains, and languages, on open-source platforms such as HuggingFace ( Jain 2022 ) , offers a new opportunity to reuse existing models rather than training new ones from scratch. In this context, Task Arithmetic ( Ilharco et al. 2022 ) emerges as a promising method. In fact, by merging the parameters of two or more fine-tuned LLMs through a simple addition or subtraction, Task Arithmetic transfers domain knowledge into a target model without further gradient-based optimization. More specifically, a Task Vector is defined as the difference between the parameters of a domain-fine-tuned model and its original pre-trained version, allowing domain-specific information to be injected or removed in parameter space with minimal overhead. Figure 1. Proposed approach: Given a pre-trained LLM \Theta_{0} and its domain-finetuned version \Theta_{D} , we compute the Task Vector \tau_{D} as their parameter difference. To build a domain-specific IR model \Theta^{\prime} , we add \tau_{D} to an IR-finetuned model \Theta_{T} . Our proposed approach: given a pre-trained Large Language Model $\Theta_0$ and its finetuned version on a specific domain $\Theta_D$, we obtain the Task Vector $\tau_D$ as the difference between the parameters of these models. To define a domain-specific IR model $\Theta'$, we add $\tau_D$ to a model $\Theta_T$ already fine-tuned for the IR task. In this work, we investigate how Task Arithmetic can be applied to Information Retrieval, emphasizing domain and language transfer under zero-shot conditions. As shown in Figure 1 , we begin with a pre-trained LLM and leverage domain-specialized vectors derived from fine-tuned models, combining them with an IR-focused baseline to produce a new domain-aware model. Our extensive evaluation covers eight publicly available datasets spanning scientific, biomedical, and multilingual tasks. We experiment with six LLMs with parameter counts ranging from 66 million to 7 billion, including encoder-only, encoder-decoder, and decoder-only architectures. Inasmuch as all specialized models are publicly available, our method remains reproducible, computationally efficient, and it does not require additional training thus minimizing carbon footprint. Our results show that Task Arithmetic consistently improves upon strong IR baselines, with gains of up to 18% in NDCG@10 and 15% in P@10, highlighting the practical value of reusing existing domain-trained models and underlying how Task Arithmetic is a lightweight but powerful strategy for zero-shot domain and language adaptation in IR. 2. Methodology In this Section, we first present related work on model merging and introduce Task Arithmetic (Section 2.1 ). We then explain how we adapt this framework for zero-shot IR (Section 2.2 ). 2.1. Related Work and Motivation Adapting LLMs to specialized domains typically requires resource-intensive fine-tuning. To address frequent domain shifts or label-scarce settings, recent studies explore weight interpolation and model merging ( Wortsman et al. 2022 ; Matena and Raffel 2022 ) , leveraging the observation that models fine-tuned on related tasks may reside in compatible regions of parameter space ( Ainsworth et al. 2023 ; Choshen et al. 2022 ) . This allows arithmetic operations, such as averaging parameters, to preserve or even boost performance ( Li et al. 2022 ; Frankle et al. 2020 ; Izmailov et al. 2018 ) . Among these methods, Task Arithmetic ( Ilharco et al. 2022 ) is particularly appealing for the fact that it requires no training. Each domain- or task-specific fine-tuning run is represented as a Task Vector, defined by the difference between the fine-tuned model’s