Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging
Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek · 2026
Read Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging by Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04731.txt.
Description
Sameh Khattab and 7 co-authors merge specialized expert models into one biomedical search system in a 2026 paper. Adapting a general language model into a domain retriever usually needs large-scale training on mixed data, which can average away each domain's strengths.
Merging independently trained experts beat that approach across several model families, merging methods, and retrieval tasks. To raise expert quality further, the authors propose a synthesize-train-merge framework. A top-tier model generates hard negative examples, meaning near-miss documents that sharpen training. Specialists are fine-tuned cheaply and then merged, with no continual pretraining. Synthesized hard negatives helped smaller models most. The framework performed strongly on biomedical tasks while staying competitive on general retrieval.
Catalog record
| CREATOR | Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04731