Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging

Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek · 2026

Read Less Finetuning, Better Retrieval: Rethinking LLM Adaptation for Biomedical Retrievers via Synthetic Data and Model Merging by Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04731.txt.

Description

Sameh Khattab and 7 co-authors merge specialized expert models into one biomedical search system in a 2026 paper. Adapting a general language model into a domain retriever usually needs large-scale training on mixed data, which can average away each domain's strengths.

Merging independently trained experts beat that approach across several model families, merging methods, and retrieval tasks. To raise expert quality further, the authors propose a synthesize-train-merge framework. A top-tier model generates hard negative examples, meaning near-miss documents that sharpen training. Specialists are fine-tuned cheaply and then merged, with no continual pretraining. Synthesized hard negatives helped smaller models most. The framework performed strongly on biomedical tasks while staying competitive on general retrieval.

Catalog record

CREATORSameh Khattab, Jean-Philippe Corbeil, Osman Alperen Koraş, Amin Dada, Julian Friedrich, François Beaulieu, Paul Vozila, Jens Kleesiek
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04731

Browse the full library · About the Tower of Babel