Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers
Francesca Pezzuti, Sean MacAvaney, Nicola Tonellotto · 2025
Read Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers by Francesca Pezzuti, Sean MacAvaney, Nicola Tonellotto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22672.txt.
Description
Fine-tuning a cross-encoder in stages may not beat one good stage, Francesca Pezzuti, Sean MacAvaney, and Nicola Tonellotto report in a 2025 paper. Cross-encoders are strong passage re-rankers for search. Standard fine-tuning needs large amounts of hand-labeled data, a contrastive objective, and heuristically sampled negatives, all of which cost time to build for each new task.
A newer approach teaches the model to mimic the rankings of a strong large language model, using a distillation objective. The strategies can be used alone or in sequence. The authors test point-wise cross-encoders fine-tuned in one stage against those tuned in two stages. Contrastive fine-tuning alone proved on par with multi-stage approaches. Code is released for reproduction, so others can check the claim.
Catalog record
| CREATOR | Francesca Pezzuti, Sean MacAvaney, Nicola Tonellotto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22672