Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers

Francesca Pezzuti, Sean MacAvaney, Nicola Tonellotto · 2025

Read Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers by Francesca Pezzuti, Sean MacAvaney, Nicola Tonellotto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22672.txt.

Description

Fine-tuning a cross-encoder in stages may not beat one good stage, Francesca Pezzuti, Sean MacAvaney, and Nicola Tonellotto report in a 2025 paper. Cross-encoders are strong passage re-rankers for search. Standard fine-tuning needs large amounts of hand-labeled data, a contrastive objective, and heuristically sampled negatives, all of which cost time to build for each new task.

A newer approach teaches the model to mimic the rankings of a strong large language model, using a distillation objective. The strategies can be used alone or in sequence. The authors test point-wise cross-encoders fine-tuned in one stage against those tuned in two stages. Contrastive fine-tuning alone proved on par with multi-stage approaches. Code is released for reproduction, so others can check the claim.

Catalog record

CREATORFrancesca Pezzuti, Sean MacAvaney, Nicola Tonellotto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22672

Browse the full library · About the Tower of Babel