Distillation versus Contrastive Learning: How to Train Your Rerankers

Zhichao Xu Zhiqi Huang Shengyao Zhuang Vivek Srikumar · 2025

Read Distillation versus Contrastive Learning: How to Train Your Rerankers by Zhichao Xu Zhiqi Huang Shengyao Zhuang Vivek Srikumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08336.txt.

Description

Zhichao Xu, Zhiqi Huang, Shengyao Zhuang, and Vivek Srikumar compare two ways to train text rerankers, in a research paper from 2025. Rerankers sort search results and matter for retrieval quality. One way learns from true labels, and the other learns from a larger teacher model, but a fair head-to-head test under real conditions was missing.

The team trained rerankers of four sizes and two designs both ways, on the same data. Learning from a stronger teacher won on in-domain and new-domain tests in most cases. A teacher of equal strength gave no such gain. The guidance is plain: use a larger teacher if one is on hand.

Catalog record

CREATORZhichao Xu Zhiqi Huang Shengyao Zhuang Vivek Srikumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08336

Browse the full library · About the Tower of Babel