Reproducing and Comparing Distillation Techniques for Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Information retrieval CCS: Computing methodologies Artificial intelligence
Victor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski · 2026
Read Reproducing and Comparing Distillation Techniques for Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Information retrieval CCS: Computing methodologies Artificial intelligence by Victor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03010.txt.
Description
Victor Morand, Mathias Vast, Basile Van Cooten, and colleagues rerun a fair race between recipes for training search rankers. Cross-encoders, which read a query and document together, are the accuracy backbone of modern search. Recent work claimed distillation from large models could lift them to top-tier quality, but training recipes were rarely compared under one roof.
The team reproduced a recent large-model distillation recipe and raced it against teacher-ensemble distillation and other supervised losses. They trained many encoder generations, from the original BERT to the recent ModernBERT, and tested in-domain and far afield. Losses based on relative comparisons won consistently across backbones and test sets. The right objective, they find, can gain as much as a bigger backbone.
Catalog record
| CREATOR | Victor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48868 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03010