Reproducing and Comparing Distillation Techniques for Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Information retrieval CCS: Computing methodologies Artificial intelligence

Victor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski · 2026

Read Reproducing and Comparing Distillation Techniques for Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Information retrieval CCS: Computing methodologies Artificial intelligence by Victor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03010.txt.

Description

Victor Morand, Mathias Vast, Basile Van Cooten, and colleagues rerun a fair race between recipes for training search rankers. Cross-encoders, which read a query and document together, are the accuracy backbone of modern search. Recent work claimed distillation from large models could lift them to top-tier quality, but training recipes were rarely compared under one roof.

The team reproduced a recent large-model distillation recipe and raced it against teacher-ensemble distillation and other supervised losses. They trained many encoder generations, from the original BERT to the recent ModernBERT, and tested in-domain and far afield. Losses based on relative comparisons won consistently across backbones and test sets. The right objective, they find, can gain as much as a bigger backbone.

Catalog record

CREATORVictor Morand, Mathias Vast, Basile Van Cooten, Laure Soulier, Josiane Mothe, Benjamin Piwowarski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48868
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03010

Browse the full library · About the Tower of Babel