Learning Task-Agnostic Representations through Multi-Teacher Distillation

Philippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida · 2025

Read Learning Task-Agnostic Representations through Multi-Teacher Distillation by Philippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18680.txt.

Introduction

Philippe Formont and his seven co-authors propose a way to merge many specialized embedding models into one representation that serves any task. Embedding models turn complex inputs into lists of numbers, and different models capture different aspects depending on their design and training data. Combining several teachers into one student is called multi-teacher distillation, but existing versions are usually tailored to a single target task.

Their framework rests on a majority vote objective, which they show is bounded by the information shared between student and teacher embeddings. That connection yields a training loss needing no task labels or prior knowledge. Evaluations across text, vision, and molecular modeling showed the merged representations improved a wide range of later tasks, from classification to regression. The team also released new embedding models trained this way that set strong marks in several modalities.

Catalog record

CREATORPhilippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE130 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18680

Browse the full library · About the Tower of Babel