Learning Task-Agnostic Representations through Multi-Teacher Distillation
Philippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida · 2025
Read Learning Task-Agnostic Representations through Multi-Teacher Distillation by Philippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18680.txt.
Introduction
Philippe Formont and his seven co-authors propose a way to merge many specialized embedding models into one representation that serves any task. Embedding models turn complex inputs into lists of numbers, and different models capture different aspects depending on their design and training data. Combining several teachers into one student is called multi-teacher distillation, but existing versions are usually tailored to a single target task.
Their framework rests on a majority vote objective, which they show is bounded by the information shared between student and teacher embeddings. That connection yields a training loss needing no task labels or prior knowledge. Evaluations across text, vision, and molecular modeling showed the merged representations improved a wide range of later tasks, from classification to regression. The team also released new embedding models trained this way that set strong marks in several modalities.
Catalog record
| CREATOR | Philippe Formont, Maxime Darrin, Banafsheh Karimian, Jackie CK Cheung, Eric Granger, Ismail Ben Ayed, Mohammadhadi Shateri, Pablo Piantanida |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 130 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18680