AMoE: Agglomerative Mixture-of-Experts Vision Foundation Model

Sofian Chaybouti Sanath Narayan Yasser Dahou Phúc H. Lê Khac, Ankit Singh Ngoc Dung Huynh Wamiq Reyaz Para, Hilde Kuehne Hakim Hacid · 2025

Read AMoE: Agglomerative Mixture-of-Experts Vision Foundation Model by Sofian Chaybouti Sanath Narayan Yasser Dahou Phúc H. Lê Khac, Ankit Singh Ngoc Dung Huynh Wamiq Reyaz Para, Hilde Kuehne Hakim Hacid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20157.txt.

Description

Sofian Chaybouti leads a nine-author team distilling two vision giants into one student family. Multi-teacher distillation promises unified visual representations. Yet its learning dynamics and data efficiency were unmapped. SigLino distills SigLIP2 and DINOv3 at once, into dense and mixture-of-experts students. The paper's payload is four findings on what makes such training cheap and strong.

A new loss keeps each teacher's geometry while it transfers knowledge. Token-balanced batching steadies learning across image resolutions. Clustering the training data, a trick from self-supervised work, beats random sampling on efficiency. The learned features transfer well into grounding vision-language models. The team releases a corpus of two hundred million images, OpenLVD200M. Five distilled checkpoints come with it.

Catalog record

CREATORSofian Chaybouti Sanath Narayan Yasser Dahou Phúc H. Lê Khac, Ankit Singh Ngoc Dung Huynh Wamiq Reyaz Para, Hilde Kuehne Hakim Hacid
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20157

Browse the full library · About the Tower of Babel