Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification

Dipam Goswami, Simone Magistri, Gido M. van de Ven, Bartłomiej Twardowski, Andrew D. Bagdanov, Tinne Tuytelaars, Joost van de Weijer · 2026

Read Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification by Dipam Goswami, Simone Magistri, Gido M. van de Ven, Bartłomiej Twardowski, Andrew D. Bagdanov, Tinne Tuytelaars, Joost van de Weijer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24528.txt.

Description

Goswami and co-authors improve few-shot image classification with vision-language models such as CLIP. Mixing text and image prototypes helps, and they show this mixing acts like a variance shrinkage estimator. The catch is that the two modalities live in partly misaligned spaces. Their fix projects image prototypes onto the main directions of the text embedding space, forming a task-semantic subspace. Mixing inside that subspace improves accuracy. A combined classifier, pairing the mixed prototype with an image-specific one, beats prior methods across several benchmarks.

Catalog record

CREATORDipam Goswami, Simone Magistri, Gido M. van de Ven, Bartłomiej Twardowski, Andrew D. Bagdanov, Tinne Tuytelaars, Joost van de Weijer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51203
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24528

Browse the full library · About the Tower of Babel