Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling

David Méndez, Roberto Confalonieri, Natalia Díaz-Rodríguez · 2026

Read Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling by David Méndez, Roberto Confalonieri, Natalia Díaz-Rodríguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22484.txt.

Introduction

Méndez and team study whether vision model class heads can serve as semantic prototypes. Vision-language models map images and text to a shared space, but need costly training on paired datasets. Post-hoc methods link pretrained encoders through light mappings, yet still need much paired data. The team repurposes class heads, discarded after pretraining, as semantic anchors. This unlocks zero-shot alignment and data augmentation mixing prototypes with real pairs. With leading post-hoc methods, the approach boosts accuracy in retrieval and zero- and few-shot classification.

Catalog record

CREATORDavid Méndez, Roberto Confalonieri, Natalia Díaz-Rodríguez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86819
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling David Méndez, Roberto Confalonieri, Natalia Díaz-Rodríguez Abstract Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired datasets. Current post-hoc alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained vision models as semantic prototypes . The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc alignment techniques consistently boosts accuracy in cross-modal retrieval, zero- and few-shot classification tasks. \icml@noticeprintedtrue † † footnotetext: \forloop @affilnum1¡ 0 AUTHORERR: Missing \icmlaffiliation. . AUTHORERR: Missing \icmlcorrespondingauthor. \Notice@String 1 Introduction Vision-language models have demonstrated powerful capabilities—such as zero-shot classification, cross-modal retrieval, image captioning, and visual question answering—by embedding images and text into a shared representation space. Classic approaches ( 32 ; 11 ) achieve this by jointly training image and text encoders on massive paired datasets. However, collecting and curating billions of image-caption pairs and then optimizing two large models end-to-end incurs prohibitive data and computational costs for most researchers and institutions, effectively restricting the training of such models to those with considerable resources. Recent decoupled techniques ( 43 ; 3 ) freeze an encoder that is pretrained on one modality and then rely on contrastive training to align the other modality’s encoder to this fixed representation, still requiring substantial paired data and computation. Post-hoc alignment methods ( 27 ; 15 ; 23 ) learn lightweight functions that map representations from independently pretrained encoders to a common latent space, eliminating large-scale contrastive training but still relying on extensive paired image-text data to achieve vision-language alignment. We study whether semantic information already present in supervised vision models can be reused for vision-language alignment. We demonstrate that the classification head weights of vision models pretrained on ImageNet-21K ( 33 ) , which are typically discarded after pretraining, function as high-quality semantic prototypes . By repurposing classification head weights as semantic prototypes to align an image feature extractor to a text encoder, we enable vision-language capabilities in two regimes: alignment without any image-text paired data, and a data augmentation strategy where these prototypes are combined with real image-text pairs thanks to their intrinsic compatibility. In this post-hoc alignment setting, pretrained encoders are treated as fixed and the focus is on reducing the additional paired data needed to connect them. Our results show promising performance in cross-modal retrieval and classification tasks. In brief, we highlight our key contributions here: • Classification heads as semantic prototypes. We provide empirical validation and theoretical justification for treating classification head weights as meaningful semantic prototypes of the classes they are associated with. We show that the weights for a class align with text representations even better than the average of images belonging to that class. • Vision-language alignment without image-text pairs. Leveraging the previous insight, we introduce a weight recycling framework to align independently trained image and text encoders in a post-hoc manner without any image-text pairs. • Compatibility with real image-text pairs. We demonstrate that classification head weights are highly compatible with real image-text representations. Beyond enabling alignment in the absence of paired data, they serve as a robust, complementary data augmentation source that consistently boosts the performance of state-of-the-art post-hoc methods when combined with available image-text pairs. • Analysis of weight representations. We provide geometric and domain-specific analyses of classification head weights, including their relationship to downstream tasks and their separation from image representations. This work offers a new perspective on the emergence of semantics in deep vision models pretrained at scale with supervision, paving the way for alignment strategies that recycle supervised pretraining weights. 2 Related Work Vision-Language Models trained from scratch. Pioneering vision-language models like CLIP ( 32 ) or ALIGN ( 11 ) jointly train vision and text encoders with a contrastive loss, so that an image and a text are mapped to similar representations if and only if they share the same semantic content. The resulting aligned vision-language space enables zero-shot transfer across downstream tasks without requiring task-specific fine-tuning. More recent approaches demonstrate that powerful off-the-shelf encoders can be leveraged to avoid training both modality encoders. LiT ( 43 ) freezes a pretrained vision backbone and trains only the text encoder, while FLAME ( 3 ) does the opposite–freezing the text model and learning the parameters of the vision encoder. However, these approaches still require substantial computational costs, as they involve training a large encoder with hundreds of millions of parameters. Post-hoc vision-language alignment with image-text pairs. A growing body of work focuses on connecting image and text encoders that were not jointly trained. The text-to-concepts framework ( 25 ) , for example, starts with a pretrained vision-only encoder and learns a single-layer MLP that maps its image representations into CLIP’s latent space. The combination of the vision encoder and the MLP, together with CLIP’s text encoder, produces similar representations for images and texts that share the same semantic content, in a manner analogous to CLIP’s original image and text encoders. Additionally, some methods aim to align image and text encoders that have been trained each exclusively on unimodal data. ASIF ( 27 ) embeds visual and text data into a common space using a fixed set of image-caption pairs: for a given image, its representation is the vector whose i -th component is the similarity (in the original image encoder’s space) between that image and the i -th image in the fixed set; likewise, for a given text, its representation is the vector whose i -th component is the similarity (in the original text encoder’s space) between that text and the i -th caption in the fixed set. CSA ( 15 ) applies Canonical Correlation Analysis to project pretrained image and text embeddings into a maximal correlation subspace. SAIL ( 44 ) contrastively trains two linear transformations on millions of image-caption pairs to align representations from both modalities. The QPA method ( 23 ) addresses alignment as a seeded graph-matching problem: it formulates a Fast Quadratic Assignment Problem optimization that directly aligns the similarity graphs of the two modalities. The issue with these approaches is that they require a sufficiently large and diverse set of multimodal image-text pairs for the alignment to work effectively, which may not be available in many practical domains and thus limits their applicability. Zero-shot classification by mapping text to weights. A related line of work creates zero-shot classifiers by mapping text representations directly to the weights of image classifiers. For example, methods like ICIS ( 5 ) and Zero-Shot Natural Language Explanations ( 34 ) use lightweight MLPs to transform text embeddings into classification weights. These approaches are constrained to the ImageNet-1K or other more specific domains, limiting their effectiveness beyond these settings. While previous work focuses on either text-to-image alignment or mapping text to new weights

Read the complete text