Topology-Aware CLIP Few-shot Learning

Dazhi Huang · 2025

Read Topology-Aware CLIP Few-shot Learning by Dazhi Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01694.txt.

Introduction

Dazhi Huang proposes a topology-aware way to adapt the CLIP vision-language model from just a few examples, in a 2025 paper. Fine-tuning risks erasing what the model already knows, while older add-on methods ignore the shape of the model's inner representation space. Huang's approach adds a topology term that keeps the visual and text representation structures aligned while a small residual module learns the task.

Only the light residual parameters train; the base encoders stay frozen. Across six benchmark data sets the method beat relevant baselines by one to two accuracy points on average in few-shot tests. Huang presents topological alignment as a cheap way to squeeze more from pretrained models.

Catalog record

CREATORDazhi Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01694

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Topology-Aware CLIP Few-shot Learning Dazhi Huang Abstract Efficiently adapting large Vision-Language Models (VLMs) like CLIP for few-shot learning poses challenges in balancing pre-trained knowledge retention and task-specific adaptation. Existing methods often overlook valuable structural information within the VLM’s latent space. We introduce a topology-aware tuning approach integrating Representation Topology Divergence (RTD) into the Task Residual (TR) framework. By explicitly aligning the topological structures of visual and text representations using a combined RTD and Cross-Entropy loss, while freezing base VLM encoders, our method enhances few-shot performance. We optimize only lightweight Task Residual parameters, effectively leveraging topological information. Across 6 diverse benchmark datasets, our approach demonstrates significant gains, achieving an average accuracy improvement of 1-2% over relevant baseline methods in few-shot settings. This work presents an effective strategy to boost VLM few-shot capabilities by incorporating topological alignment. Introduction Vision-Language Models (VLMs) have emerged as a powerful paradigm in artificial intelligence, bridging the gap between visual perception and natural language understanding. These models are typically trained on vast datasets comprising billions of image-text pairs scraped from the web, enabling them to learn rich and transferable semantic representations [ \citeauthoryearRadford et al.2021 , \citeauthoryearLi et al.2023 , \citeauthoryearAlayrac et al.2022 , \citeauthoryearLi et al.2022 ] . By learning to associate visual concepts with their textual descriptions at scale, prominent VLMs such as CLIP [ \citeauthoryearRadford et al.2021 ] , BLIP [ \citeauthoryearLi et al.2022 ] , BLIP-2 [ \citeauthoryearLi et al.2023 ] , and Flamingo [ \citeauthoryearAlayrac et al.2022 ] have demonstrated remarkable abilities in capturing nuanced semantic information directly from raw multimodal data. This large-scale pre-training imbues the models with a strong foundation of general-purpose knowledge, effectively aligning visual and linguistic modalities within a shared embedding space. Leveraging these powerful pre-trained representations for downstream tasks, particularly in few-shot scenarios where labeled data is scarce, has motivated the development of various efficient transfer learning (ETL) techniques. Mainstream approaches strive to adapt VLMs while preserving the valuable prior knowledge learned during pre-training. For instance, prompt tuning methods like CoOp [ \citeauthoryearZhou et al.2022 ] introduce learnable context vectors into the input prompts, effectively creating task-specific prompts. Adapter-style tuning approaches, such as CLIP-Adapter [ \citeauthoryearGao et al.2023 ] , insert lightweight modules, often with residual connections, into the VLM architecture to adapt features. Tip-Adapter [ \citeauthoryearZhang et al.2022 ] offers a training-free alternative by constructing a key-value cache model from the few-shot examples to modulate predictions. Methods like TaskRes [ \citeauthoryearYu et al.2023 ] take a different route by explicitly preserving the original VLM classifier and introducing tunable, prior-independent parameters as a task-specific residual. The central challenge in few-shot VLM adaptation lies in effectively utilizing the rich prior knowledge encoded within the pre-trained model while accurately inferring the posterior knowledge distribution specific to the target task from only a handful of examples. The inherent structure of the shared image-text latent space, however, remains largely unexplored by current ETL methods, which often focus on instance-level or pairwise relationships. Topological Data Analysis (TDA), particularly through the lens of persistent homology, offers a novel perspective by providing tools to compare the multi-scale topological structures of data manifolds. Metrics derived from TDA, such as cross-barcodes [ \citeauthoryearBarannikov et al.2021 ] and Representation Topology Divergence (RTD) [ \citeauthoryearBarannikov et al.2022 ] , have proven effective in quantifying structural differences between deep learning representations. Previous works have demonstrated the benefits of incorporating topological constraints; for example, Topological Autoencoders [ \citeauthoryearMoor et al.2020 ] and RTD Autoencoders [ \citeauthoryearTrofimov et al.2023 ] showed improved representation learning by aligning the topology of input and latent spaces. Similarly, TopoKD [ \citeauthoryearKim et al.2024 ] leveraged persistence images to indirectly match topological structures between teacher and student models for knowledge distillation. More directly relevant to VLMs, recent work explicitly constrained the structural equivalence of image and text latent manifolds using 0-dimensional persistent homology, achieving more effective ETL by aligning homology persistence between modalities [ \citeauthoryearZhang et al.2024 ] . We hypothesize that enforcing topological consistency between image and text representations can significantly enhance VLM few-shot learning. In data-scarce settings, the topology of the latent space, shaped by contrastive pre-training like CLIP’s which explicitly aligns modalities, encodes rich posterior knowledge. Specifically, 0-dimensional persistent homology captures semantic clustering information, including both intra-cluster similarity and inter-cluster separation. Furthermore, 1-dimensional homology can encode cyclic or periodic structures arising from continuous variations within related concepts, such as different breeds of dogs or cats within the same superclass. Capturing and aligning these multi-scale topological features offers a promising direction for improving the generalization and accuracy of VLMs when adapted with limited data. Motivated by the potential of TDA to capture rich structural information within latent spaces, this paper focuses on leveraging Representation Topology Divergence (RTD) to enhance the few-shot learning performance of VLMs. We employ a differentiable RTD-based loss function, specifically adapted to align the topological structures of image and text representations within the VLM’s shared embedding space. By explicitly enforcing topological consistency between modalities during the efficient transfer learning process, we aim to improve the model’s ability to generalize from limited data. The main contributions of this work are summarized as follows: • We are the first to introduce higher-dimensional topological features, quantified by RTD, as a constraint for few-shot learning in vision-language models, moving beyond simpler topological measures. • We employ a differentiable loss function derived from RTD, specifically tailored to align the multi-scale topological structures of cross-modal representations (image and text) within VLMs during efficient adaptation. • Through extensive experiments on multiple few-shot classification benchmarks, we demonstrate that our topology-guided alignment method significantly boosts performance compared to existing state-of-the-art efficient transfer learning techniques. Background Persistent Homology Persistent Homology (PH) is a central technique within Topological Data Analysis (TDA) designed to quantify and analyze the multi-scale topological structure inherent in data. Given a point cloud P sampled from some underlying space, PH employs the concept of filtration to understand how its topology changes across different proximity scales. A common way to construct a filtration from a point cloud P residing in a metric space is by using the Vietoris-Rips (VR) complex. For a given scale parameter \epsilon\geq 0 , the VR complex, denoted \mathcal{R}_{\epsilon}(P) , is a simplicial complex whose vertices are the points in P . A set of vertices \{p_{0},p_{1},\dots,p_{k}\}\subseteq P forms a k -simplex in \mathcal{R}_{\epsilon}(P) if the distance between any pair of vertices p_{i},p_{j} in the set is less than or equal to \epsilon . As \epsilon increases, more simplices are included, resulting in a sequence of nested complexes: \mathcal{R}_{\epsilon_{0}}\subseteq\mathcal{R}_{\epsilon_{1}}\subseteq\dots\subseteq\mathcal{R}_{\epsilon_{n}} , where 0=\epsilon_{0}<\epsilon_{1}<\dots<\epsilon_{n} . This sequence is known as the VR filtration. Figure 1: A filtration of a complex from a point cloud of four points. H_{p} K_{0} K_{1} K_{2} K_{3} K_{4} K_{5} K_{6} 0 \mathbb{Z}^{4} \mathbb{Z}^{3} \mathbb{Z}^{2} \mathbb{Z}^{2} \mathbb{Z} \mathbb{Z} \mathbb{Z} 1 0 0 0 \mathbb{Z} \mathbb{Z} \mathbb{Z}^{2} \mathbb{Z} Betti(p) K_{0} K_{1} K_{2} K_{3} K_{4} K_{5} K_{6} 0 4 3 2 2 1 1 1 1 0 0 0 1 1 2 1 By applying homology computation to each complex \mathcal{R}_{\epsilon_{i}} in the filtration, PH tracks the evolution of topological features, such as connected components ( H_{0} ), loops ( H_{1} ), voids ( H_{2} ), and higher-dimensional analogues. A specific p -dimensional topological feature

Read the complete text