Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann · 2026

Read Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings by Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26798.txt.

Introduction

Schwalbe and co-authors present a post-hoc framework to explain and fix the semantic hierarchies inside vision-language embeddings. Models like CLIP sort classes into trees that users rarely inspect. Their method clusters class centres into a binary tree and names inner nodes from a concept bank. Plausibility is scored against human ontologies, and a light transform can bend the space toward a wanted tree. Across 13 models, image encoders split classes more sharply, while text encoders build trees closer to human ones.

Catalog record

CREATORGesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80545
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann Abstract Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image–text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain , verify , and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces. 1 Introduction Figure 1 : Our post-hoc framework to explain (embed given concepts, cluster and name found parents ), verify , and align the semantic hierarchy induced by a VLM in its embedding space. Contrastive vision-language models (VLMs) such as CLIP ( 44 ) demonstrate remarkable capabilities in learning joint representations of images and text 6 . These representations are adopted in various applications, including image retrieval 5 , captioning 6 , and zero-shot classification 44 . However, their evaluation is still dominated by task-level performance metrics such as classification accuracy that provide little insight into how concepts are internally organized and related. In particular, understanding the semantic hierarchies of these representations has not been thoroughly explored 26 . Semantic hierarchies capture how concepts relate via commonalities and differences, and are central to human categorization and symbolic knowledge representation via ontologies 10 ; 21 . Extracting the hierarchy induced by a VLM is crucial for explainability 9 ; 3 ; 26 ; 53 . It further enables identifying unintuitive or biased groupings (e.g., grouping man and woman primarily via hair rather than under person ) and aligning learned similarities with human knowledge 45 ; 10 . Lastly, hierarchies provide an inductive bias for more robust and informative classification 8 ; 13 ; 18 , such as explainable hierarchical tree traversal 53 . Meanwhile, key desiderata formulated for VLM embedding spaces are text-to-image alignment, and that classes can be discriminated via zero-shot classification. We complement this by two new desiderata: Text and image encoders should also align in their induced hierarchies; and they should be plausible , i.e., the hierarchy should match human taxonomies. Note that plausibility and discriminability may diverge, since embedding similarities may accurately discriminate class members, but still violate human expectations. Into that direction, prior work has explored the interpretability of VLM embeddings ( 9 ) ( Which concepts are encoded? ) and found and reinforced pre-defined semantic hierarchies in them ( 3 ) ( Is a given hierarchy encoded? ). However, there is a lack of systematic methods to answer the following questions for a trained VLM encoder, along which we organize our pipeline (cf. Figure 1 ): Q1: Can we explain which semantic hierarchy is induced by an encoder for a given set of leaf classes? How faithful and reliable are local explanations via explainable hierarchical inference? ( Sections 4.1 , 4.2 and 5.2 ) Q2: Can we verify the plausibility of induced hierarchies relative to reference ontologies? How do they vary across modalities, models, and hierarchy depths? ( Sections 4.3 and 5.3 ) Q3: Can we post-hoc align an induced hierarchy to a target similarity structure without sacrificing zero-shot accuracy? ( Sections 4.4 and 5.4 ) Contributions. We demonstrate the utility of semantic hierarchy extraction: \blacktriangleright We propose a post-hoc pipeline to explain, verify, and align semantic hierarchies in VLM embedding spaces. We extract a hierarchy via hierarchical clustering of leaf classes and dictionary-based parent naming, verify it with efficient ontology-based comparison metrics, and align it using UMAP-guided target neighborhoods and a lightweight embedding transformation. \blacktriangleright We empirically study 13 pretrained VLMs across 4 datasets and multiple common-sense ontologies, enabling systematic comparison of encoder semantics. \blacktriangleright We find a consistent modality gap: image encoders yield higher zero-shot accuracy, while text encoders induce more plausible hierarchies; and enriching hierarchical inference with UAES improves semantic correctness, bridging the explainability-faithfulness tradeoff. {forest} {forest} (a) Comparison of two extracted hierarchies over the CIFAR-10 leaves (shown in blue ) from SigLIP 56 , using image-only leaf embeddings (left) versus combined text&image leaf embeddings (right). (b) Illustration of tree-traversal vs. zero-shot inference in embedding space of the text & image tree. Figure 2 : Running toy example on CIFAR-10 2 leaf classes. 2 Related Work Explaining Learned Concepts. The field of concept-based XAI (c-XAI) has evolved around the goal to associate vectors in the latent representations with semantic concepts 26 . Such associations have found manifold applications in the inspection of a DNN’s learned ontology, such as: which concepts from a given ontology are learned? 45 , how similar are pairs of concepts? 16 ; 49 , and how are concepts from different layers or models correlated? 24 ; 36 . Notably, we here build upon the approach by ( 55 ) to name parent embeddings. They utilize the text-to-image alignment in VLMs to match vectors to closest text embeddings from a dictionary. To go from concepts to hierarchical concept relations, ( 53 ) employed hierarchical clustering on latent representations, but unlike our approach restricted to the last layer of a classifier. And despite rising interest in verification and steering of DNNs against ontologies 45 ; 42 ; 4 ; 3 , it remains largely unexplored how to post-hoc explain and control the learned semantic similarity structure within DNN embedding spaces 10 . Understanding VLM Embedding Spaces. The CLIP-based learning scheme 44 ; 22 of concurrent VLMs have been shown to emerge deep concept understanding 9 ; 62 , and even class hierarchies 3 ; 18 —though existing hierarchy extraction techniques rely on costly parent supervision and don’t align with ontology concept dictionaries. Moreover, previous findings that integrating hierarchy information into the inference process makes classification more explainable 53 and reliable 8 ; 13 , were shown to transfer to VLMs 39 . We therefore adapt the hierarchy tree traversal approach from ( 53 ) for VLM classification, newly adding early stopping. Despite these impressive properties, the modality alignment in CLIP embedding spaces remains suboptimal, mapping text and image to distinct latent regions 29 ; 14 , and requiring prompt finetuning to boost zero-shot performance 44 ; 30 ; 18 . Other than prior studies, we here for the first time investigate the text-to-image alignment with respect to the primarily learned semantic hierarchies of the encoders. Ontologies and Ontology Matching. An ontology reflects a perspective on a domain

Read the complete text