Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation
Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh · 2026
Read Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation by Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08874.txt.
Introduction
Hoang Truong, Hai Nguyen-Truong, and Dang Huynh segment images for any described category, including ones never trained on, by aligning language and picture features inside hyperbolic space. Open vocabulary segmentation strains because image level models must suddenly make pixel level calls, and hierarchy tangles with meaning. Earlier hyperbolic work aligned levels of the hierarchy but ignored meaning drift among items at the same level.
Their HyRo framework splits the fix in two: radial position in the Poincaré ball handles hierarchy, while a rotation that preserves radius cleans up angular semantic relations. Across standard benchmarks for open vocabulary segmentation, the method reaches state of the art results. Dense prediction gains geometry that respects both family trees and fine meaning.
Catalog record
| CREATOR | Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation Hoang M. Truong, Hai Nguyen-Truong, Dang Huynh Abstract Open-vocabulary semantic segmentation requires adapting image-level vision-language models such as CLIP to dense pixel-level prediction, which is challenging due to the mismatch between hierarchical structure and semantic alignment in the embedding space. While recent works leverage hyperbolic geometry to model hierarchical relationships, they align embeddings across hierarchical levels but overlook semantic misalignment among embeddings within the same level. In this work, we propose HyRo, a hyperbolic fine-tuning framework that decouples hierarchical and semantic alignment in the Poincaré ball model. HyRo aligns hierarchical levels by adjusting the hyperbolic radius and refines semantic relationships through angular alignment using an orthogonal transformation that theoretically preserves the hyperbolic radius. Experiments on standard open-vocabulary semantic segmentation benchmarks demonstrate that HyRo achieves state-of-the-art performance over prior methods. 1 Introduction Open-vocabulary semantic segmentation aims to assign each pixel in an image to a semantic category specified by textual descriptions, including categories unseen during training. To tackle this challenging task, vision-language foundation models such as CLIP 32 have emerged as powerful tools, as they are trained on large-scale image–text pairs and capture rich semantic alignments between visual and linguistic modalities. Consequently, CLIP is widely adopted for open-vocabulary learning by representing class names as text embeddings, effectively enabling language-driven classification for downstream tasks. However, CLIP is originally pre-trained for image-level representation learning, and thus requires additional adaptation to produce dense, pixel-level predictions for semantic segmentation. To achieve this adaptation, early approaches 38 ; 22 ; 45 ; 39 ; 40 decoupled the problem into two stages, utilizing mask proposal generators followed by a pre-trained CLIP model to classify the resulting regions. While such methods benefit from explicit object localization and clear boundaries, the reliance on pre-defined mask proposals introduces a closed-set bias that can limit scalability and generalization to unseen categories. Recent methods 5 ; 43 ; 37 instead directly fine-tune CLIP within a shared representation space, enabling dense pixel-level predictions while preserving the semantic richness of the original CLIP representation. Figure 1 : HyRo rotates the text embeddings to achieve a smaller angle ( \beta ) relative to the target image embeddings compared to the initial angle ( \alpha ). This geometric adjustment enables the model to resolve semantic ambiguities and produce more accurate, fine-grained segmentations. Notably, hyperbolic space has recently gained attention for its ability to capture intrinsic hierarchical structures. Several works 6 ; 29 incorporate hierarchical representations into CLIP-like vision–language models by embedding visual and textual entities in hyperbolic space, enabling the model to better capture hierarchical relationships between modalities. In contrast, a recent advancement in open-vocabulary semantic segmentation, HyperCLIP 31 , observes that the hierarchy of image embeddings shifts from an image-level to a pixel-level during fine-tuning. To explain this, they integrate the Poincaré ball model to adjust the hyperbolic radius of text embeddings to match this pixel-level granularity. However, a critical limitation of this approach is the absence of explicit constraints on the semantic alignment between embeddings, which is essential for effective open-vocabulary learning. Semantic relationships can be represented in hyperbolic space through geometric structures such as imaginary cones, as explored in prior work 6 ; 29 ; 36 , or equivalently through angular relationships at the origin. To address this key limitation, we introduce HyRo (Hyperbolic Rotation), a novel fine-tuning strategy that explicitly disentangles hierarchical and semantic information in hyperbolic space. While prior work primarily focuses on the hierarchical geometry of the embedding space, our approach explicitly optimizes the semantic aspect. HyRo is motivated by the observation that hierarchy and semantics correspond to distinct geometric properties, as hierarchy is encoded by radial distance, while semantic similarity is reflected in angular orientation. Figure 1 illustrates a failure case in prior work and demonstrates how HyRo resolves it. Specifically, prior work 31 fine-tunes solely via radius scaling and overlooks semantic orientation. This potentially leads to semantically dissimilar concepts being embedded at similar radii but incorrect angles, degrading the discrimination between categories. For instance, in an image depicting a person lying on a chair, HyperCLIP loses semantic understanding and misclassifies both entities as a single “chair” object. HyRo addresses this by rotating the initial embedding to achieve improved angular alignment with related embeddings. After this rotation, the model better segments the scene, correctly recognizing the person as the distinct foreground subject. We theoretically demonstrate that orthogonal transformations in the Poincaré ball model act as ideal rotation operations (see Section 3.2 ). These transformations enable angular adjustment of embeddings, refining semantic alignment without altering their radius, thereby preserving the hierarchical structure established by prior radius tuning. By decoupling these geometric properties, HyRo enables CLIP to simultaneously maintain the pixel-level granularity required for segmentation and enhance the semantic discrimination necessary for open-vocabulary generalization. Overall, our approach highlights the importance of explicitly modeling both hierarchical and angular relationships when adapting vision–language representations for dense open-vocabulary prediction. Our contributions are summarized as follows: • We propose HyRo, an orthogonal transformation strategy that adjusts angular relationships in hyperbolic space while preserving hierarchical structure, with theoretical justification provided. • We introduce a hyperbolic fine-tuning framework that decouples hierarchical alignment (radius) and semantic refinement (angle) for open-vocabulary semantic segmentation. • We demonstrate that HyRo achieves state-of-the-art performance on standard benchmarks, validating the effectiveness of our geometric approach. 2 Related Works 2.1 Open-Vocabulary Semantic Segmentation. Previous works 37 ; 5 ; 39 ; 43 ; 45 on open-vocabulary semantic segmentation aim to adapt foundation vision–language models pre-trained on large-scale datasets, such as CLIP 32 , for pixel-level prediction. This adaptation requires transforming high-level vision–language representations into dense predictions, which poses a key challenge for the task. Early approaches directly fine-tune the encoders of CLIP; however, prior methods 39 ; 42 ; 45 observe overfitting to seen classes, as such fine-tuning can degrade the generalization ability of CLIP. As a result, many works 12 ; 16 ; 40 ; 22 freeze the CLIP encoders to preserve their generalization. More recent studies 5 ; 43 ; 37 show that fine-tuning CLIP within cost aggregation–based frameworks can alleviate this issue. Beyond Euclidean representations, several works 31 ; 30 further explore alternative geometric spaces for open-vocabulary semantic segmentation, providing promising insights into modeling complex semantic structures across vision and language. While existing hyperbolic methods primarily optimize the radius of embeddings to capture hierarchical relationships, the angle governing semantic similarity remains largely unexplored. We address this gap by explicitly modeling angular relationships while preserving the hierarchical structure. 2.2 Hyperbolic Deep Learning. Hyperbolic geometry provides a natural framework for representing hierarchical structures due to its exponential volume growth, which mirrors the branching nature of tree-like hierarchies 18 . In contrast, Euclidean space exhibits polynomial volume growth, making it less suitable for embedding hierarchical data with low distortion 26 . This property has led to widespread adoption of hyperbolic spaces in natural language processing 28 ; 35 ; 7 ; 20 , where concepts are naturally organized in taxonomies and can be embedded as tree graphs with minimal distortion 33 ; 34 . Visual data also exhibits inherent hierarchical structures 1 , spanning multiple levels from pixels and patches to objects and scenes. This observation has motivated recent works 31 ; 19 ; 13