Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Chenhao Wang, Yingrui Ji, Yu Meng, Yao Zhu · 2026
Read Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation by Chenhao Wang, Yingrui Ji, Yu Meng, Yao Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15942.txt.
Introduction
Chenhao Wang and three colleagues propose Decomposed Vision-Language Alignment for fine-grained open-vocab segmentation. Open-vocab models struggle with new category-attribute mixes, since descriptions arrive as whole sentences mixing several meaning units. The framework splits textual prompts into a concept token plus attribute tokens, allowing separate cross-modal links per unit. A Feature-Gated Cross-Attention module builds attribute gating maps to fuse information by products, keeping compositional meaning. Per-token scores combine in log-space for stable, clear matching, lifting generalization to unseen attribute-category pairs.
Catalog record
| CREATOR | Chenhao Wang, Yingrui Ji, Yu Meng, Yao Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38988 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15942
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation Chenhao Wang, Yingrui Ji, Yu Meng, Yao Zhu Abstract Open-vocabulary segmentation models often struggle to generalize to unseen combinations of object categories and attributes, because fine-grained descriptions are typically encoded as holistic sentences that entangle multiple semantic units. We propose a Decomposed Vision-Language Alignment framework that explicitly factorizes textual prompts into a concept token and multiple attribute tokens, enabling separate cross-modal interactions for each semantic unit. At the feature level, we introduce a Feature-Gated Cross-Attention module that generates attribute-specific gating maps to fuse information in a multiplicative manner, effectively enforcing compositional semantics. At the scoring level, per-token similarities are aggregated in log-space, producing a stable and interpretable compositional matching. The method can be seamlessly integrated into existing transformer-based segmentation architectures and significantly improves generalization to unseen attribute–category compositions in fine-grained open-vocabulary segmentation benchmarks. 1 Introduction Open-vocabulary segmentation (OVS) aims to localize and segment image regions conditioned on arbitrary textual prompts. Benefiting from large-scale vision–language pretraining [ 24 ] , OVS moves beyond conventional closed-set category vocabularies and enables models to generalize to previously unseen semantic concepts at inference time [ 15 , 9 , 32 , 18 , 35 , 33 ] . In real-world scenarios, targets are often described not only by object categories but also by multiple attributes, such as industrial building with flat roof or residential building with gable roof. These descriptions form compositional semantics, where a category is jointly specified by several attributes. Training data rarely covers all possible attribute–category combinations, so models must generalize to unseen compositions at test time. Compositional generalization has been extensively studied in Compositional Zero-Shot Learning (CZSL). Prior works represent categories as attribute–object pairs and perform compatibility modeling [ 8 , 1 ] , graph-based propagation [ 21 ] , or feature factorization [ 23 , 28 ] to recognize unseen compositions at the image level. In the generalized CZSL setting, bias calibration and harmonic mean evaluation protocols are introduced to balance seen and unseen compositions [ 31 ] . These methods explicitly decompose semantic descriptions into attribute and object components and model their interactions in the label embedding space. Decomposition is performed at the classification level: each composition is scored by measuring compatibility between a global image representation and a composed label vector. This design is tailored for image-level recognition and does not carry over to dense prediction, where semantic units must be independently aligned with spatial regions rather than composed in a label space disconnected from pixel-level features. Recent analyses further indicate that even large-scale vision–language models exhibit systematic biases in fine-grained compositional reasoning [ 39 , 27 ] . In detection, localization, and instance segmentation, the challenge shifts from label-level composition to cross-modal alignment of semantic units at region and pixel levels. Although existing open-vocabulary detection and segmentation methods leverage vision–language pretraining to enhance category generalization [ 41 , 34 ] , they encode the entire textual prompt as a single sequence. Such holistic encoding entangles category and attribute semantics in a shared representation. The compositional structure within the prompt is not explicitly modeled. When certain attribute–category combinations are absent during training, the model has no mechanism to decompose and recombine semantic units at inference, leading to degraded performance on unseen compositions. Transferring the decomposition insight from CZSL to this setting therefore requires a different formulation: each semantic unit must interact with visual features independently through cross-modal attention, and their outputs must be composed with explicit AND-style constraints at the feature and scoring levels. To address this, we propose a decomposed vision–language compositional modeling framework for fine-grained open-vocabulary segmentation. Each textual prompt is decomposed into a category token and multiple attribute tokens, and cross-modal interaction is performed with each semantic unit separately, avoiding the entanglement introduced by holistic encoding. At the feature level, we introduce a Feature-Gated Cross-Attention module that generates attribute-specific gating maps and enforces AND constraints via element-wise multiplication, treating each attribute as a necessary condition filter. At the scoring level, we propose a log-space compositional aggregation strategy that sums per-unit matching scores in log-probability space, yielding a numerically stable formulation of AND semantics with improved optimization behavior. We construct a compositional generalization evaluation protocol that explicitly includes unseen attribute–category compositions and conduct extensive experiments on two benchmarks. Our main contributions can be summarized as follows: • We propose a Decomposed Vision-Language Alignment framework that can be seamlessly integrated into existing transformer-based segmentation architectures, significantly improving fine-grained open-vocabulary segmentation. • We propose Feature-Gated Cross-Attention, where attributes act as necessary-condition filters, decoupling attribute-specific feature learning. • We introduce a Log-Space AND Compositional Scoring scheme that explicitly models logical AND relationships among semantic units. • We introduce a compositional generalization evaluation protocol for open-vocabulary segmentation, and conduct extensive experiments that demonstrate the effectiveness of our framework. 2 Related Work 2.1 Open-Vocabulary Segmentation Traditional semantic segmentation models operate under a closed-set assumption, requiring retraining whenever new categories are introduced [ 20 , 3 , 4 ] . Open-vocabulary segmentation (OVS) relaxes this constraint by conditioning on free-form text prompts, enabling recognition of arbitrary categories at inference time. CLIP [ 24 ] and ALIGN [ 12 ] establish joint image–text embedding spaces through contrastive pretraining, and have become the standard backbone that OVS methods build upon for dense prediction. Early OVS approaches follow a two-stage pipeline: a class-agnostic proposal network generates mask candidates, which are then ranked using CLIP similarity. OpenSeg [ 9 ] , ZSSeg [ 36 ] , and ZegFormer [ 6 ] are representative of this decoupled design. Subsequent methods tighten the integration between vision–language alignment and segmentation. OVSeg [ 18 ] fine-tunes CLIP on masked image regions to better handle partial observations; FC-CLIP [ 38 ] processes both tasks within a single convolutional pass; CAT-Seg [ 5 ] constructs explicit cost volumes between image and text features to improve localization precision. In remote sensing, MovSeg [ 11 ] extends this line by incorporating near-infrared bands alongside RGB through parameter-efficient adaptation. A related direction focuses on sub-sentence grounding, aligning individual phrases with specific image regions. GLIP [ 16 ] reformulates detection as phrase grounding, enabling region–text correspondence below sentence level. MDETR [ 13 ] conditions cross-attention on different text spans to handle multi-phrase queries, and X-Decoder [ 42 ] unifies referring segmentation with panoptic segmentation under a shared text-conditioned query mechanism. SAM [ 14 ] establishes strong promptable segmentation capabilities by training on over one billion masks, and SAM 2 [ 26 ] extends this to video with memory-augmented decoding. Building upon these foundations, SAM 3 [ 2 ] further unifies geometric localization with native vision–language reasoning. Several works adopt this family of foundation models as frozen mask generators, pairing them with separate vision–language modules for semantic labeling. SOPSeg [ 29 ] follows this paradigm for small object segmentation in remote sensing, combining region-adaptive magnification with an edge-aware decoder to preserve fine spatial details. Referring image segmentation addresses a closely related problem of localizing regions described by natural language. LAVT [ 37 ] and CRIS [ 30 ] align visual and linguistic features through cross-modal attention, while GRES [ 19 ] handles more general expressions that may refer to multiple instances or no target at all. 2.2 Compositional Zero-Shot Learning