Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Text Representations for Language-based Object Detection
Sojung An, Kwanyong Park, Yong Jae Lee, Donghyun Kim · 2025
Read Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Text Representations for Language-based Object Detection by Sojung An, Kwanyong Park, Yong Jae Lee, Donghyun Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24192.txt.
Introduction
Finding an object from a rich description, full of traits and relations, still trips up vision-language models. Sojung An and 3 co-authors propose rebuilding the text side: split each sentence into objects, attributes, and relations, then stack those parts into a layered representation. Their framework, TaSe, follows that plan for language-based object detection.
TaSe adds a synthetic caption data set with three tiers, from plain category names to full descriptive sentences. A disentanglement module, guided by a new loss, turns text embeddings into composed subspaces. Layered objectives then merge the parts into sentence-level codes. On the OmniLabel benchmark, the method improved performance by about a quarter.
Catalog record
| CREATOR | Sojung An, Kwanyong Park, Yong Jae Lee, Donghyun Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Talk in Pieces, See in Whole: Disentangling and Hierarchical Aggregating Text Representations for Language-based Object Detection Sojung An, Kwanyong Park, Yong Jae Lee, Donghyun Kim Abstract While vision-language models (VLMs) have made significant progress in multimodal perception (e.g., open-vocabulary object detection) with simple language queries, state-of-the-art VLMs still show limited ability to perceive complex queries involving descriptive attributes and relational clauses. Our in-depth analysis shows that these limitations mainly stem from text encoders in VLMs. Such text encoders behave like bags-of-words and fail to separate target objects from their descriptive attributes and relations in complex queries, resulting in frequent false positives. To address this, we propose restructuring linguistic representations according to the hierarchical relations within sentences for language-based object detection. A key insight is the necessity of disentangling textual tokens into core components—objects, attributes, and relations (“talk in pieces”)—and subsequently aggregating them into hierarchically structured sentence-level representations (“see in whole”). Building on this principle, we introduce the TaSe framework with three main contributions: (1) a hierarchical synthetic captioning dataset spanning three tiers from category names to descriptive sentences; (2) Talk in Pieces , the three-component disentanglement module guided by a novel disentanglement loss function, transforms text embeddings into subspace compositions; and (3) See in Whole , which learns to aggregate disentangled components into hierarchically structured embeddings with the guide of proposed hierarchical objectives. The proposed TaSe framework strengthens the inductive bias of hierarchical linguistic structures, resulting in fine-grained multimodal representations for language-based object detection. Experimental results under the OmniLabel benchmark show a 24% performance improvement, demonstrating the importance of linguistic compositionality. 1 Introduction Vision-language (VL) understanding, which aims to perceive each modality and form associations between them, is a long-standing and fundamental problem. Recently, foundational VLMs such as CLIP ( Radford et al. 2021a ) have leveraged web-scale image-text pairs to learn generic VL representations, achieving strong generalization performance on tasks like image classification and image-text retrieval. Building upon these advances, recent studies have actively explored grounding language queries into specific image regions (e.g., open-vocabulary object detection ( Liu et al. 2024b ; Zhao et al. 2024 ; Yin et al. 2025 ) ). Many approaches ( Liu et al. 2024a ; Li et al. 2022 ) distill the general VL knowledge embedded in foundational models into object detectors and have demonstrated remarkable results in detecting previously unseen object categories—commonly referred to as open-vocabulary object detection ( Gu et al. 2021 ) . (a) Detection results from w/o and w/ our hierarchy entailment (b) t-SNE of sentence features from text encoder of VLMs Figure 1: (a) VL detectors struggle with attributes or relations due to limitations in capturing fine-grained semantics from image-text similarity. We propose a hierarchical multimodal representation learning to enhance the linguistic compositionality of complex language queries. (b) Limitations of text encoders in VLMs for compositional understanding. Although some sentences refer to different target objects, their embeddings remain close due to shared tokens, contrary to the ideal case where their embeddings should be well-separated (e.g., “with a man” vs. “with a woman” ). Despite these advances, current VL detectors often succeed only when the input queries are short and consist of simple category names. They still struggle to fully comprehend complex language queries and accurately localize the corresponding objects. To illustrate this limitation, we conduct a preliminary analysis using the state-of-the-art foundation model for visual grounding, GLEE ( Wu et al. 2024 ) (see Fig. 1(a) ). The model reliably detects objects given simple noun phrases (e.g., “segway”). However, it fails when faced with more complex and specific queries (e.g., “segway with a man”), indicating its limited compositional understanding. To investigate the underlying cause of this limitation, we visualize sentence-level text features using t-SNE ( Van der Maaten & Hinton 2008 ) in Fig. 1(b) . Interestingly, we observe that although some sentences (“a segway with a man” vs. “a segway with a woman”) refer to different target objects (“with a man” vs. “with a woman”), their embeddings remain close in the feature space due to shared tokens (“segway”). The contrasts with the ideal cases, where embeddings of distinct target objects should be well-separated, while those of the same object with different attributes should be closer for object detection (“a segway” vs. “a black segway”). These results indicate limited hierarchical and compositional understanding in current VLMs ( Yuksekgonul et al. 2022 ) . Most VL detectors ( Liu et al. 2024a ; Li et al. 2022 ) are trained to align a few positives with image regions while distinguishing them from negatives using contrastive learning. For example, training with augmented captions ( Li et al. 2023b ; Yuksekgonul et al. 2022 ) labeled as positive or negative is effective for coarse-grained concept analysis. Still, detectors often struggle to handle tasks that require understanding of fine-grained text, such as reasoning over interactions between objects (e.g., “bigger than”). Sentence embeddings obtained via simple pooling compress token-level information and fail to capture contextualization in language queries. Beyond augmentation, sentence embeddings need to disentangle text tokens and encode compositional features. We argue that VL detectors should go further: representations need to see the whole sentence from meaningful pieces. In this paper, we propose a novel framework that disentangles component-wise text features (“Talk in Pieces”) and explicitly learns hierarchical knowledge (“See in Whole”) from these disentangled representations to construct sentence-level understanding for language-based object detection. We refer to our framework as TaSe ( Ta lk in Pieces, Se e in Whole). We begin by formally defining a hierarchical structure tailored for language-based object detection. Specifically, we design our new HiVG dataset, a three-tiered hierarchy, object–attribute–relation, where the first tier represents object category names, the second tier adds descriptive attributes, and the last tier includes relational phrases that describe interactions or contexts. Our approach builds on phrase grounding datasets like Visual Genome ( Krishna et al. 2017 ) , which provide densely annotated phrases associated with images and object regions. Using a large language model (LLM) ( Dubey et al. 2024 ) , we abstract these phrases into a three-tier hierarchy—object, attribute, relation—by sequentially removing relational and attribute information in phrases to obtain the final object categories. Unlike typical generation-based approaches (e.g., generating sentences from category names ( Li et al. 2023b ) or captions from images), our abstraction-based process performs reverse abstraction, effectively mitigating hallucination issues ( Ji et al. 2023 ) common in generative models. To effectively construct contextualized (i.e., holistic) sentence representations from the HiVG dataset, we disentangle into several key aspects (“Talk in Pieces”)—such as objects, attributes, and relations. This design allows us to disentangle text representations into subspaces to adjust targeted token embeddings and preserve meaningful information in the remaining features. For this purpose, we further design a lightweight learnable attention module for the TriDe (Three-component disentanglement), enabling efficient fine-tuning of conventional text encoders. The key idea of TriDe is to leverage the hierarchical structure of the HiVG dataset to contrast component-wise tokens so that targeted tokens to be adjusted without loss of meaningful information. Then, we guide the model to learn linguistic representations that capture these levels of abstraction. This facilitates learning of sentence context enriched with descriptive attributes and relational clauses. We introduce a hierarchical aggregation method