Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding

Byeongju Woo, Zilin Wang, Byeonghyun Pak, Sangwoo Mo, Stella X. Yu · 2026

Read Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding by Byeongju Woo, Zilin Wang, Byeonghyun Pak, Sangwoo Mo, Stella X. Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02977.txt.

Introduction

Long, detailed captions defeat many vision-language models, and Byeongju Woo and 4 co-authors propose a part-to-whole remedy in a 2026 paper. Models like CLIP lean on dominant scene cues and miss fine visual evidence. The authors argue a model should first find what parts appear where, then build its whole-scene understanding from those parts.

Their model, CAFT, aligns local text with image regions at intermediate layers. It also aligns whole images with full captions at the final layer. Long captions supply the part descriptions, organized from fine to coarse. Trained on thirty million image-text pairs, CAFT set new best results on six long-text retrieval benchmarks. It localized textual details in images without explicit region supervision.

Catalog record

CREATORByeongju Woo, Zilin Wang, Byeonghyun Pak, Sangwoo Mo, Stella X. Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding Byeongju Woo, Zilin Wang, Byeonghyun Pak, Sangwoo Mo, Stella X. Yu Abstract Large vision-language models such as CLIP struggle with long captions because they align images and texts as undifferentiated wholes. Fine-grained vision-language understanding requires hierarchical semantics capturing both global context and localized details across visual and textual domains. Yet linguistic hierarchies from syntax or semantics rarely match visual organization, and purely visual hierarchies tend to fragment scenes into appearance-driven parts without semantic focus. We propose CAFT ( C ross-domain A lignment of F orests and T rees ), a hierarchical image-text representation learning framework that aligns global and local semantics across images and long captions without pixel-level supervision. Coupling a fine-to-coarse visual encoder with a hierarchical text transformer, it uses a hierarchical alignment loss that matches whole images with whole captions while biasing region-sentence correspondences, so that coarse semantics are built from fine-grained evidence rather than from aggregation untethered to part-level grounding. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that hierarchical cross-domain alignment enables fine-grained, visually grounded image-text representations to emerge without explicit region-level supervision. Figure 1 : Long-text image retrieval demands grounded understanding. Identifying the correct image (bottom) requires recognizing both its coarse composition ( what major elements are present : bus, car, building etc.) and its fine-grained details ( what each element is : a modern red double-decker bus labeled route 38 ). In the wrong image (top), a long caption causes the baseline model to latch onto some dominant descriptions while neglecting others, leading to errors in both scene composition and local details. This example illustrates the need for models that align visual and textual structure across scales, seeing both the forest and the trees . 1 Introduction A red bus is never just a red bus ( Figure 1 ): a modern red double-decker, labeled route 38, pedestrians nearby, a hatchback car passing, a stone building with arched windows and a flagpole on its roof . Such long text can be both a blessing and a trap: Every extra clause adds detail but also distraction. A model that merely matches whole captions to whole images may seize on a dominant yet incomplete cue ( 37 ; 59 ; 47 ) , overlooking critical details such as the bus number or the broader scene composition involving buses , a car , and a stone building . To get it right, a system must see both forest and trees , within and across visual and textual domains ( 4 ; 31 ) . Figure 2 : Vision-language understanding requires hierarchical semantics to be aligned across domains. Traditional textual hierarchies , such as syntactic parse trees or semantic graphs, do not naturally correspond to the spatial organization of visual scenes. In contrast, spatially oriented textual hierarchies , where each part describes a distinct image region, align naturally with visual hierarchies . Visual understanding requires organizing image patches into objects , and objects into a coherent scene , while capturing spatial and compositional relationships such as a person stepping onto a bus , a car passing by , or a building behind the vehicles ( 12 ; 27 ) . Language can assist this process, but not through its standard syntactic or semantic representations alone ( Figure 2 ). Parse trees encode grammatical structure ( 38 ; 11 ) , while semantic graphs emphasize events and abstract relations rather than the geometric and compositional organization of buses , cars , and buildings ( 22 ; 57 ; 52 ) . What is needed instead is a hierarchical textual representation that mirrors how scenes are composed, enabling textual units at multiple levels of granularity to be grounded in corresponding visual components and their relationships. We propose CAFT that operationalizes long-text grounded understanding by C ross-domain A lignment of F orests and T rees , based on three design insights. 1. Spatially structured long captions. Many long captions decompose into sentences describing scene components. We leverage this structure to induce a textual hierarchy , in which each sentence represents a candidate semantic unit that may be grounded in a corresponding visual component, although the specific region it corresponds to is unknown and must be discovered. 2. Coherent within-domain hierarchies. Both images and texts are represented hierarchically, with coarse semantics explicitly constructed from finer-grained evidence rather than bypassing it. On the text side, we design a hierarchical text transformer that first encodes sentence-level units in parallel and then composes them into a global description. On the visual side, we employ a fine-to-coarse visual encoder that preserves part-whole structure, ensuring that a red double-decker bus labeled route 38 is not indiscriminately aggregated into a generic bus . These design choices ensure that global semantics are composed from fine-grained, grounded evidence, rather than being learned by indiscriminately aggregating details into a single global representation. 3. Hierarchical cross-domain alignment. At the core of CAFT is a hierarchical alignment objective that jointly discovers aligned parts and wholes across image and text, with visual regions discovered without any image-domain supervision and aligned to sentence-level units without explicit image–text cross-domain correspondences. The objective matches whole images with whole captions while biasing mid-level correspondences between visual components and sentence-level textual units, so that global scene semantics are grounded in localized evidence rather than dominated by a single salient cue. a) Top-5 image retrievals for the example in Figure 1 . b) Attention maps of individual sub-captions. Figure 3 : CAFT demonstrates fine-grained, grounded vision-language understanding. a) Among visually similar bus images, the baseline method, FLAIR, wavers among look-alikes, whereas our proposed CAFT selects the correct match with a clearly higher retrieval score. b) This improvement arises from representations that attend individual sub-captions to their corresponding image regions, emerging naturally without explicit supervision. Trained on 30M image–text pairs, CAFT exhibits strong scaling behavior and achieves state-of-the-art performance across six long-text retrieval benchmarks. Beyond aggregate metrics, CAFT demonstrates qualitatively stronger grounded understanding. As shown in Figure 3 , it clearly distinguishes between visually similar scenes by attending to fine-grained caption components and grounding them in the appropriate image regions, whereas baseline models waver among look-alikes. These gains stem from attending to both trees and forest simultaneously. Rather than collapsing a long caption or an image through indiscriminate aggregation, CAFT preserves and aligns fine-grained components, grounding each caption segment in corresponding visual evidence. By explicitly accounting for individual details, the model constructs more accurate global scene representations, enabling reliable discrimination between subtly different scenes. This compositional grounding explains why CAFT consistently outperforms a range of alignment baselines, particularly in long-text retrieval settings where attending to only the forest or only the trees – without modeling their hierarchical relationship – leads to ambiguous matches. An additional and exciting outcome is that CAFT acquires meaningful segmentation from images and text alone, enabling unsupervised visual parsing without any segmentation annotations. Benchmarked on zero-shot referring segmentation, CAFT significantly outperforms representative methods such as GroupViT ( 50 ) and MaskCLIP ( 61 ) . These results corroborate that aligning natural spatial hierarchies across image and text is key to visually

Read the complete text