COCOTree : A Dataset and Benchmark for Open Tree-Structured Visual Decomposition

Junhyub Lee, Seunghun Chae, Hyosu Kim · 2026

Read COCOTree : A Dataset and Benchmark for Open Tree-Structured Visual Decomposition by Junhyub Lee, Seunghun Chae, Hyosu Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22068.txt.

Introduction

Junhyub Lee, Seunghun Chae, and Hyosu Kim formalize open tree decomposition, which splits an image into hierarchical trees of visual parts with free granularity. Manual annotation for such trees is very hard, so the team builds a fully automated generation pipeline. Human checks confirm the generated annotations match human judgment well, including on complex physical assemblies. They also set a standard evaluation protocol with the Open Tree Quality metric, which jointly rates mask precision, label accuracy, and structural consistency, and they release dataset and code.

Catalog record

CREATORJunhyub Lee, Seunghun Chae, Hyosu Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51341
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22068

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

COCOTree : A Dataset and Benchmark for Open Tree-Structured Visual Decomposition Junhyub Lee, Seunghun Chae, Hyosu Kim Abstract We formalize and enable the task of open tree decomposition , which segments an image into hierarchical trees of visual components with unconstrained granularity and flexibility. Specifically, we provide the foundation benchmark for this new paradigm with the following three key contributions. First, we overcome the prohibitively high cognitive and physical bottlenecks of manual annotation by developing a fully automated generation pipeline that synergizes the semantic reasoning of Large Vision-Language Models (LVLMs) with the precise geometric grounding of SAM 3. Second, leveraging this pipeline, we construct COCOTree , a massive-scale benchmark featuring over 21K images and 1.8M structural nodes. By embracing an open-vocabulary space of over 3.5K unique labels, it successfully captures the long-tail distribution of complex physical assemblies. Notably, rigorous human evaluation confirms our generated annotations demonstrate strong alignment with human structural judgment. Third, we establish a standardized evaluation protocol by proposing the Open Tree Quality (OTQ) metric, which jointly assesses mask precision, label accuracy, and structural consistency. We release our dataset and benchmark code at https://github.com/melonkick3090/COCOTree . 1 Introduction Image segmentation has long served as a fundamental pillar of visual recognition. It partitions an image into either semantic category regions, distinct object instances, or panoptic outputs at the pixel level [ 19 , 10 , 11 ] . In particular, recent advances in image segmentation have enabled the segmentation of images into extremely fine-grained units, e.g., constituent parts of objects [ 24 ] , providing granular comprehension of complex physical environments. However, existing methods typically provide a single-level abstraction of these fine-grained components, entirely ignoring the hierarchical dependencies inherent in real-world assemblies. This lack of structural information not only hinders compositional reasoning over visual scenes but also causes severe functional ambiguity: an isolated "handle" segment provides no structural clues as to whether it can be used to open a door, hold a cup, or turn a mechanical valve. These problems pose a critical bottleneck for autonomous systems, such as embodied AI agents, which require complex physical manipulation rather than mere visual perception. In response to the need for hierarchical image understanding, hierarchical datasets [ 3 , 5 , 9 , 26 , 16 ] have been introduced. These datasets capture compositional relationships but with severe limitations in granularity and flexibility. First, they primarily confine their structural depth to single-step object-part dependencies, rarely extending to the deeper sub-part hierarchies. This shallow abstraction thus loses the deep, critical structural chain of objects (e.g., tracing from a car, to its door, but not down to the specific door handle). Furthermore, these structural annotations are strictly constrained by closed-set vocabularies. Such a rigid taxonomy-based annotation might be effective for object recognition, but is entirely inadequate for modeling the unconstrained compositional structures inherent to the real world. That is, these datasets ignore any novel, long-tail components or structural configurations falling outside the taxonomies, although they are clearly present in the images. In this paper, we facilitate the development of algorithms for hierarchical analysis with unconstrained granularity and flexibility, a task which we call open tree decomposition . Specifically, we provide a large-scale dataset in which images are represented as hierarchical trees of visible components, called open trees . Each node in the tree encapsulates a single instance mask paired with its corresponding semantic label. In particular, the tree’s hierarchical structure and each node’s label are derived from the unconstrained visual reality of the image, rather than forced into a predefined template. However, constructing such highly granular annotations through human labor is prohibitively challenging at scale. Even worse, deriving unconstrained structural and linguistic annotations based on the given visual evidence exponentially amplifies the cognitive burden. To overcome the fundamental limitations of manual annotation, we develop a fully automated annotation pipeline that recursively leverages recent Large Vision-Language Models (LVLMs) and SAM 3 2 . Starting from the entire scene, it prompts LVLMs for a semantic decomposition of the current visual region. These semantic proposals are then spatially grounded into precise instance masks by SAM 3. Each generated mask is subsequently cropped and fed back into the LVLM as a new parent node, repeating this cycle until the model determines no further meaningful subdivisions exist. Figure 1: Overview of COCOTree . Left: COCOTree provides dense open-tree annotations over COCO images. Middle: each image is decomposed into visible components grounded by instance masks. Right: the same annotation can be viewed as a semantic-node tree, which groups repeated masks under a shared local label, and as an instance-node tree, where each mask becomes a node with an explicit visual parent. Finally, we introduce COCOTree , built upon this automated pipeline. Originating from the COCO dataset 17 , COCOTree benefits from a high density of complex, interacting objects within everyday environments like other COCO-based datasets [ 8 , 6 , 26 , 5 ] , while simultaneously providing unprecedented hierarchical granularity and flexibility (see examples in Fig. 1 ). It features over 21K images and 1.8M total instance masks (i.e., nodes), achieving an incredibly dense annotation rate of 85.7 masks per image and an average tree depth of 3.448. Furthermore, it catalogs over 3.5K unique open-vocabulary labels, successfully capturing the long-tail distribution of real-world physical components. It should be noted that we ensured the practical utility of these automated annotations by assessing their quality via extensive human evaluation, which confirms strong alignment with human judgment regarding both hierarchical structures and open-vocabulary labeling. COCOTree then serves as a benchmark for open tree decomposition tasks with a metric, called Open Tree Quality (OTQ) . This is a PQ (Panoptic Quality [ 11 ] )-like metric that jointly evaluates mask quality, label quality, and structural consistency. 2 Related Work 2.1 Hierarchical Segmentation Hierarchical segmentation datasets have extended image segmentation beyond a single level of abstraction by annotating visual structures. Specifically, numerous benchmarks capture the compositional relationships between parent objects and their part components 3 , 9 , 26 , 21 . Recently, this hierarchical granularity has been further increased to capture part-subpart relationships in natural images 36 , 24 . Despite these advancements, existing paradigms force visual representations into rigid, predefined templates. By limiting structural hierarchies to a shallow, two- or three-tier depth and restricting labels to a closed vocabulary, these datasets strip away the unconstrained structural context of the physical world. This shallow abstraction creates a fundamental data bottleneck for complex compositional reasoning. In addition, diverse metrics have been introduced as evaluation protocols for hierarchical segmentation. For instance, PartPQ (part panoptic quality) 5 extends standard panoptic quality to assess segmentation performance across a strict two-level object-part hierarchy, while HPQ (hierarchical panoptic quality) 31 broadens this to evaluate fixed, multi-tier semantic hierarchies. Despite their utility, both metrics systematically fail when applied to unconstrained-granularity, open-vocabulary structures. Consequently, hierarchical segmentation models have traditionally mirrored these structural constraints. While they support the prediction of basic object-part relations 16 or even more granular hierarchies 15 , 33 , 14 , 31 , they inherently operate within predefined semantic taxonomies or structural depths. 2.2 Hierarchical Vision-Language Segmentation With recent advance of LVLMs (Large Vision-Language Models) 18 , the paradigm of image segmentation has increasingly shifted towards open-vocabulary, instruction-driven parsing.

Read the complete text