3D Part Segmentation via Geometric Aggregation of 2D Visual Features
Marco Garosi University of Trento [email protected] Riccardo Tedoldi University of Trento [email protected] Davide Boscaini Fondazione Bruno Kessler [email protected] and Massimiliano Mancini University of Trento [email protected] Nicu Sebe University of Trento [email protected] · 2024
Read 3D Part Segmentation via Geometric Aggregation of 2D Visual Features by Marco Garosi University of Trento [email protected] Riccardo Tedoldi University of Trento [email protected] Davide Boscaini Fondazione Bruno Kessler [email protected] and Massimiliano Mancini University of Trento [email protected] Nicu Sebe University of Trento [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04247.txt.
Introduction
Marco Garosi, Riccardo Tedoldi, Davide Boscaini, Massimiliano Mancini, Nicu Sebe, and Fabio Poiesi wrote a 2024 computer vision paper on splitting 3D objects into their parts without training on those objects. Supervised models only know the fixed part lists they were trained on. Vision language models offer an open ended alternative, but they demand fiddly prompts and ignore 3D shape.
Their system, COPS, renders a point cloud from many viewpoints and pulls features from the images. It then projects those features back onto the 3D points and merges them with a geometry aware procedure that keeps neighbors consistent. Points are clustered into parts and labeled. Across five datasets, COPS set the state of the art for zero shot part segmentation while staying efficient enough to scale.
Catalog record
| CREATOR | Marco Garosi University of Trento [email protected] Riccardo Tedoldi University of Trento [email protected] Davide Boscaini Fondazione Bruno Kessler [email protected] and Massimiliano Mancini University of Trento [email protected] Nicu Sebe University of Trento [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D Part Segmentation via Geometric Aggregation of 2D Visual Features Marco Garosi University of Trento [email protected] Riccardo Tedoldi University of Trento [email protected] Davide Boscaini Fondazione Bruno Kessler [email protected] and Massimiliano Mancini University of Trento [email protected] Nicu Sebe University of Trento [email protected] Fabio Poiesi Fondazione Bruno Kessler [email protected] Abstract Supervised 3D part segmentation models are tailored for a fixed set of objects and parts, limiting their transferability to open-set, real-world scenarios. Recent works have explored vision-language models (VLMs) as a promising alternative, using multi-view rendering and textual prompting to identify object parts. However, naively applying VLMs in this context introduces several drawbacks, such as the need for meticulous prompt engineering, and fails to leverage the 3D geometric structure of objects. To address these limitations, we propose COPS, a COmprehensive model for Parts Segmentation that blends the semantics extracted from visual concepts and 3D geometry to effectively identify object parts. COPS renders a point cloud from multiple viewpoints, extracts 2D features, projects them back to 3D, and uses a novel geometric-aware feature aggregation procedure to ensure spatial and semantic consistency. Finally, it clusters points into parts and labels them. We demonstrate that COPS is efficient, scalable, and achieves zero-shot state-of-the-art performance across five datasets, covering synthetic and real-world data, texture-less and coloured objects, as well as rigid and non-rigid shapes. The code is available at https://3d-cops.github.io . 1 Introduction PointCLIPv2 [ ] Original prompts mIoU = 18.7 GPT-generated mIoU = 14.5 Template-based mIoU = 9.9 Part names only mIoU = 5.1 COPS (ours) CLIP visual features mIoU = 24.3 DINOv2 features mIoU = 41.9 Ground-truth Fuselage [formula] Wing [formula] Engine [formula] Tail [formula] Figure 1 : The quality of part descriptions significantly affects the segmentation performance of methods based on vision-language models. For example, the performance of PointCLIPv2 [ 12 ] (left) deteriorates rapidly when replacing the default textual prompt with a GPT-generated description, the template “This is a depth image of an airplane’s [part]”, or simply using part names. In contrast, our pipeline (center) achieves more accurate segmentations by disentangling part decomposition from part classification. The improvement is evident when using the same CLIP visual features as PointCLIPv2 (top) and becomes even more pronounced when using DINOv2 [ 6 ] features (bottom), the default choice of COPS. COPS generates more uniform segments with sharper boundaries, resulting in higher segmentation quality. Recognising object parts has raised widespread attention in both industry and academia, with applications ranging from robotic manipulation [ 14 , 5 ] to augmented reality [ 8 ] . Understanding and interacting with objects requires identifying their building components. For example, we open a bottle by detecting its cap, or hold a mug by its handle. Currently, well-annotated 3D part segmentation datasets contain only a limited amount of samples in a restricted number of categories. This limits the transferability of models, as their knowledge is narrowed to the closed-set parts and objects seen during training. In contrast, vision-language models (VLMs) can handle objects and parts that were not seen during training, making them more versatile in open-vocabulary settings [ 39 , 23 , 24 ] . In this context, methods such as PointCLIP [ 55 , 12 ] , PartSLIP [ 4 , 57 ] , ZeroPS [ 50 ] , and SATR [ 1 ] leverage VLMs like CLIP [ 39 ] or GLIP [ 24 ] in a two-step procedure to tackle open-vocabulary settings. First, a point cloud is rendered into multiple views, which are then processed by the VLM’s visual encoder to extract visual features. Second, these visual features are projected back into 3D and compared with the text features derived from prompts describing object parts to produce the final segmentation. This approach suffers from two drawbacks: (i) it does not incorporate 3D geometric knowledge, and (ii) its performance depends on the quality of the input prompts ( Fig. 1 ), as some parts can be challenging to describe in natural language. The latter issue can be partially addressed by crafting better textual prompts with the assistance of large language models such as ChatGPT [ 4 , 57 ] , or through prompt ensembles to reduce description ambiguity [ 39 ] . However, the inherent ambiguity of natural language remains a significant challenge, particularly when generalising to unseen objects and parts. In this paper, we blend features from a 2D vision foundation model (VFM) with 3D geometric understanding to detect object parts. COPS follows a five-step procedure that involves: (i) rendering the object from multiple views that cover a large as possible portion of the object, (ii) extracting pixel-level features using DINOv2 [ 6 ] and upsampling its output, (iii) back-projecting these features onto the points, (iv) aggregating point-level features from multiple views using the geometric feature aggregation (GFA) module, and (v) segmenting object parts through unsupervised clustering of point-level features, followed by zero-shot classification. For tasks requiring only part decomposition without knowledge of part categories, the zero-shot classifier of COPS can be disabled to reduce computational costs and avoid inaccuracies caused by natural language ambiguities [ 43 , 26 ] . In summary, our contributions are: • A novel training-free method for 3D part segmentation, which disentangles part segmentation from semantic labelling; • A geometric feature aggregation module that fuses 3D structural information with semantic knowledge from VFMs; • A thorough evaluation on five benchmarks, establishing a unified evaluation that future zero-shot part segmentation methods can use for comparison. 2 Related works We identify two main approaches to 3D point cloud part segmentation: using networks specifically designed to handle 3D data natively and adapting 2D VLMs to the 3D task. 3D networks for point cloud part segmentation. A general approach to point cloud part segmentation [ 2 , 11 , 5 , 9 ] is to leverage point-based methods [ 36 , 37 ] that extract a semantic representation for each point. These works focus either on supervised or self-supervised methods [ 3 , 25 , 29 , 46 , 17 , 4 ] , and they produce feature representations that transfer well to the part segmentation task [ 36 , 37 , 40 , 41 ] . Other works focus directly on part segmentation. They group points belonging to the same part by leveraging different strategies. The most common techniques are prototype-based, modelling and matching shape parts [ 18 , 53 , 56 , 38 ] , grouping [ 49 , 19 , 47 , 54 , 27 ] , and directly training for point clustering. However, datasets with 3D annotated shapes are scarce and costly to produce. Therefore, a thread of works explored the use of alternative supervision signals, such as bounding boxes [ 10 ] , assembly manuals [ 48 ] , and reference games [ 22 ] . Similarly, other works introduced unsupervised objectives, such as learning hierarchies [ 32 ] , co-segmentation [ 58 , 51 , 9 ] , abstraction [ 42 , 35 , 34 , 15 , 12 ] . In addition to data scarcity, some methods have to face costly training procedures, which results in limited transfer learning capabilities and robustness to domain gaps and unseen objects. With COPS, we overcome all these limitations by offering a