Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description
Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi Wang, Luc Van Gool, Danda Pani Paudel · 2024
Read Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description by Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi Wang, Luc Van Gool, Danda Pani Paudel (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01398.txt.
Introduction
Anna-Maria Halacheva and co-authors bank a set of rooms whose parts can move in a 2024 paper. A chair that folds, a door that swings: such things fill homes, yet scene lore has read rooms as still sets of stiff goods.
Their bank holds near three hundred rooms, dressed by adepts with eight kinds of mark on each joint good: its parts, its swing, its span. All sits in one shared dress for scene trade. A new net reads a room and calls parts and swing at once, and beat split ways on this bank and two more. The bank then served a scene edit by word cue, and a robot's school for joint goods. The authors give bank, test, and code free.
Catalog record
| CREATOR | Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi Wang, Luc Van Gool, Danda Pani Paudel |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01398
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi Wang, Luc Van Gool, Danda Pani Paudel Abstract Figure 1 : The Articulate3D Dataset features (a) articulation annotations with motion types, origin, axis and range; (b) high-detail segmentations at object and part levels; (c) part connectivity graphs; (d) part articulation roles enabling interaction understanding: movable (articulated) parts, their corresponding interactable parts, and fixed parts. Articulate3D is the first large-scale real-world dataset of simulation-ready indoor scenes. 1 1 footnotetext: Equal contribution 1 USDNet We propose USDNet, a unified framework of simultaneously predicting all necessary components for the construction of an interaction-oriented USD (movable and interactable parts, and motion parameters) given the point cloud of a scene. Given a scene point cloud as input, USDNet predicts (1) instance masks M=\{m_{k}\}^{K}_{k=1} of movable parts and their motion type C_{M}=\{c_{k}\in\{\text{background, rotation, translation}\}\} ; (2) instance masks I=\{i_{k}\}^{L}_{k=1} of interactable parts and their motion type label t_{k}\in\{\text{ background, rotation, translation}\}\} ; (3) motion parameters per movable part m_{k} , including motion origin \{o_{k}\in\mathbb{R}^{3}\} and axis \{a_{k}\in\mathbb{R}^{3}\} . Beyond these tasks, we can also predict connectivity graphs as a separate task based on Articulate3D. This can be achieved using spatial proximity calculations or learning-based methods (Sec. ). 1.1 Network Architecture The overall architecture is shown in Fig. . USDNet extends a Mask3D [ 53 ] backbone with a dense-prediction mechanism to leverage point features for part segmentation and articulation prediction. Part Segmentation and Motion Type Prediction . Following the implementaion in Mask3D, 3D sparse convolution [ 8 ] is applied to extract the point-wise features. Given the point features and initial queries, stacked transformer decoder layers are applied to generate masks of both movable and interactable instances and the instance queries. The motion types are predicted with MLP layers on instance queries. Considering small movable and interactable parts (e.g. switches and buttons) in Articulate3D , we apply the coarse-to-fine learning strategy from [ 12 ] during training. Noticeably, we propose an auxiliary task in which the spatial vector pointing to the part center is predicted for each point of the interactable part, which can speed up the convergence and improve segmentation accuracy. Motion Parameter Prediction. Given the fact that the motion parameter is closely related to the geometry and semantic and spatial attributes of the corresponding movable part, we integrate query-based prediction with point-wise dense prediction to better capture those attributes for parameter prediction. As illustrated by the red blocks in Fig. , the masked point features of the movable parts are passed through the ”Point-wise Axis and Origin Branch” formed by MLPs to produce point-wise predictions of the axis and origins motion parameters. These predictions are then averaged, with the mean value used for subsequent processing. On the other hand, movable part queries also go through the MLPs (yellow block in the figure) to produce an axis prediction. This prediction is averaged with the point features-based mean prediction. By doing so, both local features (point features) and global context (queries) are leveraged for the motion parameter prediction. USDNet differs from the method proposed in [ 12 ] in two key aspects: 1) USDNet simultaneously predicts movable and interactable parts and motion paramter; and 2) we introduce a point-wise dense prediction mechanism (instead of a query-based prediction), which we illustrate below. 1.2 Implementation Details Losses. For part segmentation, we apply dice loss and binary cross-entropy loss for mask generation: L_{seg}=\lambda_{dice}L_{dice}+\lambda_{ce}L_{ce} . Additional grouping loss is applied for auxiliary task of interactable part grouping: L_{aux}=\lambda_{aux}\cdot\sum_{p\in\mathbf{i_{k}}}|\mathbf{v_{p}^{*}}-\mathbf{v_{p}}| , where \mathbf{v_{p}^{*}} and \mathbf{v_{p}} are ground truth and predicted point-to-part-center vector of point p belonging to interactable part i_{k} . For motion type prediction, we apply cross-entropy loss \lambda_{cls}L_{cls} . For motion parameter prediction, we apply category-specific articulation loss. For translation motion type, we consider the axis loss, formulated as the cos-dissimilarity between the predicted axis and the ground truth: L_{arti}=\lambda_{arti}(1-\cos{<\mathbf{a_{k}},\mathbf{a^{*}_{k}}>}) . For rotation motion type, we also calculate the distance of the predicted origin to the ground truth axis: L_{arti}=\lambda_{arti}(1-\cos{<\mathbf{a_{k}},\mathbf{a^{*}_{k}}>})+\lambda_{arti}\|\mathbf{a^{*}_{k}}\times(\mathbf{o_{k}}-\mathbf{o^{*}_{k}})\| . Training. We weight losses during training L=L_{seg}+L_{cls}+L_{aux}+L_{arti} with \lambda_{dice}=2.0,\lambda_{ce}=5.0,\lambda_{cls}=2.0,\lambda_{aux}=1.0,\lambda_{arti}=1.0 . USDNet is firstly trained for 1160 epochs on part segmentation tasks and then further trained on joint prediction of part segmentation and motion parameters for 680 epochs. The training batch size is 1, on 1 NVIDIA A100-40g GPU and the learning rate is 0.0001. The input point cloud is cropped with cuboid of 6\times 6\;m^{2} to save memory during training. 2 Experiments We compare USDNet against baselines on Articulate3D. We also evaluate USDNet on existing datasets [ 12 , 35 ] on the task of part segmentation and motion parameter prediction. More implementation details can be found in Sec. . 2.1 Baselines As representative approaches for the task of semantic-instasnce segmentation, Softgroup [ 65 ] and Mask3D [ 53 ] are adapted as baselines in our evaluation. We denote these adapted baselines as SoftGroup † and Mask3D † . For fair comparison, losses of the baselines are the same as USDNet. SoftGroup † re-uses the network of SoftGroup for instance segmentation of movable and interactable parts. Origin and axis prediction branches are implemented to aggregate per-part backbone features for motion parameter prediction. Mask3D † . We apply Mask3D for movable and interactable part segmentation with its original 3D instance segmentation framework. For articulation parameter prediction, inspired by OPDMulti [ 58 ] and SceneFun3D [ 12 ] , we add an additional head that maps the per-part instance queries to motion parameter predictions. 2.2 Movable Part Segmentation Evaluation Metric . To evaluate the 3D part segmentation and motion type classification accuracy, we report the mean average precision (AP) and AP with IoU thresholds of 0.25 ( AP_{25} ) and 0.5 ( AP_{50} ). Results . The performance of the benchmarked methods, reported in Tab. 1 , is the highest for USDNet both for AP_{50} and AP_{25} . SoftGroup † shows the lowest AP_{50} and AP_{25} but the highest AP , due to the IoU prediction branch in [ 65 ] , which filters out false positive instance proposals. In contrast, Mask3D-based methods are prone to over-confident predictions, as instance confidence is directly estimated from category and mask scores. Method AP AP_{50} AP_{25} SoftGroup † 22.7 32.7 37.2 Mask3D † (baseline) 18.1 39.1 58.9 USDNet (ours) 19.8 41.8 59.9 Table 1: Segmentation of movable part on Articulate3D. 2.3 Interactable Part Segmentation Evaluation Metric . Similarly, we report AP_{25} and AP_{50} for measuring accuracy of interactable part segmentation. Results . Tab. 2 shows that USDNet performs better than Mask3D † , illustrating the effectiveness of the auxiliary task of the point-center-shift prediction as shown in Fig. . It is noticeable that SoftGroup † performs inferior to USDNet and Mask3D † , which is in accordance with the benchmarks of [ 9 , 78 , 12 ] . Method AP AP_{50} AP_{25} SoftGroup † 6.8 14.5 25.4 Mask3D † (baseline) 12.7 30.2 55.6 USDNet (ours) 12.7 31.1 55.9 Table 2: Segmentation of interactable part on Articulate3D. 2.4 Motion Parameter Prediction Evaluation Metric . Inspired by [ 12 ] , we extend AP_{50} with our own articulation loss to evaluate the accuracy of motion parameter prediction given the part segmentation prediction. As shown in Table 3 ,