Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation
Jiahao Lu, Jiacheng Deng, Tianzhu Zhang · 2025
Read Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation by Jiahao Lu, Jiacheng Deng, Tianzhu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04139.txt.
Introduction
Jiahao Lu, Jiacheng Deng, and Tianzhu Zhang improve 3D instance segmentation in this 2025 paper. The task is to pick out each object in a 3D scene and label it, and transformer methods lead the field but lose objects as their decoder layers deepen.
Their system, BFL, attacks two weak points. A new initialization module builds queries that hold both position and content well. A fusion decoder then keeps low overlap queries alive, protecting recall in deep layers. Tests on four standard scene datasets, including ScanNet and S3DIS, showed top results. The design lesson is to protect weak candidates instead of discarding them early.
Catalog record
| CREATOR | Jiahao Lu, Jiacheng Deng, Tianzhu Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04139
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation Jiahao Lu, Jiacheng Deng, Tianzhu Zhang Abstract 3D instance segmentation aims to predict a set of object instances in a scene and represent them as binary foreground masks with corresponding semantic labels. Currently, transformer-based methods are gaining increasing attention due to their elegant pipelines, reduced manual selection of geometric properties, and superior performance. However, transformer-based methods fail to simultaneously maintain strong position and content information during query initialization. Additionally, due to supervision at each decoder layer, there exists a phenomenon of object disappearance with the deepening of layers. To overcome these hurdles, we introduce Beyond the Final Layer: Hierarchical Query Fusion Transformer with Agent-Interpolation Initialization for 3D Instance Segmentation (BFL). Specifically, an Agent-Interpolation Initialization Module is designed to generate resilient queries capable of achieving a balance between foreground coverage and content learning. Additionally, a Hierarchical Query Fusion Decoder is designed to retain low overlap queries, mitigating the decrease in recall with the deepening of layers. Extensive experiments on ScanNetV2, ScanNet200, ScanNet++ and S3DIS datasets demonstrate the superior performance of BFL. 1 Introduction Indoor instance segmentation is one of the fundamental tasks in 3D scene understanding, aiming to predict masks and categories for each foreground object. With the increasing popularity of AR/VR Park et al. 2020 ; Manni et al. 2021 , 3D indoor scanning Lehtola et al. 2017 ; Halber et al. 2019 , 3D/4D reconstruction Wu et al. 2024 ; Zhu et al. 2024 ; Lu et al. 2024b ; Luiten et al. 2023 ; Lu et al. 2024c ; Zhang et al. 2024 , and autonomous driving Neven et al. 2018 ; Yurtsever et al. 2020 , 3D instance segmentation has become a pivotal technology enabling scene understanding. However, the complexity of scenes and the diversity of object categories pose significant challenges to 3D instance segmentation. To address the aforementioned challenges, a series of 3D instance segmentation methods Yi et al. 2019 ; Hou et al. 2019 ; Yang et al. 2019 ; Engelmann et al. 2020 ; Liu et al. 2020 ; Chen et al. 2021b ; Liang et al. 2021 ; Vu et al. 2022 ; Schult et al. 2022 ; Sun et al. 2023 ; Lu et al. 2023 ; Lai et al. 2023 have been proposed. Generally, these methods can be categorized into three groups: proposal-based Yi et al. 2019 ; Hou et al. 2019 ; Yang et al. 2019 , grouping-based Engelmann et al. 2020 ; Liu et al. 2020 ; Jiang et al. 2020b ; Chen et al. 2021b ; Liang et al. 2021 ; Vu et al. 2022 , and transformer-based Schult et al. 2022 ; Sun et al. 2023 ; Lu et al. 2023 ; Lai et al. 2023 . Proposal-based methods adopt a top-down approach, where they first extract 3D bounding boxes and then utilize a mask learning branch to predict the object mask within each box. Grouping-based methods initially generate predictions for each point (e.g., semantic categories and geometric offsets) and then generate instance proposals. Recently, transformer-based methods have attracted researchers’ attention due to their elegant pipelines, reduced manual selection of geometric properties, and superior performance. These methods typically initialize a fixed number of object queries, which are then fed into the decoder to aggregate scene features. After the feature aggregation of each decoder layer, the queries output instance predictions, with each layer’s predictions supervised by the ground truth. We refer to this design as per-layer auxiliary loss. The predictions from the final layer are used as the final output. In this process, query initialization plays a crucial role. Current transformer-based methods propose various designs for query initialization, mainly categorized into FPS-based (farthest point sampling) Schult et al. 2022 ; Lu et al. 2023 and learnable-based Sun et al. 2023 ; Lai et al. 2023 approaches. Furthermore, inspired by 2D instance segmentation Cheng et al. 2022 ; Li et al. 2023 ; Jain et al. 2023 , the design of per-layer auxiliary loss has significantly improved the training effectiveness of 3D instance segmentation. However, we observe a phenomenon of Object Disappearance , where predictions for certain objects vanish as the deepening of layers. As shown in Figure 1 , where the object “picture” obtained from the prediction at layer 4 disappears in layer 5 and layer 6. This is reflected in a decrease in recall in the quantized results, as shown in Figure 2 (b), contradicting the intuition that more interactions between features lead to better results. Figure 1: The phenomenon of Object Disappearance with the deepening of layers. Figure 2: (a) The comparison of different query initialization methods. The FPS-based methods conduct farthest point sampling separately for each scene, placing more emphasis on positional information but lacking in aggregating content information. The learnable-based methods initialize a fixed number of queries for aggregating content information across all scenes, which is prone to empty sampling, thereby compromising foreground coverage. Our method leverages the advantages of both approaches to achieve a balanced and comprehensive solution. (b) The recall difference. The recall of the baseline shows instability during the iterative optimization process across layers, whereas our method, with the assistance of the Hierarchical Query Fusion Decoder, demonstrates a steady improvement in recall across each layer. Based on the discussion above, we have identified two challenges that need to be addressed: 1) How to better initialize queries? As illustrated in Figure 2 (a), current transformer-based methods Schult et al. 2022 ; Sun et al. 2023 ; Lu et al. 2023 ; Lai et al. 2023 can mainly be categorized into FPS-based Schult et al. 2022 ; Lu et al. 2023 and learnable-based approaches Sun et al. 2023 ; Lai et al. 2023 . Mask3D Schult et al. 2022 and QueryFormer Lu et al. 2023 utilize FPS to obtain the initialization distribution of queries, which can more likely distribute candidates to the region where objects are located, thus reducing the empty sampling rate. However, these FPS-based approaches fail to learn content embedding across scenes effectively for feature aggregation. On the other hand, SPFormer Sun et al. 2023 and Maft Lai et al. 2023 employ learnable queries, which can update and learn across multiple scenes in the dataset. Nevertheless, the empty sampling rate is higher, leading to a decrease in model recall. Therefore, balancing the sampling positions of candidates and learning content embedding effectively is crucial for initializing queries. 2) How to mitigate the issue of inter-layer recall decline? During the decoding phase, due to the existence of auxiliary loss, the predictions of each decoder layer are supervised by ground truth. For instances that are difficult to predict, such as pictures, bookshelfs, the quality of the mask corresponding to the matched query is poor. Consequently, the mask attention Schult et al. 2022 ; Sun et al. 2023 focuses on a large amount of noisy features, causing the optimization direction of the query to be unstable, and there is a possibility of further deterioration in mask quality. Moreover, for other unmatched queries, due to the lack of supervision signal, the optimization direction is even more random. Predicting better quality for such difficult-to-predict instances is therefore more challenging. As a result,