IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals

Markus Gross, Aya Fahmy, Danit Niwattananan, Dominik Muhle, Rui Song, Daniel Cremers, Henri Meeß · 2025

Read IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals by Markus Gross, Aya Fahmy, Danit Niwattananan, Dominik Muhle, Rui Song, Daniel Cremers, Henri Meeß (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20671.txt.

Introduction

Filling a full three-dimensional scene from camera views, object by object, is a fresh task. Markus Gross and co-authors give it a first method built on scene-aware object guesses, in work from twenty twenty-five. Older systems start from fixed learned queries that never adapt to the room at hand. Their method seeds queries from the image itself.

Attention stages then refine each guess against the voxel map. Scores topped the field on home data and held up on unseen data far better than rivals. Run time fell by more than fourteen-fold. Robot navigation is the aimed use, where a camera-only stack must know both the room and its things.

Catalog record

CREATORMarkus Gross, Aya Fahmy, Danit Niwattananan, Dominik Muhle, Rui Song, Daniel Cremers, Henri Meeß
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals Markus Gross, Aya Fahmy, Danit Niwattananan, Dominik Muhle, Rui Song, Daniel Cremers, Henri Meeß Abstract Semantic Scene Completion (SSC) has emerged as a pivotal approach for jointly learning scene geometry and semantics, enabling downstream applications such as navigation in mobile robotics. The recent generalization to Panoptic Scene Completion (PSC) advances the SSC domain by integrating instance-level information, thereby enhancing object-level sensitivity in scene understanding. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent Transformer-based SSC approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To overcome these limitations, we propose IPFormer, the first approach that leverages context-adaptive instance proposals at train and test time to address vision-based 3D Panoptic Scene Completion. Specifically, IPFormer adaptively initializes these queries as panoptic instance proposals derived from image context and further refines them through attention-based encoding and decoding to reason about semantic instance-voxel relationships. Experimental results show that our approach surpasses state-of-the-art methods in overall panoptic metrics PQ † and PQ-All, matches performance in individual metrics, and achieves a runtime reduction exceeding 14 × \times . Furthermore, our ablation studies reveal that dynamically deriving instance proposals from image context, as opposed to random initialization, leads to a 3.62 % 3.62\text{\,}\% increase in PQ-All and a remarkable average improvement of 18.65 % 18.65\text{\,}\% in combined Thing-metrics. These results underscore the effectiveness of IPFormer and highlight its introduction of context-adaptive instance proposals as a pioneering effort in addressing vision-based 3D Panoptic Scene Completion. 1 Introduction Figure 1: Comparison of query initialization for Panoptic Scene Completion (PSC). Existing methods, e.g. Symphonies symphonies , randomly initialize instance queries and incorporate context-awareness during training. However, these queries retain their static nature at test time, as they are shared across all inputs. Our method IPFormer initializes them as instance proposals , which preserve context-adaptivity at test time, effectively aggregating directed features for improved PSC performance. Due to the absence of established baselines for vision-based PSC, we apply DBSCAN ester1996_dbscan to Symphonies’ SSC output to retrieve its individual instances. Panoptic scene completion provides a holistic scene understanding that can serve applications like autonomous driving and robotics by reconstructing volumetric geometry from sparse sensor data and assigning meaning to objects in the scene. Such holistic scene understanding requires both accurate geometry and semantic information about a scene. Historically, these tasks have been treated separately and evolved independently due to the distinct nature of geometric reconstruction and semantic interpretation, each requiring specialized algorithms and data representations. Despite the inherent interconnection between geometry and semantics, their separation hindered unified scene understanding. Classical structure-from-motion evolved into simultaneous localization and mapping (SLAM) systems for accurate reconstructions mur2015orb ; engel2014lsd ; koestler2022tandem , while deep learning facilitated monocular depth estimation (MDE) methods godard2017unsupervised ; godard2019digging ; wimbauer2021monorec , which are restricted to visible surfaces. In contrast, scene completion refers to the process of inferring the complete 3D structure of a scene, including both visible and occluded regions, from partial observations wimbauer2023behind ; han2024boosting . Apart from geometry, semantic image segmentation divides an image into semantically meaningful regions and labels them yu2015multi ; chen2017deeplab . Panoptic segmentation kirillov2019panoptic ; kirillov2019panoptic2 ; porzi2019seamless extends this approach by segmenting instances hariharan2014simultaneous for things (countable objects like person or car) while maintaining semantic segmentation for stuff (amorphous regions like road or vegetation). This unified approach is vital for comprehensive scene understanding, as it distinguishes individual object instances while preserving semantic context for background elements, enabling applications like autonomous navigation. The recent trend in scene understanding has been to unify these tasks into semantic scene completion (SSC) hayler2024s4c and panoptic scene completion (PSC) pasco . SSC integrates geometric completion with semantic labeling, predicting both the 3D structure and semantic categories, represented as a 3D grid composed of voxels. PSC builds on this by adding instance-level segmentation for each voxel, thus distinguishing between individual objects. While PSC was introduced using LiDAR modality, methods based on camera images remain largely unexplored. Moreover, recent transformer-based SSC approaches utilize a fixed set of learned queries to reconstruct objects within the scene volume. Although these queries are typically updated with image context during training, they remain static at test time, limiting their ability to dynamically adapt specifically to the observed scene. To address these limitations, we introduce IPFormer, a 3D vision transformer designed to address panoptic scene completion. IPFormer lifts contextual 2D image features to 3D and samples them based on visibility to initialize instance proposals, each implicitly representing semantic instances within the camera view. Based on this context-adaptive initialization, we establish a robust reconstruction signal that enhances subsequent encoding and decoding stages to reconstruct a complete panoptic scene. Sampling instance proposals solely from visible surfaces is grounded in the principle that all potentially identifiable objects within the observed scene must exhibit a visual cue in the camera image to facilitate their completion. This approach of directed feature aggregation at train and test time significantly improves instance identification, semantic classification, and geometric completion. Our contributions can be summarized as follows: • We present IPFormer, the first approach that leverages context-adaptive instance proposals at train and test time to address vision-based 3D panoptic scene completion. • We introduce a visibility-based sampling strategy, which initializes instance proposals that dynamically adapt to scene characteristics, improving PQ-All by 3.62\text{\,}\% and Thing-metrics on average by 18.65\text{\,}\% , compared to non-adaptive initialization. • Our experimental results demonstrate that IPFormer surpasses state-of-the-art methods in overall panoptic metrics PQ † ( 14.45 ) and PQ-All ( 6.30 ), performs on par in individual metrics, and offers a runtime reduction of over 14 \times , from 4.51 seconds to 0.33 seconds. • Comprehensive ablation studies reveal that employing a dual-head architecture combined with a two-stage training strategy, in which SSC and PSC tasks are trained independently and sequentially, significantly improves three out of four PSC metrics in the All-category. 2 Related work Semantic Scene Completion. The task of semantic scene completion has seen a large amount of interest after it was first introduced in lmscnet . The initial work focused on solving the task for indoor scenes from camera data zhang2018efficient ; zhang2019cascaded ; liu2018see ; li2019depth ; li2019rgbd ; li2020anisotropic ; chen20203d ; cai2021semantic and used point clouds in outdoor scenarios with LiDAR-based methods roldao20223d ; cheng2021s3cnet ; yan2021sparse ; li2021semisupervised ; rist2021semantic . In the following, we will discuss the most relevant contributions for semantic scene completion in the context of autonomous driving. For a more thorough overview, we refer the reader to the surveys of xu2025survey and zhang2024vision . Recent progress in perception for autonomous driving has increasingly adopted vision transformer architectures dosovitskiyimage , notably DETR carion2020end . DETR introduced learnable object queries—trainable embeddings that allow the model to directly predict object sets in parallel, eliminating hand-crafted components like anchor boxes. In SSC, learnable queries can be adapted to predict semantic labels for 3D scene elements ( e.g. , voxels), streamlining the process and enhancing efficiency. Architectures like OccFormer occformer and

Read the complete text