ROODI: R econstructing O ccluded O bjects with D enoising I npainters

Yeonjin Chang, Erqun Dong, Seunghyeon Seo, Nojun Kwak, Kwang Moo Yi · 2025

Read ROODI: R econstructing O ccluded O bjects with D enoising I npainters by Yeonjin Chang, Erqun Dong, Seunghyeon Seo, Nojun Kwak, Kwang Moo Yi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10256.txt.

Introduction

Yeonjin Chang, Erqun Dong, Seunghyeon Seo, Nojun Kwak, and Kwang Moo Yi put forth ROODI in 2025, a way to lift single objects out of 3D scenes. New view tools can rebuild a scene as a cloud of soft blobs, yet pulling one object free is hard when other things hide parts of it. Their method rests on two moves: cut away blobs that do not belong, and paint in the parts that cover has hidden.

For the cut, blobs far from their nearest peers, once true size is weighed, are dropped as strays. For the paint-in, a stock image maker fills gaps with the full scene as guide. Tests on real and made-up sets beat the prior best. The pair of moves works as a team, the team stresses.

Catalog record

CREATORYeonjin Chang, Erqun Dong, Seunghyeon Seo, Nojun Kwak, Kwang Moo Yi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ROODI: R econstructing O ccluded O bjects with D enoising I npainters Yeonjin Chang, Erqun Dong, Seunghyeon Seo, Nojun Kwak, Kwang Moo Yi Abstract While the quality of novel-view images has improved dramatically with 3D Gaussian Splatting, extracting specific objects from scenes remains challenging. Isolating individual 3D Gaussian primitives for each object and handling occlusions in scenes remains far from being solved. We propose a novel object extraction method based on two key principles: (1) object-centric reconstruction through removal of irrelevant primitives; and (2) leveraging generative inpainting to compensate for missing observations caused by occlusions. For pruning, we propose to remove irrelevant Gaussians by looking into how close they are to its K-nearest neighbors and removing those that are statistical outliers. Importantly, these distances must take into account the actual spatial extent they cover—we thus propose to use Wasserstein distances. For inpainting, we employ an off-the-shelf diffusion-based inpainter combined with occlusion reasoning, utilizing the 3D representation of the entire scene. Our findings highlight the crucial synergy between proper pruning and inpainting, both of which significantly enhance extraction performance. We evaluate our method on a standard real-world dataset and introduce a synthetic dataset for quantitative analysis. Our approach outperforms the state-of-the-art, demonstrating its effectiveness in object extraction from complex scenes. 1 Seoul National University, 2 University of British Columbia 1 Introduction 3D scenes are typically composed of numerous objects. Naturally, beyond the original 3D Gaussian Splatting (3DGS) ( Kerbl et al. 2023 ) , several works attempt to interpret 3D scenes at the object level to better capture their semantic structure ( Ye et al. 2024 ; Choi et al. 2024 ; Cen et al. 2025 ; Chacko et al. 2025 ; Zhou et al. 2024 ; Ying et al. 2024 ) . These studies primarily train semantic features of 3D scenes by leveraging semantic maps from 2D segmentation models, modeling the overall structure and semantics of the scene. However, extracting the 3DGS model of target objects with these semantic features introduces additional challenges—precise separation of the target object from the rest of the scene is difficult and (self-)occlusions further make this problem harder. As shown in Fig. 1 , none of the existing methods yield a clearly extracted target model and have so-called floaters . Figure 1: Qualitative highlight – We propose a method to reconstruct specific objects in complex scenes from posed images, even if they are occluded. To reduce floaters, we prune Gaussians that are not of the object of interest, and use an off-the-shelf denoising diffusion inpainter together with occlusion reasoning. Our method provides significantly improved results compared to the state of the art. In this work, we propose a novel method for 3D object extraction, based on three key ideas: (1) regardless of the choice of which object, scene geometry must remain the same—the full scene geometry serves as base knowledge for object extraction, (2) objects are local —we are interested in a single cluster of primitives that form a single object, and (3) occlusions can easily be reasoned by comparing the primitives of the target object with those of the full scene. Based on (1), we adopt a two-stage pipeline that trains the scene geometry first, followed by additional training of object features to distinguish objects. Based on (2), we look into the local structure of the primitives and prune those that are not part of the target object. We propose to use the distance of each Gaussian to their nearest neighbors and remove those that are statistical outliers. As Gaussians are entities that cover a region in 3D space, we consider the distances in terms of Wasserstein distances—we show later in Sec. 4.3 that this is important for proper pruning. From (3), we use a pretrained diffusion model to inpaint the occluded regions of the target object. This leads to clear object extraction, one that is robust to occlusions and floaters; see Fig. 1 . In more detail, we first train a 3DGS model of the scene, then extract the 3DGS model of the target objects in the second stage. To extract an object, we start with Segment Anything Model 2 (SAM2) ( Kirillov et al. 2023 ; Ravi et al. 2024 ) to obtain the segmentation map of the object of interest. We then distill this map into the 3DGS model by briefly fine-tuning the model for 500 iterations. We then update 3D Gaussians specific to the target object by: (1) pruning non-object Gaussians by removing those that are statistical outliers when looking into their Wasserstein distances to nearest neighbors; (2) rendering both the target object and the full scene to identify occluded regions of the target object; and (3) inpainting the occluded regions of the target object with a pretrained inpainting diffusion model ( Podell et al. 2023 ; The Diffusers team 2023 ) ; and finally (4) updating the parameters of the 3D Gaussians to better match the inpainted target object. Notably, all of these steps are crucial, as without pruning floaters, the diffusion model is unable to recognize or inpaint the target object due to floaters, and inpainting is further critical to obtain clear, sharp images—we show empirically that simple general diffusion-based enhancement is insufficient. To evaluate our method, we conduct experiments on both synthetic and real-world datasets. We use the standard LERF dataset ( Kerr et al. 2023 ) to allow qualitative comparison with existing methods. As the real-world dataset does not have a ground-truth object-centric extraction (e.g., parts of object can be occluded), we create a new synthetic dataset, named MultiObjectBlender . We render images from a 3D scene composed of many objects from ShapeNet ( Chang et al. 2015 ) using Blender ( Blender Online Community 2024 ) , where many objects are occluding each other. We then render both the full scene and only the target object, the latter of which can be used to quantitatively evaluate the extraction quality. Our method significantly outperforms existing methods on both datasets. Contributions. To summarize, in this work: • we propose a method to extract objects within a 3DGS model that prunes non-object Gaussians through statistical outlier removal, and performs occlusion reasoning and inpainting; • we propose to use Wasserstein distances when measuring distances between Gaussians for this purpose; • we demonstrate that the two-stage pipeline, the removal of non-object Gaussians, and the inpainting of occluded regions are all essential; • we create a new synthetic dataset to evaluate the performance of object extraction methods; 1 1 1 We will make this dataset publicly available. • we show that our method significantly outperforms existing methods on both synthetic and real-world datasets. 2 Related Work 3D reconstruction. Research on 3D scene reconstruction from multi-view images has gained momentum since NeRF ( Mildenhall et al. 2020 ) , which introduced an implicit representation of 3D space using neural networks, enabling photo-realistic novel view synthesis. However, NeRF suffers from long training and inference times, which led to subsequent research incorporating voxel grids ( Yu et al. 2021 ; Fridovich-Keil et al. 2022 ; Hu et al. 2022 ; Sun, Sun, and Chen 2022 ) , tri-planes ( Khatib and Giryes 2024 ; Fridovich-Keil et al. 2023 ; Chen et al. 2022 )

Read the complete text