Tuning-Free Amodal Segmentation via the Occlusion-Free Bias of Inpainting Models
Jae Joong Lee, Bedrich Benes, Raymond A. Yeh · 2025
Read Tuning-Free Amodal Segmentation via the Occlusion-Free Bias of Inpainting Models by Jae Joong Lee, Bedrich Benes, Raymond A. Yeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18947.txt.
Introduction
Jae Joong Lee, Bedrich Benes, and Raymond Yeh present a 2025 method for amodal segmentation, labeling the full shape of objects even where they are hidden. Most methods learn this from hand-labeled masks or synthetic data, which limits variety. The new approach needs no training at all. It borrows image inpainting models, which tend to paint objects whole and unoccluded when filling gaps.
The pipeline inpaints the hidden parts of an object, then segments the completed picture. Across five data sets, the masks beat the best prior methods by about 5 percent on average. The authors stress the method's generality: any pretrained inpainting model can drive it, with no fine-tuning. The trick turns a quirk of generative models into a perception tool.
Catalog record
| CREATOR | Jae Joong Lee, Bedrich Benes, Raymond A. Yeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18947
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tuning-Free Amodal Segmentation via the Occlusion-Free Bias of Inpainting Models Jae Joong Lee, Bedrich Benes, Raymond A. Yeh Abstract Amodal segmentation aims to predict segmentation masks for both the visible and occluded regions of an object. Most existing works formulate this as a supervised learning problem, requiring manually annotated amodal masks or synthetic training data. Consequently, their performance depends on the quality of the datasets, which often lack diversity and scale. This work introduces a tuning-free approach that repurposes pretrained diffusion-based inpainting models for amodal segmentation. Our approach is motivated by the “occlusion-free bias” of inpainting models, i.e., the inpainted objects tend to be complete objects without occlusions. Specifically, we reconstruct the occluded regions of an object via inpainting and then apply segmentation, all without additional training or fine-tuning. Experiments on five datasets demonstrate the generalizability and robustness of our approach. On average, our approach achieves 5.3% more accurate masks over the state-of-the-art. 1 Introduction Amodal segmentation refers to predicting segmentation masks even under occlusions [ 25 ] . This challenging task involves reasoning about the unseen portion of an object under complex occlusion and illumination scenarios. It is an important problem with potential applications in autonomous driving and robot planning, which require reasoning beyond what is directly observed to predict possible future events in the environment [ 51 , 12 , 5 ] . Input Inpainted samples Figure 1: Occlusion-free bias for a diffusion inpainting model. We observe that an inpainted object is always placed without occlusions inside the inpainting area (blue box), e.g . , a tree could have been inpainted behind the fence. Following the success of deep segmentation methods [ 21 , 3 , 30 , 41 ] , amodal segmentation is often formulated as a supervised learning task, i.e . , a dataset of (image, amodal mask) pairs is collected to train a model. However, preparing a large dataset for amodal segmentation is challenging. Annotating amodal masks requires reasoning over occluded reason, which may be difficult and inconsistent among human annotators. Furthermore, scaling the diversity of the dataset is challenging as it requires numerous combinations of occluders and objects. Inevitably, several amodal segmentation methods turns to synthetically generating occlusions [ 49 , 10 , 34 , 1 ] and 3D game engine rendering [ 18 ] to obtain the annotations. However, the performance is still limited by (a) the distribution gap between the synthetic and real data and (b) the size of the dataset. For example, the currently available SOTA [ 34 ] uses only 800k data pairs for training/fine-tuning, which is relatively small compared to the recent internet-scale datasets for other tasks [ 40 , 35 , 36 ] . Another work [ 50 ] also proposes to use a pre-trained diffusion model, but it requires iterative occlusion removal and is constrained to a fixed 83 object categories, which has limited generalizability. To address these challenges, we present a tuning-free approach that utilizes existing foundation models trained on internet-scale datasets. Our method does not require any amodal data. Hence, the method is naturally zero-shot and without restriction to pre-defined object classes. Our approach is motivated by the observation that diffusion inpainting models have an “occlusion-free bias”, i.e . , the inpainting model prefers to generate a whole object rather than the occluder given a reasonable mask as shown in Fig. 1 . We propose to perform inpainting over an enlarged modal mask, where the diffusion model fills the occluded regions. With the inpainted occluded regions, we extract the modal segmentation as the amodal prediction. Figure 2 : Our approach takes two inputs an RGB image {\bm{I}} and a visible mask {\bm{V}} . From {\bm{I}} , we generate a conditioned RGB image with a color distribution-aware background {\bm{x}}_{\tt bck} and a partial Gaussian noise-added object {\bm{x}}_{\tt obj} . From {\bm{V}} , we create a customized inpainting area {\bm{M}} so that we utilize any diffusion-based inpainting models to create an inpainted image \hat{\bm{x}}_{0} to extract amodal mask \hat{\bm{A}} . We demonstrate the effectiveness of our approach on five diverse amodal segmentation datasets, namely, COCO-A [ 62 ] , BSDS-A [ 61 ] , KINS [ 37 ] , FishBowl [ 45 ] , and SAILVOS [ 18 ] . Notably, our tuning-free approach outperforms the current supervised SOTA [ 34 ] by an average of 5.3% in the mIoU. Our contributions are as follows: • We propose a tuning-free method for amodal segmentation (zero-shot) by exploiting the occlusion-free bias of diffusion inpainting models. • The method involves several novel components, including a context-aware approach to background composition using RGB distribution, a noising process image for conditioning, and a modal mask construction procedure. • We demonstrate the generalizability of the proposed method by conducting extensive experiments over four diffusion inpainting models on five diverse datasets. 2 Related Work Amodal perception and segmentation. Humans can often detect and identify an object even if it is (partially) occluded [ 23 ] . Seminal work by Li and Malik [25] begins the line of work of using deep learning for amodal tasks. Many architectures and models have been proposed, e.g . , CNN [ 25 , 60 , 51 , 49 ] , Generative Adversarial Networks [ 8 ] , Transformer [ 46 , 11 ] , and Diffusion-based models [ 34 , 59 ] . Existing works are evaluated on datasets such as COCO-A [ 62 ] , BSDS-A [ 61 ] , KINS [ 37 ] , and MP3D-Amodal [ 59 ] , which consist of common objects from the real world. Other synthetic benchmarks are also popular, e.g . , SAILVOS [ 18 ] or FishBowl [ 45 ] . These synthetic datasets provide more diverse object categories and precise amodal mask annotations without human errors. The current SOTA in amodal segmnetation that involves training are pix2gestalt [ 34 ] and AmodalWild [ 59 ] . We refer to pix2gestalt as the SOTA as its code is ultimately released, whereas AmodalWild did not release the training code. pix2gestalt [ 34 ] , which trains a deep-net to predict the occluded pixels following an analysis by synthesis framework [ 56 ] . Specifically, they created a synthetic amodal dataset by occluding objects with randomly sampled overlays using another object. Our work does not require any amodal datasets, i.e . , it is tuning-free. Next, the closest related to our work is the tuning-free method proposed by Xu et al. [50] , where they propose to iteratively use an inpainting modal for amodal completion, i.e . , they are interested in high image quality. Nonetheless, as a modal mask can be extracted from the completed image, we consider it to be an amodal segmentation method. The key limitation of Xu et al. [50] , upon reviewing their code [ 20 ] , is that the approach leverages class information and is limited to only 83 classes his greatly limits the usability of their approach. In contrast, our method does not have class restrictions and does not require multiple calls to the inpainting model. Image inpainting is the task of filling in missing regions of a given image, where the missing region is indicated using a mask. Early works in