Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding
Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li · 2024
Read Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding by Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00684.txt.
Introduction
Zilin Du, Haoxin Li, Jianfei Yu, and Boyang Li tackle a data shortage in teaching AI to find objects named in text, in a 2024 paper. The task, visual grounding, asks a model to locate the image region a phrase describes. Large labeled sets are hard to build. The authors ask whether made-up data can fill the gap.
Their framework paints new content outside the target box in an image. That avoids a label mismatch that hurt older methods. A filter then picks the most useful training examples, using scores for hardness and overfit risk. Across four benchmarks, the method beat training on real data alone by about six percent on average. It also topped leading rival methods.
Catalog record
| CREATOR | Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding Zilin Du, Haoxin Li, Jianfei Yu, Boyang Li Abstract Visual grounding aims to localize the image regions based on a textual query. Given the difficulty of large-scale data curation, we investigate how to effectively learn visual grounding under data-scarce settings in this paper. To address the data scarcity, we propose a novel framework, POBF (Paint Outside the Box and Filter). POBF synthesizes images by inpainting outside the box, tackling a label misalignment issue encountered in previous works. Furthermore, POBF leverages an innovative filtering scheme to select the most effective training data. This scheme combines a hardness score and an overfitting score, balanced by a penalty term. Extensive experiments across four benchmark datasets demonstrate that POBF consistently improves performance, achieving an average gain of 5.83% over the real-data-only method and outperforming leading baselines by 2.29%–3.85% in accuracy. We validate the robustness and generalizability of POBF across various generative models, training data sizes, and model architectures. keywords Visual Grounding, Cross-Modality Data Generation, Data Augmentation, Image Editing, Data Filtering 1 Introduction Visual grounding Deng et al. (2021) ; Xiao et al. (2024a) ; Ye et al. (2023) ; Yu and Li (2024) ; Su et al. (2024) ; Liu et al. (2025) ; Cao et al. (2025) ; Yi et al. (2024) ; Ma et al. (2024) is a fundamental task in vision-language understanding Li et al. (2025c) ; Li et al. (2025b) . The objective is to locate a specific image region based on a free-form textual query. However, constructing visual grounding datasets with dense region-text annotations is both time-intensive and costly, which hinders the progress in this field. Motivated by the difficulty of large-scale data curation, we investigate the feasibility of effectively learning visual grounding under data-scarce (commonly referred to as few-shot Xue et al. (2024) , data-efficient Chen et al. (2022) , or low-resource Jiang and Zheng (2023) ) settings, where only limited real training data are accessible. Figure 1: Examples of generated images from different methods. X-Paste Zhao et al. (2023) synthesizes unnatural images. Gen2Det Suri et al. (2024) exhibits the label misalignment issue. GeoDiffusion Chen et al. (2024c) either fails to generate the correct object or introduces excessive artifacts. In contrast, our method synthesizes high-quality images with accurately aligned bounding boxes. With recent advances in image generation and captioning Li et al. (2022) ; Podell et al. (2023) ; Rombach et al. (2022) ; Liu et al. (2024) , a natural approach to address data scarcity is to synthesize training data Dunlap et al. (2023) ; Wang et al. (2024d) . These powerful generative models, pretrained on large-scale real-world data, have captured the correspondence between images and textual descriptions. It is probable that distilling such knowledge can benefit visual grounding through training data generation. Many recent works Zhao et al. (2023) ; Fan et al. (2024) ; Yang et al. (2024) ; Feng et al. (2024) ; Li et al. (2023) ; Fang et al. (2024) ; Chen et al. (2024c) ; Chen et al. (2024a) have explored the generation of densely annotated data to train models for fine-grained understanding tasks, such as object detection and image segmentation. However, they often rely on specialized networks trained on extensive dense annotations, which limits their flexibility and generalizability. To overcome this limitation, we propose to leverage off-the-shelf generative models Podell et al. (2023) ; Li et al. (2022) ; Rombach et al. (2022) , trained solely on image-caption pairs in a weakly-supervised manner, for training data generation. Compared to the methods Wang et al. (2025) ; Yang et al. (2024) ; Wang et al. (2024c) ; Zheng et al. (2024) ; He et al. (2024b) relying on specialized networks, our method offers two advantages. First, our method could generalize across a wider range of domains, since the generative models we used are trained on web-scale data. Second, we can benefit from the rapid progress in generative models, incorporating the latest advancements into our data generation pipeline without additional training on dense annotations. Although the off-the-shelf generative models are capable of synthesizing images across diverse domains, they struggle to directly generate data with region-text labels, as they are not trained with dense annotations. Therefore, we leverage the limited region-text annotations available in real data to generate training data through inpainting. However, a naive approach—inpainting the object within the bounding box ( e.g. , Gen2Det Suri et al. (2024) , see Figure 1 )—can result in label misalignment issues. Specifically, the generated object may not spatially align well with the box or even fail to appear entirely. For example, the synthetic ‘right cow’ in Figure 1 (a) misaligns with the box, and the ‘batter’ in Figure 1 (b) disappears. To tackle the label misalignment problem, we propose a paint-outside-the-box strategy, which preserves the content within the box while generating the background outside. This results in higher-quality images that align well with the original annotations. Moreover, training neural networks with synthetic data presents another challenge: not all samples contribute positively to the model performance Chen et al. (2024a) ; Chen et al. (2024e) ; Malach and Shalev-Shwartz (2017) ; Mahmoud et al. (2024) . To identify the most effective training data, we propose a filtering scheme consisting of three components: a hardness score, an overfitting score, and a penalty term. The hardness score prioritizes the easiest samples, as it is challenging to model outliers when the basic patterns are not adequately captured in our data-scarce regime Sorscher et al. (2022) . The overfitting score penalizes samples exacerbating overfitting the unintended features outside the box, as shortcut features from the background could lead the gigantic model to overfit Geirhos et al. (2020) ; Beery et al. (2018) ; Xiao et al. (2020) , especially with limited training data. To manage the trade-off between two scores, we introduce a penalty term. In summary, we propose a novel framework named POBF (Paint Outside the Box and Filter). POBF synthesizes high-quality training samples by image inpainting and captioning, then employs an effective filtering scheme for data selection. Our contributions are: • We address a challenging problem of visual grounding in data-scarce scenarios by introducing POBF, a flexible and adaptable framework that leverages off-the-shelf data generators pretrained solely on image–caption pairs. • We propose a novel data generation strategy for visual grounding, paint outside the box, which overcomes the label misalignment issues in previous approaches. • We introduce a new filtering scheme designed for data-scarce settings to identify the most effective synthetic training samples. This scheme integrates a hardness score, an overfitting score, and a penalty term. • Experiments demonstrate that POBF outperforms strong baselines by a 2.29%–3.85% margin. We further validate its robustness and generalizability across generative models, training data sizes, and model architectures. 2 Related Work Visual Grounding. Early approaches addressed visual grounding in a two-stage framework Liu et al. (2019a) ; Yu et al. (2018) ; Liu et al. (2019b) . Recently, more efficient one-stage methods have been developed Deng et al. (2021) ; Kamath et al. (2021) ; Ye et al. (2022) ; Su et al. (2023) ; Deng et al. (2023) ; Xiao et al. (2023) ; Shi et al. (2023)