Personalized Generative Models for Contextual Debiasing
Xinran Liang, Esin Tureci, Prachi Sinha, Ye Zhu, Vikram V. Ramaswamy, Olga Russakovsky · 2026
Read Personalized Generative Models for Contextual Debiasing by Xinran Liang, Esin Tureci, Prachi Sinha, Ye Zhu, Vikram V. Ramaswamy, Olga Russakovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26353.txt.
Introduction
Xinran Liang and five colleagues tackle contextual bias in vision: models learn objects best in common settings. Yet spotting a beach ball on a road may matter more than spotting one on sand. Their DecoupleGen method personalizes text-to-image diffusion models to synthesize coherent images in rare contexts while keeping original visual details. Verification constraints keep the augmented data relevant. On object classification and recognition over complex scenes, the approach consistently beats prior methods, and analysis pins down factors behind the gains.
Catalog record
| CREATOR | Xinran Liang, Esin Tureci, Prachi Sinha, Ye Zhu, Vikram V. Ramaswamy, Olga Russakovsky |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64184 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Personalized Generative Models for Contextual Debiasing Xinran Liang, Esin Tureci, Prachi Sinha, Ye Zhu, Vikram V. Ramaswamy, Olga Russakovsky Abstract Different visual patterns appear with different frequencies in the world: e.g., beach balls appear on sand more often than they do on a road. These statistics are reflected in vision datasets, and as a result trained models more easily recognize objects in common scenarios. However, recognizing a beach ball on a road may arguably be even more important than recognizing it on sand. We study how to mitigate this discrepancy. Since collecting uncommon images in the real world may be difficult, we explore whether generating images with less frequent contexts can serve as effective training augmentation. A key challenge is guiding generations to remain close to the original dataset distribution while creating diverse images with uncommon contexts. We introduce Decoupl ing Cont e xtual Patterns with Gen erations ( DecoupleGen ), a method that personalizes text-to-image diffusion models to facilitate coherent synthesis of images with rare contexts while preserving original visual details. The generated images contain semantically meaningful content and remain visually aligned with the original datasets. We further apply verification constraints to ensure relevance of the augmented data. We evaluate our approach on object classification and recognition tasks on complex scene datasets. Our experiments demonstrate consistent improvements over previous approaches, and our analyses identify factors underlying these improvements. 1 Introduction Improvements in computer vision models are fueled, in part, by advances in data collection methods, resulting in ever-larger datasets. However, despite diverse curation strategies, these datasets still contain multiple forms of bias 10 ; 51 ; 9 ; 58 ; 64 . One way these biases can manifest is through common co-occurring patterns; for example, skis frequently appear with people, while images of skis alone are rare. These contextual biases often cause models to fail to generalize to uncommon scenarios; going back to our example, these models often have low accuracy on images of skis without people 53 . While recent general-purpose vision-language models may exhibit greater robustness to such biases due to large-scale pretraining ( 7 ; 52 ) , many real-world and industrial applications still utilize smaller specialized models given compute and memory constraints ( 36 ; 11 ) , motivating the need to improve generalization of compact classifiers. (a) Original (b) Txt2Img ( 47 ) (c) Inpaint ( 47 ) (d) DecoupleGen Figure 1 : An example illustrating challenges in contextual debiasing. Our goal is to generate an image of skis without a person. Using common Txt2Img models typically produces images with visual appearances that differ substantially from those in the original dataset (Figure 1(b) ). Applying existing local image editing methods, e.g. inpainting, often generates less plausible results (Figure 1(c) ). Modifying other elements related to person is necessary to ensure visual coherence of the entire image (Figure 1(d) ). One way to address these biases is through training objectives, such as reweighting, that penalize models for over-relying on contextual cues ( 53 ; 14 ; 8 ; 17 ) . However, these methods are limited to using only the original datasets and can often result in poorer performance on in-context samples. An alternative way to tackle these biases is to leverage recent advances in generative models 47 ; 25 ; 41 ; 45 ; 24 to synthesize images with diverse content. These models can be used to edit images to remove common contexts, or generate images with difficult contexts. Recent works explore the potential to incorporate these generations as data augmentation to improve recognition performance. However, they usually focus on classifying a single object in an image ( 6 ; 21 ; 15 ; 23 ) , thus are not well equipped to handle interactions between multiple objects in real world scenarios, limiting their potential scope and applicability. Constructing images that contain objects in uncommon contexts using generative models can be challenging. Since objects and scenes usually interact with each other in real images, manipulating contextual patterns requires handling their complex relationships in order to generate natural looking images. Furthermore, in order for these samples to improve model performance, the resulting generations should ideally remain close to the distribution of the original dataset. Consider the COCO 12 dataset and, in particular, images containing skis, the person within the image usually interacts with the skis in some way (Figure 1(a) ). When generating images with rare contexts such as skis without people, pre-trained text-to-image (Txt2Img) models often produce images with viewpoints and backgrounds backgrounds that differ substantially from the original dataset (Figure 1(b) ), which may not be relevant to downstream tasks. On the other hand, local editing methods, e.g. inpainting ( 47 ; 9 ) , which takes mask as input to explicitly remove or replace the context, are unable to deal with interactions among objects. Removing the person in this manner results in the skis floating (Figure 1(c) ). Instead, generating a plausible version of the image without the person requires repositioning the skis as well, as shown in Figure 1(d) . To address this issue, we introduce Decoupl ing Cont e xtual Patterns with Gen erations ( DecoupleGen ) (Figure 2 ). We adapt pre-trained diffusion models to downstream datasets by fine-tuning them on images containing common contextual patterns, and then generate samples containing uncommon contextual patterns. Specifically, inspired by personalization methods of diffusion models for subject-driven generations ( 48 ; 18 ) , we learn new word tokens that capture visual details from original images. We then prompt the fine-tuned diffusion models with the learned text embeddings to manipulate contextual patterns. The resulting generations are natural looking and visually similar to images from the original datasets. To obtain generations more relevant to downstream recognition tasks, we further select synthesized images that do not contain common contextual patterns, ensuring that the additional training samples capture underrepresented groups in the original datasets and supplement out-of-context samples. We evaluate our method on visual recognition tasks using real-world object and scene datasets ( 12 ; 22 ) . Our experiments show that DecoupleGen outperforms existing approaches that modify loss functions or leverage generative models for data augmentation by 1.5%, improving over standard classifier by 14.2% on the NICO dataset and by 5.4% on the COCO dataset. Our analyses show two distinctive advantages of DecoupleGen: first, synthesized images from our method remain closer to the downstream dataset, indicating better alignment; and second, our method is better accounts for complex relationships between objects and scenes depicted in natural images. 1 1 1 In this work, “personalization” refers to subject-driven visual generation techniques that extract and reproduce specific visual concepts from one or a few reference samples. 2 Related Work Addressing model bias via dataset manipulations. Prior work has shown that existing datasets are not fully representative of the real world ( 31 ; 56 ) , causing models trained on them to struggle to generalize to uncommon scenarios. We focus specifically on failures where vision models cannot recognize objects without their co-occurring contexts, a limitation observed in both deep neural network classifiers ( 13 ; 53 ) and recent large vision-language models ( 34 ; 35 ) . One approach to address these limitations is to rebalance