SafeFix: Targeted Model Repair via Controlled Image Generation
Ouyang Xu, Baoming Zhang, Ruiyu Mao, Yunhui Guo · 2025
Read SafeFix: Targeted Model Repair via Controlled Image Generation by Ouyang Xu, Baoming Zhang, Ruiyu Mao, Yunhui Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08701.txt.
Introduction
Ouyang Xu, Baoming Zhang, Ruiyu Mao, and Yunhui Guo present a way to repair image recognition models that keep failing on rare kinds of cases, in a 2025 paper. Such models often stumble on underrepresented groups in their training data. Debugging tools can name what goes wrong, but fixing the model without breaking it elsewhere is harder.
Their module, SafeFix, creates targeted training images with a text-to-image generator guided by the known failure traits. A large vision-language model then filters the made-up images, keeping only those that match the real data in meaning and style. Retraining with this rare-case boost cut the targeted errors in their experiments. Robustness rose without fresh bugs appearing, and the code is public.
Catalog record
| CREATOR | Ouyang Xu, Baoming Zhang, Ruiyu Mao, Yunhui Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08701
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SafeFix: Targeted Model Repair via Controlled Image Generation Ouyang Xu, Baoming Zhang, Ruiyu Mao, Yunhui Guo Abstract Deep learning models for visual recognition often exhibit systematic errors due to underrepresented semantic subpopulations. Although existing debugging frameworks can pinpoint these failures by identifying key failure attributes, repairing the model effectively remains difficult. Current solutions often rely on manually designed prompts to generate synthetic training images—an approach prone to distribution shift and semantic errors. To overcome these challenges, we introduce a model repair module that builds on an interpretable failure attribution pipeline. Our approach uses a conditional text-to-image model to generate semantically faithful and targeted images for failure cases. To preserve the quality and relevance of the generated samples, we further employ a large vision–language model (LVLM) to filter the outputs, enforcing alignment with the original data distribution and maintaining semantic consistency. By retraining vision models with this rare-case-augmented synthetic dataset, we significantly reduce errors associated with rare cases. Our experiments demonstrate that this targeted repair strategy improves model robustness without introducing new bugs. Code is available at https://github.com/oxu2/SafeFix . 1 Introduction Despite extensive training on annotated images and evaluation on static test sets, computer vision models can still fail in unexpected ways [ 7 , 18 , 29 ] . One common cause is dataset bias: certain subpopulations are underrepresented in the training data. For instance, facial recognition systems have shown significantly higher error rates for darker-skinned women compared to lighter-skinned men, due to imbalanced representation in training datasets [ 9 ] . Therefore, identifying and repairing potential model failures caused by underrepresented semantic subpopulations before deployment is essential to improve the reliability of computer vision systems in real-world applications. Several recent studies have addressed the identification and repair of model failures caused by underrepresented semantic subpopulations. For example, given a natural language description of a potentially underrepresented semantic subpopulation, AdaVision [ 18 ] introduces a model debugging pipeline that leverages CLIP [ 38 ] to iteratively retrieve relevant images from LAION-5B [ 40 ] and evaluate model predictions. HiBug [ 11 ] advances interpretable model debugging by automatically discovering failure slices based on shared visual attributes using a pre-trained vision–language model. DCD [ 41 ] identifies failure modes by collecting misclassified samples, computing a failure direction that captures their distinguishing attributes, and retrieving similar unlabeled examples that reveal systematic model weaknesses and underrepresented subpopulations. The latent-space-based filtering method [ 25 ] identifies failure patterns by training a linear SVM in a vision–language latent space to separate correct from incorrect predictions, revealing interpretable subpopulation errors without requiring labels. In addition, D3M [ 24 ] proposes to remove harmful training samples that disproportionately hurt underrepresented groups, thus enhancing overall model performance. While existing methods can effectively identify model errors caused by underrepresented semantic subpopulations, the subsequent and critical step—model repair—still presents key challenges. For instance, AdaVision [ 18 ] employs a simple fine-tuning strategy to retrain the model on retrieved images that cause model errors. However, this approach can degrade performance on images where the model originally performed well. HiBug [ 11 ] addresses model repair by turning attributes describing certain underrepresented subpopulations into language prompts to synthesize new training images via generative models. However, the synthesized images may suffer from distribution mismatch compared to the original training data. Moreover, the repair process can introduce new bugs, as the generated images may fail to preserve attributes unrelated to the original model errors. Building on these observations, we propose SafeFix, a targeted model repair method designed to correct failures caused by underrepresented semantic subpopulations through controlled image generation—without degrading overall performance or introducing new errors. Compared to AdaVision [ 18 ] and DCD [ 41 ] , which retrieve samples from another dataset, our method generates new examples grounded in the training set to ensure distributional consistency. Unlike HiBug [ 11 ] , which simply composes failure attributes into language prompts, SafeFix anchors generation on real instances in the training set to preserve unrelated attributes and avoid unintended changes. While the latent-space-based filtering method [ 25 ] explores repair via diffusion models, their approach relies solely on caption-based prompting and does not guarantee the semantic accuracy of critical attributes. In contrast, SafeFix mitigates the unreliability of generative models in rendering sensitive attributes—such as skin tone or hair color—by further filtering out semantically inconsistent images via a large vision–language model (LVLM) [ 6 , 33 ] . This ensures that the final outputs faithfully reflect the intended attribute change (e.g., generating a sad expression on a darker-skinned woman with red hair, whereas standard diffusion models often mistakenly produce a darker-skinned man with black hair instead), enabling precise and robust repair of underrepresented semantic subpopulation failures. SafeFix directly addresses two major challenges in existing model repair strategies: 1) ensuring the generated images reflect the true failure attributes, and 2) aligning them with the original data distribution. By retraining models on this augmentation set, we reduce errors associated with underrepresented semantic subpopulations, where model failures stem from insufficient training coverage of specific attributes. Our contributions are as follows: • We propose a targeted model repair pipeline, SafeFix, which leverages conditional text-to-image generation and LVLM-based filtering to synthesize high-quality, attribute-faithful data for correcting model failures arising from underrepresented subpopulations. • We show that our approach mitigates distribution mismatch in generated samples, ensuring that the augmented data remains aligned with the original dataset distribution. This alignment helps correct model failures while maintaining overall performance. • We demonstrate that retraining with curated, distribution-consistent synthetic data significantly enhances model robustness in underrepresented scenarios—without introducing new errors. 2 Related Work Failure Pattern Discovery. HiBug [ 11 ] proposes a pioneering pipeline that identifies interpretable failure cases in vision models by clustering semantically meaningful attributes, revealing both rare categories and spurious correlations. HiBug2 [ 12 ] extends this approach with more efficient error-slice discovery and a closed-loop debugging mechanism, improving coherence and coverage of discovered model bugs. Several follow-up works attempt to strengthen interpretability in model debugging [ 1 , 2 ] , though often limited by static failure patterns or inadequate visual grounding. Beyond HiBug, MODE [ 43 ] introduces a state-differential analysis framework that locates internal model faults and proposes data-driven remedies. TCAV [ 27 ] further enriches interpretability by testing the model’s sensitivity to high-level concepts and by correcting spurious activations at the concept level. 3DB [ 29 ] complements these efforts by constructing structured attribute spaces over failure modes, allowing discovery of underrepresented attributes through unsupervised analysis of visual model errors. Targeted Synthetic Augmentation via Diffusion Models. Diffusion models such as Stable Diffusion [ 39 ] and classifier-free guidance [ 22 ] enable controllable, semantically faithful image synthesis. These models have proven useful in debugging [ 10 , 17 , 23 ] , counterfactual visualization [ 4 , 8 ] , and training data augmentation [ 42 , 15 ] , especially in few-shot and fine-grained recognition tasks. Recent work like DiGA [ 47 ] shows how editing spurious attributes while preserving class semantics can mitigate bias without requiring new annotations. Others, such as GenMix [ 30 ] , propose mixing synthetic and real images to enhance robustness. These advances highlight how targeted generation can shape training distributions