SIDiffAgent: Self-Improving Diffusion Agent
Shivank Garg, Ayush Singh, Gaurav Kumar Nayak · 2026
Read SIDiffAgent: Self-Improving Diffusion Agent by Shivank Garg, Ayush Singh, Gaurav Kumar Nayak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02051.txt.
Introduction
Shivank Garg and colleagues offer Self-Improving Diffusion Agent, a training-free system using Qwen models. Text-to-image diffusion models enable photo images, yet use is blocked by prompt quirks, meaning mix-ups, and warped anatomy flaws. Current methods require extra training and offer limited control. SIDiffAgent manages prompt craft, corrects poor outputs, performs flaw removal, and stores past trials for prompt-based guidance. It hit a VQA score of zero point eight eight four on GenAIBench, beating open-source and closed methods.
Catalog record
| CREATOR | Shivank Garg, Ayush Singh, Gaurav Kumar Nayak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95881 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.02051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SIDiffAgent: Self-Improving Diffusion Agent Shivank Garg, Ayush Singh, Gaurav Kumar Nayak Abstract Text-to-image diffusion models have revolutionized generative AI, enabling high-quality and photorealistic image synthesis. However, their practical deployment remains hindered by several limitations: sensitivity to prompt phrasing, ambiguity in semantic interpretation (e.g., “mouse” as animal vs. a computer peripheral), artifacts such as distorted anatomy, and the need for carefully engineered input prompts. Existing methods often require additional training and offer limited controllability, restricting their adaptability in real-world applications. We introduce Self-Improving Diffusion Agent ( SIDiffAgent ), a training-free agentic framework that leverages the Qwen family of models (Qwen-VL, Qwen-Image, Qwen-Edit, Qwen-Embedding) to address these challenges. SIDiffAgent autonomously manages prompt engineering, detects and corrects poor generations, and performs fine-grained artifact removal, yielding more reliable and consistent outputs. It further incorporates iterative self-improvement by storing a memory of previous experiences in a database. This database of past experiences is then used to inject prompt-based guidance at each stage of the agentic pipeline. SIDiffAgent achieved an average VQA score of 0.884 on GenAIBench, significantly outperforming open-source, proprietary models and agentic methods. † † affiliationnotice: Equal contribution Figure 1: Qualitative comparison between our model, SIDiffAgent , and the state-of-the-art, Qwen-Image model. SIDiffAgent demonstrates superior text-to-image alignment and realism, accurately rendering prompts that require nuanced understanding. Notably, our model successfully handles: negative constraints (‘no umbrellas’), complex scene composition, anthropomorphic roles (‘athlete cat’), and precise object counting. Figure 2: Workflow of the Self-Improving Diffusion Agent ( SIDiffAgent ). An input prompt is processed by the Generation Orchestrator Agent ( A_{\text{ORC}} ), which employs sub-agents ( S_{\text{CRE}} , S_{\text{INT}} , S_{\text{REF}} , S_{\text{NEG}} ) to assess creativity, clarify intent, refine the prompt, and add adaptive negative constraints before generation ( S_{\text{GEN}} ). The Evaluation Agent ( A_{\text{EVAL}} ) scores the generated image on aesthetic quality and text–image alignment, triggering targeted refinements if the evaluation score is less than the pre-defined threshold \tau . Each trajectory is stored in the knowledge base, where the Guidance Agent ( A_{\text{GUID}} ) stores pitfalls and successes into corrective and workflow guidance, which are injected back into decision nodes to improve future generations. 1 Introduction Image generation models ( 60 ; 56 ; 61 ; 64 ; 24 ; 5 ; 15 ) have transformed digital image synthesis, enabling the creation of high-quality visuals with remarkable detail and realism from simple text descriptions. Despite these advances, the performance of state-of-the-art text-to-image diffusion models remains highly sensitive to the precise phrasing of input prompts, making them unable to capture the true intent of the user and creating a significant “intent gap” between the user’s intended meaning and the textual instructions provided to the model ( 8 ) . Here, intent refers to the underlying concept or goal that the user wishes the model to capture. This gap arises because natural language prompts are often ambiguous or casually phrased, leading the model to generate outputs that deviate from the user’s goal. For instance, the prompt “a mouse” may yield different outputs, either an animal or a computer peripheral, depending on how the model interprets the text. Another key challenge lies in the distributional mismatch between the unstructured, free-form language typical of end-user prompts and the highly curated, descriptive captions created by human experts and subjected to intensive filtering during the training of diffusion models ( 42 ; 73 ; 35 ) . This discrepancy often results in inadequate textual guidance for generation. For instance, prompts are frequently underspecified (e.g., “a car on a road” without details on the car’s model, color, or environment), which forces the model to make unguided assumptions. Such underspecification not only compromises output quality but also increases computational costs, as users must repeatedly regenerate outputs to achieve satisfactory results. These issues constitute a critical bottleneck for the scalability and reliability of diffusion models in professional domains where robust and predictable alignment with user intent is essential. To mitigate these challenges, prior work such as T2I-Copilot ( 8 ) and MCCD ( 39 ) have proposed agentic frameworks that address ambiguities through image editing and agentic coordination. However, these approaches often lack fine-grained controllability and robust mechanisms for post-generation artifact correction. A key challenge in such Multi-Agent Systems is coordination. Agents often have only partial observability and limited knowledge of others capabilities or the effects of their own actions, leading to suboptimal outcomes ( 7 ) . Recent works in the field of LLM Agents address this via Theory of Mind (ToM), the ability to attribute goals and beliefs to others ( 13 ; 81 ) . With ToM, agents can understand peers intentions and anticipate future actions, enabling better coordination. To the best of our knowledge, no work has applied this to diffusion-based agents. Additionally, ReNeg ( 40 ) highlights the role of negative prompts and embeddings ( 25 ) in constraining diffusion processes and guiding models away from undesired artifacts. In this work, we introduce a training-free, multi-agent framework SIDiffAgent that creates a robust and self-improving Theory of Mind inspired image generation pipeline. SIDiffAgent leverages the capabilities of the Qwen model family (Qwen-VL ( 2 ) , Qwen-Image ( 73 ) , Qwen-Edit ( 73 ) , Qwen-Embedding ( 84 ) ) as the backbone of our agentic flow. Our framework uses multiple agents to refine prompts, reuse past experience, and fix artifacts with local edits. Further building upon the observations from ReNeg ( 40 ) , SIDiffAgent also incorporates an additional adaptive negative prompt generation agent. Beyond controllability and artifact mitigation, we emphasize the importance of self-improvement: unlike prior agentic frameworks, SIDiffAgent is designed to leverage past successes and failures to iteratively refine its generation strategies. While our contribution focuses on system-level architecture rather than a new training objective, we highlight three technically novel components. First, SIDiffAgent introduces the first experience-driven memory mechanism for diffusion-based agents, enabling retrieval-conditioned guidance throughout the generation workflow. Second, our Theory-of-Mind–inspired inter-agent reasoning models predictive expectations over other agents’ behaviors using accumulated success/pitfall patterns, providing dynamic workflow guidance beyond prompt optimization. Third, SIDiffAgent offers a training-free alternative to RL-based or reward-guided finetuning approaches (e.g., DPO-Diffusion, ReNeg), enabling practical quality improvements even for closed-weight diffusion systems without any retraining. We empirically validate the effectiveness of SIDiffAgent across multiple benchmarks, including GenAIBench and DrawBench, where it establishes new state-of-the-art performance. Our approach not only surpasses prior agentic systems such as T2I-Copilot, achieving a VQA Score ( 44 ) improvement of 8.73%, but also outperforms leading proprietary models (e.g., Recraft V3, Imagen 3, Flux 1.1-Pro), delivering a 5.36% improvement over Imagen 3. Furthermore, it significantly exceeds the performance of powerful open-source systems (e.g., Flux.1-Dev, HunyuanDiT v1.2, and SD 3.5), achieving a 15.70% improvement over SD 3.5. Our key contributions are summarized as follows: • We introduce a multi-agent system that refines the inputs to a diffusion model at test time, ensuring consistent understanding of prompts even when they are expressed in different ways. This results in generations that more faithfully capture the user’s intended concepts. • We adapt a structured image editing approach that enables targeted, incremental corrections to generated images without requiring complete regeneration, narrowing the intent gap and improving alignment between user goals and model generations. • We build an iterative self-improvement mechanism