DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation

Zhuoling Li, Hossein Rahmani, Jiarui Zhang, Yu Xue, Majid Mirmehdi, Jason Kuen, Jiuxiang Gu, Jun Liu · 2026

Read DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation by Zhuoling Li, Hossein Rahmani, Jiarui Zhang, Yu Xue, Majid Mirmehdi, Jason Kuen, Jiuxiang Gu, Jun Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20470.txt.

Introduction

Li and co-authors present DiffGraph, an agent-driven framework for merging text-to-image models. Online communities host many expert models with distinct generative skills. Older merging methods cannot fully exploit that growing pool. DiffGraph organizes experts in a scalable graph with calibration. Registration adds new experts as the ecosystem expands. For each user request, it activates a fitting subgraph of experts. Flexible combinations then serve varied in-the-wild generation needs. Extensive experiments show the approach works well in practice.

Catalog record

CREATORZhuoling Li, Hossein Rahmani, Jiarui Zhang, Yu Xue, Majid Mirmehdi, Jason Kuen, Jiuxiang Gu, Jun Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51775
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20470

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation Zhuoling Li, Hossein Rahmani, Jiarui Zhang, Yu Xue, Majid Mirmehdi, Jason Kuen, Jiuxiang Gu, Jun Liu Abstract The rapid growth of the text-to-image (T2I) community has fostered a thriving online ecosystem of expert models, which are variants of pretrained diffusion models specialized for diverse generative abilities. Yet, existing model merging methods remain limited in fully leveraging abundant online expert resources and still struggle to meet diverse in-the-wild user needs. We present DiffGraph, a novel agent-driven graph-based model merging framework, which automatically harnesses online experts and flexibly merges them for diverse user needs. Our DiffGraph constructs a scalable graph and organizes ever-expanding online experts within it through node registration and calibration. Then, DiffGraph dynamically activates specific subgraphs based on user needs, enabling flexible combinations of different experts to achieve user-desired generation. Extensive experiments show the efficacy of our method. Project page: https://zhuoling.site/DiffGraph . 1 Introduction Benefiting from the extensive general knowledge acquired through large-scale pretraining, developers can fine-tune large diffusion models Rombach et al. 2022 ; hug on their own customized datasets to produce specialized variants (e.g., checkpoints and LoRA-based models) endowed with task-specific generative capabilities, such as creating novel artistic styles Ruiz et al. 2023 ; Zhang et al. 2023 ; Gal et al. 2022 or controlling particular visual attributes (e.g., image blur). We refer to these specialized variants as expert models . Nowadays, driven by the potential of text-to-image (T2I) generation, several online platforms (e.g., Civitai and Hugging Face) have rapidly flourished, where numerous experts with diverse skills are continuously developed, shared, and discussed, greatly enriching the online expert ecosystem and fostering a vibrant AI-driven creative community. Motivated by the growing popularity of expert models, several recent studies Biggs et al. 2024 ; Zhong et al. ; Gu et al. 2023 ; Wu et al. have explored merging multiple experts to combine their specialized skills for more composite and complex image generation, such as generating an image of a specific character in a particular artistic style by merging a specific character expert and a style expert. Despite the progress, existing approaches are mainly designed to merge a very small, fixed set of pre-provided experts Biggs et al. 2024 ; Chen et al. 2025 ; Shah et al. 2024 ; Wu et al. , and often struggle to handle different (new) expert combinations without retraining Wu et al. or test-time optimization Chen et al. 2025 . However, in-the-wild T2I generation scenarios are far more diverse. Users come from various backgrounds, such as graphic designers, web editors, and professional artists, and naturally exhibit distinct preferences and expectations for generated images. Consequently, prompts from different users, or even from the same user in different contexts, usually require combining very different experts or even different numbers of experts to satisfy highly varying in-the-wild user needs. Meanwhile, online platforms host an abundant and ever-expanding ecosystem of experts, where new experts with novel generative skills are constantly emerging. Therefore, fully organizing the large-scale online expert resources and flexibly utilizing (merging) them could unlock new opportunities to more effectively address diverse in-the-wild user demands. In terms of this, we note that some more recent methods Ouyang et al. 2025 ; Shenaj et al. 2025 ; Li et al. 2025a might have the potential to realize this vision due to their exploration of flexible merging of different experts. A key commonality among these methods is that they primarily rely on using experts’ model parameters as input features. For instance, LoRA.rar Shenaj et al. 2025 trains a supernetwork directly taking the parameter matrices of experts (typically one character LoRA and one style LoRA) as input features to predict merging coefficients for experts. Despite their achieved efficacy, we argue that these parameter-dependent methods Ouyang et al. 2025 ; Shenaj et al. 2025 ; Li et al. 2025a are still not suitable for fully harnessing the real-world large-scale online resources. This is mainly because online experts, even when derived from the same pretrained diffusion models, can be developed using vastly different and even ever-emerging fine-tuning strategies Hu et al. 2022 ; Liu et al. 2024b ; Clark et al. , diverse datasets, and varied training configurations. As a result, online experts exhibit substantial and very complex parameter diversity and even architectural differences (e.g., checkpoint-based or LoRA-based experts) , making these parameter-dependent model merging methods Ouyang et al. 2025 ; Li et al. 2025a ; Shenaj et al. 2025 struggle to generalize well to such diverse, heterogeneous, and time-evolving online resources. Moreover, these methods Ouyang et al. 2025 ; Shenaj et al. 2025 ; Li et al. 2025a are still limited to either combining a fixed number of experts or manually specifying the number of experts to be merged. In this paper, we propose DiffGraph , a novel automated agent-driven graph-based model merging framework , which automatically harnesses and flexibly merges various (and even various numbers of) online expert resources, serving for diverse user needs without any retraining or test-time optimization after deployment, while seamlessly scaling to the evolving expert ecosystem. Drawing on the insight that graphs can naturally encode heterogeneous entities and their relationships Wu et al. 2022 ; Yang et al. 2021 ; Feng et al. 2025 ; Hamilton et al. 2017 , we introduce a novel graph formulation tailored to expert management and merging. As shown in Fig. 1 , our DiffGraph incorporates two LLM-powered agents, namely, the Graph Construction Agent (GCA) and Expert Selection Agent (ESA) together with a variational graph autoencoder (VGAE)-based Merging Planner (MP) . These components collectively provide the management and utilization of abundant online expert resources to fulfill in-the-wild T2I generation needs. Specifically, GCA first constructs a universal graph to manage online expert resources, by automatically collecting high-quality experts from public platforms and organizing them as expert nodes in the graph through two complementary mechanisms: node registration and node calibration . The node registration mechanism qualitatively summarizes textual descriptions of experts’ skills from their homepage information and encodes these descriptions as expert node features. Meanwhile, the node calibration mechanism quantitatively evaluates experts on a set of representative reference prompts (i.e., reference prompt nodes in the graph), and the resulting performance scores are represented as edge features between the two types of nodes, as illustrated in Fig. 1 . On the other hand, the Expert Selection Agent (ESA) and Merging Planner (MP) operate on the universal graph constructed by GCA. When a user provides a prompt (referred to as a user prompt ), ESA parses the underlying requirements conveyed by the prompt and selects a set of experts with promising skills to fulfill these requirements. Then, MP activates a subgraph centered on the selected expert nodes in the universal graph and temporarily inserts the user prompt into this subgraph as a user prompt node . Within this subgraph, the node and edge features of the selected experts provide rich contextual cues, offering an informative and holistic characterization of their skills. A trained VGAE is then employed to encode the contextual information, which then generates a high-quality merging scheme to combine the selected experts for generation. Overall, our approach is effective and efficient. The construction of the universal graph is only to be performed

Read the complete text