ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Development
Zhenran Xu, Xue Yang, Yiyu Wang, Qingli Hu, Zijiao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang · 2025
Read ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Development by Zhenran Xu, Xue Yang, Yiyu Wang, Qingli Hu, Zijiao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05010.txt.
Introduction
Zhenran Xu and nine co-authors build ComfyUI-Copilot, a language-model plugin that helps users assemble workflows in the ComfyUI art platform. ComfyUI is flexible but hard on newcomers, with thin documentation, easy misconfigurations, and complex graph design. The copilot aims to lower that barrier.
A central assistant agent delegates to specialized worker agents, backed by curated knowledge bases about nodes and models. The system recommends components, constructs workflows in one click, and helps debug and deploy them. Offline evaluations and online user feedback showed accurate recommendations and faster workflow building. Case studies found benefits for beginners and experienced users alike. The package and a demo video are publicly available.
Catalog record
| CREATOR | Zhenran Xu, Xue Yang, Yiyu Wang, Qingli Hu, Zijiao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05010
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Development Zhenran Xu, Xue Yang, Yiyu Wang, Qingli Hu, Zijiao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang Abstract We introduce ComfyUI-Copilot , a large language model-powered plugin designed to enhance the usability and efficiency of ComfyUI, an open-source platform for AI-driven art creation. Despite its flexibility and user-friendly interface, ComfyUI can present challenges to newcomers, including limited documentation, model misconfigurations, and the complexity of workflow design. ComfyUI-Copilot addresses these challenges by offering intelligent node and model recommendations, along with automated one-click workflow construction. At its core, the system employs a hierarchical multi-agent framework comprising a central assistant agent for task delegation and specialized worker agents for different usages, supported by our curated ComfyUI knowledge bases to streamline debugging and deployment. We validate the effectiveness of ComfyUI-Copilot through both offline quantitative evaluations and online user feedback, showing that it accurately recommends nodes and accelerates workflow development. Additionally, use cases illustrate that ComfyUI-Copilot lowers entry barriers for beginners and enhances workflow efficiency for experienced users. The ComfyUI-Copilot installation package and a demo video are available at https://github.com/AIDC-AI/ComfyUI-Copilot . 1 Introduction Recent advancements in large language models (LLMs) and image generation methods have democratized AI-generated content (AIGC) production, with open-source frameworks like ComfyUI comfyanonymous 2023 emerging as pivotal tools for low-code AI workflow development. Serving over 4 million active users and backed by a vibrant community contributing 12K+ components (e.g., SDXL Podell et al. 2023 , ControlNet Zhang et al. 2023 ), ComfyUI enables flexible workflow orchestration via drag-and-drop components for multimodal tasks such as text-to-image generation, face swapping, and video editing. Figure 1: An example of automated workflow generation in ComfyUI-Copilot : the copilot suggests multiple workflows based on the user instruction and loads the selected one into the canvas with a single click. Despite its convenience, newcomers may face several potential barriers when starting with ComfyUI. These challenges include the installation of dependent nodes and models, scattered documentation across forums and GitHub issues. Even experienced users require substantial expertise to debug and construct a well-designed workflow Gal et al. 2024 . Recent research on automatic workflow construction has limitations, such as instability (i.e., generating unprocessable workflows) and a narrow focus primarily on text-to-image generation tasks Xue et al. 2024 ; Sobania et al. 2024 . Addressing these challenges and facilitating the onboarding process for ComfyUI is therefore crucial. To this end, we introduce ComfyUI-Copilot , an LLM-empowered multi-agent framework designed to assist users in navigating ComfyUI. It provides the following key features: (1) Automatic workflow generation: Our copilot can identify user intent, retrieve or synthesize an appropriate workflow, and then integrate it into the ComfyUI canvas. An example of its functionality is shown in Figure 1 . (2) Node and model recommendation: Our copilot can suggest suitable nodes based on user instructions, recommend relevant checkpoints and LoRA models. (3) ComfyUI-related question answering: Our copilot provides detailed tutorials on selected nodes and models, including usage guidelines, installation steps, and parameter explanations. It can also offer multiple feasible downstream subgraphs for selected nodes. Additionally, we introduce new features aimed at enhancing workflow debugging and optimization, including prompt writing and parameter search. The framework of ComfyUI-Copilot is centered around an LLM-based assistant agent, which coordinates with various specialized worker agents and knowledge bases (KBs). Depending on the query, the assistant agent may address user queries directly or delegate tasks to appropriate worker agents. We have developed three primary worker agents focused on workflow generation, node and model recommendation. To support these agents, we have constructed extensive KBs covering 7K nodes, 62K models, and 9K workflows. These KBs are enhanced through automated documentation generation by leveraging LLM’s code comprehension capabilities, and are continuously expanded and updated daily. Unlike prior work Gal et al. 2024 ; Sobania et al. 2024 which only targets text-to-image generation, the resources in our KBs extend to conditional multimodal generation tasks, ensuring that our system accommodates both diverse tasks and cutting-edge modules with accuracy. Experiments show that ComfyUI-Copilot provides accurate assistance in node recommendation and workflow construction based on user instructions. The high recall rates for workflows and nodes (both exceeding 88.5%) validate the practical efficacy in automated workflow development and accurate node recommendation. Since its release on GitHub, online user feedback reflects a moderately high acceptance rate of 65.4% for recommended nodes and a notably high acceptance rate of 85.9% for proposed workflows. Use cases further highlight the system’s capability to reduce entry barriers for beginners and enhance workflow efficiency for experienced ComfyUI users with multilingual support. To the best of our knowledge, ComfyUI-Copilot is the first open-source project to develop a ComfyUI plugin for automating workflow creation and providing instant suggestions. As of the camera-ready date (May 29, 2025), it has already attracted a rapidly growing user base, accumulating over 1.6K GitHub stars and processing more than 85K queries from 19K users across 22 countries. In future work, we plan to incorporate feedback from the active open-source community and continuously update features to better address user needs. 2 Related Work Figure 2: Overview of the ComfyUI-Copilot framework : The central LLM-based assistant agent can either respond directly to user instructions based on the conversation history (i.e., short-term memory), or collaborate with specialized worker agents. These agents are supported by our curated ComfyUI knowledge bases. AI-generated content (AIGC) based on ComfyUI. Diffusion models have gained wide attention in AI research for image synthesis Ho et al. 2020 ; Dhariwal and Nichol 2021 . As the field of text-to-image generation progresses, new tasks and models ( Kumari et al. 2023 ; Ruiz et al. 2023 ; Li et al. 2023 ; Zhang et al. 2023 ) have been proposed to introduce controllable conditions in image generation. Therefore, researchers and practitioners are transitioning from simple text-to-image workflows to more sophisticated ones, where the open-source ComfyUI comfyanonymous 2023 offers great convenience. In ComfyUI, users can easily construct workflows by connecting a series of blocks, each representing specific models or parameter choices. Each ComfyUI workflow can be exported to a JSON file which outlines both the graph nodes and their connectivity. Instead of relying on an end-to-end diffusion model for image generation, advanced workflows combine a variety of components to enhance image quality Guo et al. 2024 ; Ye et al. 2023 . These components may include fine-tuned versions of generative models, large language models (LLMs) for refining input prompts, LoRAs trained to introduce specific artistic styles, improved latent decoders for finer details, super-resolution blocks, and more ( Hu et al. 2021 ; Mañas et al. 2024 ; Berrada et al. 2025 ; Ning et al. 2021 ) . Importantly, effective workflows are prompt-dependent, with the selection of models and nodes often based on the user intent and the desired image content Gal et al. 2024 . Therefore, creating a well-designed workflow and selecting appropriate nodes and models require significant expertise, where our ComfyUI-Copilot comes into help. LLM-based agents. Recent advancements in LLMs have demonstrated great improvements in reasoning abilities and adaptability to new content and tasks Chen et al. 2024b ; Zeng et al. 2024 ; Wang et al. 2025 . Based on these emergent capabilities Wei et al. 2022 , various