Interactive Debugging and Steering of Multi-Agent AI Systems
Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, Saleema Amershi · 2025
Read Interactive Debugging and Steering of Multi-Agent AI Systems by Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, Saleema Amershi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02068.txt.
Introduction
Teams of AI agents can handle hard tasks together, but when they go wrong, the fault is hard to find. Will Epperson and co-authors first spoke with five agent builders in 2025 to learn what makes such debugging so slow. Long chat logs, weak tools, and no easy way to change an agent's setup stood out as core problems.
In response, the team built AGDebugger, a tool for viewing agent chats and editing past messages. Users can reset a chat to an earlier point and try a new path. In studies with 14 users, that reset step proved key for steering agents back on track after a fault.
Catalog record
| CREATOR | Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, Saleema Amershi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02068
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interactive Debugging and Steering of Multi-Agent AI Systems Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, Saleema Amershi Abstract . Fully autonomous teams of LLM-powered AI agents are emerging that collaborate to perform complex tasks for users. What challenges do developers face when trying to build and debug these AI agent teams? In formative interviews with five AI agent developers, we identify core challenges: difficulty reviewing long agent conversations to localize errors, lack of support in current tools for interactive debugging, and the need for tool support to iterate on agent configuration. Based on these needs, we developed an interactive multi-agent debugging tool, AGDebugger , with a UI for browsing and sending messages, the ability to edit and reset prior agent messages, and an overview visualization for navigating complex message histories. In a two-part user study with 14 participants, we identify common user strategies for steering agents and highlight the importance of interactive message resets for debugging. Our studies deepen understanding of interfaces for debugging increasingly important agentic workflows. Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3713581 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing Interactive systems and tools CCS: Computing methodologies Multi-agent systems Figure 1. Debugging multi-agent AI systems involves reasoning over long multi-turn conversations where specialized agents use tools like web browsing and writing code with LLMs. AGDebugger allows users to interactively debug and steer multi-agent teams by resetting the agents to earlier points in the workflow then editing messages to interactively test hypotheses about their behavior. This figure shows an agent conversation with represented as boxes on the lefthand side. Several of these boxes are connected to screenshots of the tool used in that box such as a web browsing page or code writing. One of the messages is annotated with an annotation that reads What would have happened with a different plan at message 5?. This leads to a second conversation that is a fork of the first where the messages above the fork are greyed out and the new messages are different. At the bottom, an annotation shows that the original agent conversation led to the wrong output with a red x, but the new one leads to the correct output with a green check. † † cc-license: by 1. Introduction Multi-agent AI systems are emerging as a powerful paradigm for addressing tasks beyond the scope of single large language models (LLMs) ( Wu et al., 2024a ; Li et al., 2023 ; Wu et al., 2024b ; CrewAI, 2024 ) . By combining LLMs with multi-turn state tracking, external tool use, and collaborative interactions, multi-agent systems can perform complex, real-world tasks such as accessing up-to-date information or executing actions in dynamic environments ( He et al., 2024 ) . This capability has allowed multi-agent AI systems to excel in challenging benchmarks that require reasoning about complex tasks and using tools to interact with the world like web browsing ( He et al., 2024 ) , reasoning over complex files ( Mialon et al., 2024 ) , and writing and executing code ( Jimenez et al., 2024 ; Wang et al., 2024 ) . Despite these advances, existing AI development tools fall short when it comes to understanding and debugging the complex, multi-turn behaviors of agent teams. Traditional AI debugging practices, which focus on model training or correcting datasets, are inadequate in this new paradigm. Much of the work in debugging LLM systems centers around crafting effective text prompts that instruct the LLM how to accomplish a task through in-context learning ( Brown et al., 2020 ; Zamfirescu-Pereira et al., 2023 ) . Prior research has developed tools for crafting effective prompts for tasks involving individual LLM invocations ( Jiang et al., 2022 ; Tenney et al., 2024 ; Petridis et al., 2024 ; OpenAI, 2024 ) or chains of LLM calls ( Wu et al., 2022a ; Arawjo et al., 2023 ) , but has not addressed the challenges associated with debugging teams of fully-autonomous AI agents. Debugging multi-agent teams introduces new debugging challenges since agent teams require first crafting individual prompts and tools for each agent, then understanding how the team works together to accomplish a task by making numerous LLM calls over a multi-turn conversation. Agent conversations are complex and dynamic, where agents formulate a plan on the fly for a task and then execute the plan by using tools to interact with the world as needed ( Wu et al., 2024a ; CrewAI, 2024 ) . Debugging multi-agent systems requires simultaneously understanding both the individual behaviors of agents and the emergent interactions between them. This multifaceted debugging challenge demands new tools that can integrate insight across the entire agent team to avoid “cascading errors” that fix one component while breaking another ( Wu et al., 2022a ; Sculley et al., 2014 ; Nushi et al., 2017 ) . While recent systems like AutoGen Studio ( Dibia et al., 2024 ) and OpenDevin ( Wang et al., 2024 ) enable developers to interact with multi-agent AI teams, they primarily focus on task execution and agent construction. These platforms lack robust debugging features, particularly for multi-turn interactions. As agent conversations grow longer, existing tools do not provide the ability to pause, rewind, or edit agent behaviors in real time. Additionally, they lack comprehensive visualizations to help developers track and understand the evolving dynamics between agents. These gaps in literature makes debugging multi-agent systems particularly challenging, as developers need to iteratively diagnose and adjust interactions across the team. To address the limitations of current tools and better support the debugging of multi-agent AI systems, we explore two key research questions: (1) How can we design systems that enable developers to effectively debug multi-agent AI teams? (2) How do developers use such a system to debug and improve agent workflows in practice? To investigate these research questions, we first conducted formative interviews with five expert developers who have extensive experience building multi-agent AI systems. These interviews revealed several key challenges in the debugging process, such as difficulties in understanding long, multi-turn agent conversations, the lack of interactive debugging support in existing tools, and the need for better tooling to iterate on agent configurations. Using these findings, we designed an interactive debugging system, AGDebugger to address these challenges. AGDebugger has three primary features to facilitate the debugging process. The first builds off prior interfaces for agent configuration ( Dibia et al., 2024 ; Wang et al., 2024 ; LangChain, 2024 ) and allows users to interactively send messages to the agents and inspect the history of messages sent with fine-grained control on the execution of messages. An extension to chatting with agents is the ability to interactively control a conversation by resetting to previous points and editing previously sent agent messages. AGDebugger enables such an interaction by check-pointing agent state (including state that might be impacted by agent actions and tool use) before each message is sent to enable resetting to earlier points in a conversation and editing agent messages. Finally, as agent conversations grow longer and users edit the