Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development

Ming Shen, Raphael Shu, Anurag Pratik, James Gung, Yubin Ge, Monica Sunkara, Yi Zhang · 2025

Read Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development by Ming Shen, Raphael Shu, Anurag Pratik, James Gung, Yubin Ge, Monica Sunkara, Yi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16086.txt.

Introduction

Ming Shen and colleagues optimize teams of language model agents by reading their failure reports and rewriting the worst performers' instructions. Multi-agent systems split software work among role specialists, but tuning them is unsolved. Writing in twenty twenty-five, their pipeline runs in two steps. First, textual feedback from evaluations identifies underperforming agents and explains their failures. Second, those explanations drive prompt revisions for the flagged agents only. The team then compares settings: online versus offline optimization, and individual versus group tuning. Group tuning itself comes in one-pass and multi-pass flavors.

Software development tasks were graded on several dimensions. The method improved the role-based systems, the authors report. The comparisons also revealed how each setting changes group behavior. The study is framed as practical guidance for builders of agent teams.

Catalog record

CREATORMing Shen, Raphael Shu, Anurag Pratik, James Gung, Yubin Ge, Monica Sunkara, Yi Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16086

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development Ming Shen, Raphael Shu, Anurag Pratik, James Gung, Yubin Ge, Monica Sunkara, Yi Zhang Abstract We have seen remarkable progress in large language models (LLMs) empowered multi-agent systems solving complex tasks necessitating cooperation among experts with diverse skills. However, optimizing LLM-based multi-agent systems remains challenging. In this work, we perform an empirical case study on group optimization of role-based multi-agent systems utilizing natural language feedback for challenging software development tasks under various evaluation dimensions. We propose a two-step agent prompts optimization pipeline: identifying underperforming agents with their failure explanations utilizing textual feedback and then optimizing system prompts of identified agents utilizing failure explanations. We then study the impact of various optimization settings on system performance with two comparison groups: online against offline optimization and individual against group optimization. For group optimization, we study two prompting strategies: one-pass and multi-pass prompting optimizations. Overall, we demonstrate the effectiveness of our optimization method for role-based multi-agent systems tackling software development tasks evaluated on diverse evaluation dimensions, and we investigate the impact of diverse optimization settings on group behaviors of the multi-agent systems to provide practical insights for future development. 1 Introduction Autonomous agents utilizing large language models have achieved promising results on tasks across various domains such as reasoning Yao et al. (2023) ; Shinn et al. (2023) ; Ge et al. (2025) , code generation Shinn et al. (2023) , tool usage Cai et al. (2023) , and embodied AI Ahn et al. (2022) . Recent studies have demonstrated that incorporating synergistic agents into multi-agent collaboration frameworks substantially further enhances NLP problem-solving abilities Zhuge et al. (2023) ; Liang et al. (2023) ; Du et al. (2024) ; Wu et al. (2024b) ; Wang et al. (2024b) ; Li et al. (2023a) ; Hao et al. (2023) ; Zhang et al. (2024a) ; Dong et al. (2024) ; Jiang et al. (2023) ; Wu et al. (2024a) ; Chen et al. (2024) and successfully addresses complex real-world challenges, including human behavior simulation Park et al. (2023) , software development Qian et al. (2024) ; Hong et al. (2024) , code issue resolving Tao et al. (2024) , and web task execution Zhang et al. (2024b) . In our study, we focus on role-based multi-agent systems, where LLM-based agents are assigned distinct roles to accomplish a task collaboratively. A primary constraint inherent to LLMs is their reliance on prompt design Brown et al. (2020) ; Reynolds & McDonell (2021) ; Si et al. (2023) , which also extends to LLM-based agents. Specifically, it is crucial for developers to meticulously curate agent prompts, considering their respective roles and responsibilities in the collaboration framework. To this end, various automatic agent prompt optimization methods have been proposed in recent works, such as DSPy Khattab et al. (2024) , GPTSwarm Zhuge et al. (2024) , and TextGrad Yuksekgonul et al. (2024) . However, none of these works have conclusively demonstrated the efficacy of their approaches within a genuine role-based multi-agent system, wherein agents take on various roles to tackle a complex task collaboratively. Furthermore, current works primarily focus on traditional NLP benchmarks such as MMLU Hendrycks et al. (2021a) , MATH Hendrycks et al. (2021b) , and HumanEval Chen et al. (2021) . In contrast, our study aims to address the more intricate real-world software development task. In this work, we propose a two-step agent prompt optimization framework utilizing natural language feedback of the multi-agent system along various evaluation criteria for software development tasks (§ 4.2 ). In the first step, we employ an LLM-based locator to pinpoint underperforming agents, taking into account their roles and natural language feedback of the multi-agent system. The locator also provides fine-grained explanations for their underperformance. In the second step, we utilize an LLM-based optimizer Zhou et al. (2023) ; Pryzant et al. (2023) ; Yang et al. (2024) to optimize the system prompts of identified underperforming agents based on fine-grained explanations. We demonstrate the efficacy of our proposed optimization pipeline across various evaluation dimensions pertinent to software development tasks. For each evaluation dimension, we collect natural language feedback using either model-based or rule-based methods. Considering the inherent complexity of the multi-agent system, we further explore the impact of different optimization settings on the multi-agent system’s performance (§ 4.3 ). Concretely, we investigate two comparison groups: online against offline optimization and individual against group optimization. In the online setting, agents interact with the environment to collect feedback during optimization; however, feedback is collected beforehand in the offline setting. In the individual optimization setting, one agent is optimized at a time during each optimization step, and in the group setting, all agents are optimized in each step. For group optimization, we further investigate two prompting strategies during the optimization step: one-pass and multi-pass prompting. In one-pass group optimization, agents are optimized together with one inference pass, whereas in multi-pass group optimization, agents are optimized with separate inference passes. We show that online and offline optimizations are both effective, although offline performs slightly worse than online. We then show that optimizing all agents at each optimization step is necessary for our pipeline to outperform baselines consistently. Finally, we don’t observe apparent performance differences between one-pass and multi-pass prompting, so one could choose one-pass prompting optimization for efficiency. The contributions of our work are as follows: • We investigate optimizing a role-based multi-agent system on a complex real-world problem - software development. We show that the LLM-based multi-agent system can be effectively optimized utilizing textual feedback. • We study the software development optimization problem along five distinct dimensions, whereas existing literature mainly focuses on a single evaluation metric for simpler tasks. • We propose a two-step optimization pipeline by first locating and then optimizing underperforming agents. Experiment results show that our proposed pipeline outperforms baselines along the evaluation dimensions we study. • We compare optimization effectiveness in various settings, including online against offline and individual against group optimization. We demonstrate online and group optimizations as more effective settings than individual and offline. 2 Related Works 2.1 LLM-Based Agents LLM-based agents Yao et al. (2023) ; Shinn et al. (2023) have achieved impressive results on various NLP tasks. Applications Richards (2023) utilizing a single LLM-based agent, such as AutoGPT Richards (2023) and LangChain Chase (2022) , can accomplish more functionalities and provide opportunities to a diverse audience spectrum, including developers and even non-technical users. More recently, collaborations among multiple LLM-based agents demonstrated even more capabilities. Liang et al. (2023) and Du et al. (2024) incorporate multiple LLM-based agents into a debate framework, aiming to stimulate divergent thinking and enhance both factuality and reasoning capabilities. Utilizing the unique strengths and knowledge of each individual agent, LLM-based agents can also be synergistically integrated to collaboratively enhance problem-solving capabilities on a broad range of tasks Hao et al. (2023) ; Zhuge et al. (2023) ; Zhang et al. (2024a) ; Ge et al. (2023) ; Dong et al. (2024) ; Wu et al. (2024b) ; Zhang et al. (2024b) . In our work, we focus on role-based multi-agent system Wang et al. (2024b) ; Li et al. (2023a) ; Park et al. (2023) ; Qian

Read the complete text