Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale DOI: XXXXXXX.XXXXXXX Journal: TOSEM Volume: 0 0 0 0 CCS: Software and its engineering Designing software

Yangxiao Cai, Ruiyin Li, Peng Liang, Mojtaba Shahin, Zengyang Li · 2025

Read Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale DOI: XXXXXXX.XXXXXXX Journal: TOSEM Volume: 0 0 0 0 CCS: Software and its engineering Designing software by Yangxiao Cai, Ruiyin Li, Peng Liang, Mojtaba Shahin, Zengyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08475.txt.

Introduction

Yangxiao Cai, Ruiyin Li, Peng Liang, and Mojtaba Shahin systematically study how researchers design large language model based multi agent systems for software engineering tasks. Such systems promise autonomy and scalability as engineering work grows more complex, yet no dedicated study had mapped their design choices. The authors collected ninety four papers and extracted the quality attributes designers prioritize, the design patterns they use, and the reasons behind those choices.

Code generation is the most common task among ten identified, and functional suitability is the quality attribute receiving most attention. Role based cooperation is the most frequent pattern among sixteen, while improving generated code quality is the leading rationale. The authors draw implications for future system design from these patterns.

Catalog record

CREATORYangxiao Cai, Ruiyin Li, Peng Liang, Mojtaba Shahin, Zengyang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08475

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale DOI: XXXXXXX.XXXXXXX Journal: TOSEM Volume: 0 0 0 0 CCS: Software and its engineering Designing software Yangxiao Cai, Ruiyin Li, Peng Liang, Mojtaba Shahin, Zengyang Li Abstract . As the complexity of Software Engineering (SE) tasks continues to escalate, Multi-Agent Systems (MASs) have emerged as a focal point of research and practice due to their autonomy and scalability. Furthermore, through leveraging the reasoning and planning capabilities of Large Language Models (LLMs), the application of LLM-based MASs in the field of SE is garnering increasing attention. However, there is no dedicated study that systematically explores the design of LLM-based MASs, including the Quality Attributes (QAs) on which the designers mainly focus, the design patterns used by the designers, and the rationale guiding the design of LLM-based MASs for SE tasks. To this end, we conducted a study to identify the QAs that LLM-based MASs for SE tasks focus on, the design patterns used in the MASs, and the design rationale for the MASs. We collected 94 papers on LLM-based MASs for SE tasks as the source. Our study shows that: (1) Code Generation is the most common SE task solved by LLM-based MASs among ten identified SE tasks, (2) Functional Suitability is the QA on which designers of LLM-based MASs pay the most attention, (3) Role-Based Cooperation is the design pattern most frequently employed among 16 patterns used to construct LLM-based MASs, and (4) Improving the Quality of Generated Code is the most common rationale behind the design of LLM-based MASs. Based on the study results, we presented the implications for the design of LLM-based MASs to support SE tasks. 1. Introduction While traditional Multi-Agent Systems (MASs) face limitations in handling complex decision-making tasks ( 8 ) , Large Language Models (LLMs) can enhance the reasoning ( 12 ) and planning capabilities of MASs ( 23 ) . LLM-based MASs consist of multiple autonomous agents that collaborate through communication and responsibility specialization to tackle complex tasks given by users and simulate problem-solving environments ( 9 ) . LLM-based MASs have demonstrated significant potential in addressing Software Engineering (SE) tasks and have become a focal point of research and practice in SE ( 10 ) . Many researchers have designed LLM-based MASs to address specific SE tasks ( 10 ) . For example, Hong et al. ( 11 ) proposed an LLM-based MAS named MetaGPT, which is capable of automatically developing an entire software system. Jin et al. ( 14 ) proposed an LLM-based MAS named MARE (Multi-Agents Collaboration Framework for Requirements Engineering), which could complete the process of requirements engineering. Effective design is paramount to the success of LLM-based MASs, as it directly shapes key quality attributes such as reliability, maintainability, scalability, and safety, and profoundly influences collaborative behaviors of agents (e.g., volunteering, conformity, and destructive actions), which in turn affect overall task-solving efficiency among agents in MASs ( 6 ) . Therefore, a principled understanding of design rationale is essential not only for constructing high-quality MASs but also for navigating trade-offs among competing quality attributes (e.g., correctness vs. efficiency) ( 15 ) . Despite the increasing interest and adoption of LLM-based MASs for various SE tasks, no existing study has systematically investigated the specific Quality Attributes (QAs), design patterns, and underlying design rationale that inform their construction. To address this gap, our goal is to identify QAs and design patterns that practitioners prioritize when building LLM-based MASs, thereby highlighting best practices and informing future system designs. Moreover, explicitly articulating the design rationale is critical for transparency of design decision-making and design knowledge transfer. In this study, we empirically investigate quality-driven design choices in SE-focused LLM-based MASs. Our findings offer actionable guidance for practitioners and contribute a deeper, evidence-based understanding of how such systems are designed to be more robust, reliable, and effective. Our study results show that: (1) Code Generation is the most common SE task solved by LLM-based MASs among ten identified SE tasks, (2) Functional Suitability is the QA most frequently prioritized by the designers of MASs, (3) Role-Based Cooperation is the most commonly adopted design pattern among all 16 patterns we identified for MASs construction, and (4) Improving the Quality of Generated Code is the predominant rationale behind the design of LLM-based MASs. The analysis of QAs, design patterns, and design rationale was conducted specifically in the context of how LLM-based MASs are designed to complete these SE tasks, which ensures a more coherent logical flow from problem context to architectural analysis. The main contributions of this work: • We collected a comprehensive set of SE tasks solved by LLM-based MASs. Moreover, we identified the key QAs that received the most attention from designers. • We identified design patterns commonly used by designers for constructing LLM-based MASs to address specific SE tasks, and extracted underlying design rationale guiding their development. • We established mapping relationships among SE tasks addressed by LLM-based MASs, QAs considered by designers, design patterns utilized in constructing LLM-based MASs for SE tasks, along with the design rationale. The rest of this paper is structured as follows: Section 2 reviews the related work. Section 3 presents the Research Questions (RQs) and the research process conducted in this study. Section 4 presents the results of this study. Section 5 interprets the study results and discusses their implications. Section 6 outlines the potential threats to validity, and Section 7 concludes this work with future research directions. 2. Related Work In this section, we first review LLM-based MASs specially designed to address SE tasks (Section 2.1 ), and introduce the studies that explore the characteristics of these MASs (Section 2.2 ). Then, we present the recent literature surveys that focus on LLM-based MASs (Section 2.3 ). We compare recent studies with our work to highlight the research gap (Section 2.4 ). 2.1. LLM-based MASs for SE Tasks As LLMs continue to evolve at a rapid pace, SE researchers and practitioners are increasingly eager to employ the reasoning and decision-making abilities of LLMs to support SE tasks. To better exploit the potential of LLMs in complex, multi-step SE workflows (particularly those requiring coordination), LLM-based MASs have been designed to address SE tasks, such as requirements engineering, code generation, and fault localization. Arora et al. ( 1 ) developed an LLM-based MAS aimed at enhancing the efficiency and accuracy of requirements engineering tasks. This MAS reveals the significant potential of applying LLMs to requirements elicitation, analysis, specification, and validation. Li et al. ( 16 ) proposed MAAD , a knowledge-driven MAS specifically built for automated architecture design. MAAD assigns the architecture design task to four role-dedicated agents with external knowledge bases, and the four agents collaboratively generate architecture design by decomposing a given Software Requirements Specifications (SRS) into corresponding artifacts. Dong et al. ( 7 ) designed an LLM-based MAS based on a self-collaboration framework to achieve high-quality code generation. Their framework enables the MAS to solve complex repository-level tasks that are not readily solved by a single LLM agent. Pan et al. ( 19 ) proposed a novel optimization pipeline for LLM-based MASs based on self-generated textual feedback, named CodeCoR , which has a

Read the complete text