OMAC: A Broad Optimization Framework for LLM-Based Multi-Agent Collaboration
Shijun Li, Hilaf Hasson, Joydeep Ghosh · 2025
Read OMAC: A Broad Optimization Framework for LLM-Based Multi-Agent Collaboration by Shijun Li, Hilaf Hasson, Joydeep Ghosh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11765.txt.
Introduction
Shijun Li, Hilaf Hasson, and Joydeep Ghosh introduce OMAC in a 2025 paper on teams of AI agents. Such teams pair several large language models that talk and work together on hard tasks like writing code. Most teams today are built by hand, with little method behind their design.
OMAC treats the design of a team as something to be tuned step by step. The authors name five sides of a team that can be tuned, from each agent's job to how the group is wired. They give methods to tune one side at a time or several at once. In their tests, teams tuned this way beat recent rival methods.
Catalog record
| CREATOR | Shijun Li, Hilaf Hasson, Joydeep Ghosh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11765
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OMAC: A Broad Optimization Framework for LLM-Based Multi-Agent Collaboration Shijun Li, Hilaf Hasson, Joydeep Ghosh Abstract Agents powered by advanced large language models (LLMs) have demonstrated impressive capabilities across diverse complex applications. Recently, Multi-Agent Systems (MAS), wherein multiple agents collaborate and communicate with each other, have exhibited enhanced capabilities in complex tasks, such as high-quality code generation and arithmetic reasoning. However, the development of such systems often relies on handcrafted methods, and the literature on systematic design and optimization of LLM-based MAS remains limited. In this work, we introduce OMAC , a general framework designed for holistic optimization of LLM-based MAS. Specifically, we identify five key optimization dimensions for MAS, encompassing both agent functionality and collaboration structure. Building upon these dimensions, we first propose a general algorithm, utilizing two actors termed the Semantic Initializer and the Contrastive Comparator, to optimize any single dimension. Then, we present an algorithm for joint optimization across multiple dimensions. Extensive experiments demonstrate the superior performance of OMAC on code generation, arithmetic reasoning, and general reasoning tasks against state-of-the-art approaches. A demo code is available at: https://github.com/xiwenchao/OMAC-demo . 1 Introduction Autonomous agents leveraging advanced large language models (LLMs) have recently shown significant potential in addressing complex problems and executing diverse tasks [ 1 , 2 , 3 , 4 , 5 ] . Recently, employing multiple collaborating agents has emerged as a prominent research direction for overcoming the inherent limitations of single-agent approaches in handling tasks within sophisticated environments [ 6 , 7 , 8 , 9 , 1 , 10 , 11 ] . An advanced collaborative paradigm also involves a multi-step process wherein agents sequentially resolve tasks by utilizing outputs from preceding steps. This Multi-Agent Systems (MAS) approach has already yielded substantial advancements in various applications, such as code generation [ 6 , 12 , 1 ] , reasoning [ 7 , 13 , 14 ] , and decision making [ 15 , 16 ] . However, the design of existing MAS predominantly relies on hand-crafted strategies. Regarding agent construction, prior studies typically employ methods based either on human expertise [ 17 , 2 , 18 ] or LLM generation [ 8 , 19 , 20 ] . Although some research has explored optimizing the functionality of a single agent via techniques such as fine-tuning [ 21 , 22 , 23 ] or prompt-tuning [ 24 , 25 , 18 , 26 ] , these efforts don’t explicitly address agent optimization within the context of MAS involving multi-step collaborative processes. Concerning collaboration structures, existing approaches typically define fixed architectures tailored to specific application scenarios, encompassing cooperative tasks (e.g., code generation, decision-making) [ 27 , 28 , 29 , 15 ] and competitive tasks (e.g., debate) [ 7 , 13 , 14 , 30 ] . However, the design of these structures predominantly relies on human prior knowledge or particular empirical findings, thereby limiting their generality and the flexibility required for autonomous optimization towards more effective structural configurations. A recent study, DyLAN [ 31 ] , represents a promising effort toward optimizing collaboration structures in MAS. However, its approach centers solely on the unsupervised optimization of agent team composition, employing metrics derived from preliminary trials and LLM-based judgments, rather than utilizing supervised signals for validation and optimization. To address these challenges, we introduce OMAC , a comprehensive framework designed for the integrated optimization of MAS. Specifically, we identify and summarize five key optimization dimensions pertaining to both agent functionality and collaboration structure. The functional dimensions include optimizing existing agents and constructing new agents tailored for collaboration. Regarding the structural aspect, we optimize 1 1 1 Note that we use the word “optimize” in a colloquial way to signify improvement, rather than indicating a mathematical guarantee of an optimal solution. LLM-based controllers to manage collaboration structures, encompassing decisions on the overall candidate agent teams, dynamic agent selection for individual collaboration steps, and the mechanisms governing inter-agent communication. Figure 1: OMAC optimization workflow for a single dimension. We first propose a general algorithm employing two LLM-powered actors, the Semantic Initializer and the Contrastive Comparator , to optimize any individual dimension. Specifically, the Semantic Initializer leverages the knowledge and reasoning capabilities of LLMs to generate an initial collection of agents or controllers corresponding to the dimension being optimized by exploring the relevant semantic space. Subsequently, each initialized agent or controller undergoes evaluation within the multi-agent collaboration process using training data to determine its performance score. A positive-negative pair (high-performing and low-performing) is then sampled based on these scores. The Contrastive Comparator then performs contrastive reasoning on this pair to identify factors underlying the performance gap, subsequently generating a refined agent or controller. This refined agent/controller is then re-evaluated via the collaboration process, and the cycle of sampling, contrastive comparison, and refinement is iterated. Beyond single-dimension optimization, we further propose an algorithm for iterative, joint optimization across multiple dimensions, enabling further MAS enhancement through synergistic optimization of agents and controllers. More implementation details of the actors and optimization algorithm are illustrated in Section 3.2 . We conduct extensive experiments across diverse tasks, including general reasoning, code generation, and arithmetic reasoning. The results demonstrate that OMAC consistently outperforms strong baselines when optimizing each of the five individual dimensions in most scenarios. Furthermore, joint optimization across multiple dimensions is shown to further enhance performance significantly. Our key contributions can be summarized as follows: • We introduce OMAC, a general supervised framework for optimizing multi-agent systems engaged in multi-step collaboration. To achieve this, we identify five key optimization dimensions covering both agent functionality and collaboration structure. • We propose a general algorithm, utilizing two actors termed the Semantic Initializer and the Contrastive Comparator, for optimizing each of the five dimensions. Furthermore, we present an algorithm enabling iterative, joint optimization across multiple dimensions. • We evaluate OMAC on benchmark tasks including general reasoning, code generation, and arithmetic reasoning. Empirical results demonstrate that OMAC significantly outperforms existing approaches through the automated optimization of functional and structural MAS designs. 2 Problem Definition We study the optimization of MAS within the context of multi-step collaboration process. The definition of agents and the fundamental collaboration workflow are as follows: Definition 1: Agents. An LLM-powered agent is governed by natural language prompts to generate solutions for completing a given task. Formally, we define an agent a as a function: \mathcal{A}:\mathcal{P}\times\mathcal{I}\rightarrow\mathcal{O} . Here, p\in\mathcal{P} represents the instruction prompt defining the role and functionality of the agent. In an in-context learning setting, the prompt may also include few-shot examples as expected input-output pairs to guide the agent’s behavior. The input i\in\mathcal{I} typically comprises the query information, task description, and potentially instructions or solutions from other agents in MAS. The output o\in\mathcal{O} generally encompasses the agent’s generated analyses, suggestions, and solutions to the given task. Definition 2: Multi-Step Collaboration. In this work, we consider multiple agents collaborating within a multi-step procedure. Typically, each step s_{t}\in\mathcal{S} involves a subset of agents from the overall team, denoted as \{a_{t,1},a_{t,2},...,a_{t,n}\}\subseteq s_{t} . This multi-step approach aims to enhance problem-solving by enabling agents at each step to leverage solutions and outcomes produced by agents in preceding steps. For instance, when solving a complex mathematical problem, agents in the initial step may first decompose the problem