Distributed Multi-Agent Coordination Using Multi-Modal Foundation Models
Saaduddin Mahmud, Dorian Benhamou Goldfajn, Shlomo Zilberstein · 2025
Read Distributed Multi-Agent Coordination Using Multi-Modal Foundation Models by Saaduddin Mahmud, Dorian Benhamou Goldfajn, Shlomo Zilberstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14189.txt.
Introduction
Saaduddin Mahmud, Dorian Benhamou Goldfajn, and Shlomo Zilberstein present a 2025 paper on coordinating teams of agents with help from multimodal foundation models. Distributed constraint optimization is a standard framework for such coordination, but building the problem by hand is laborious. Their extension lets large models that read images and text generate the constraints automatically from visual and written instructions.
They then define a spectrum of solver agents. At one end, a neuro-symbolic agent delegates only some decisions to the model. At the other, a fully neural agent leans on the model for everything. Three new tasks test the designs against leading language and vision models. The authors weigh each archetype's strengths and weaknesses. They close by connecting the work to open challenges in multi-agent coordination research.
Catalog record
| CREATOR | Saaduddin Mahmud, Dorian Benhamou Goldfajn, Shlomo Zilberstein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distributed Multi-Agent Coordination Using Multi-Modal Foundation Models Saaduddin Mahmud, Dorian Benhamou Goldfajn, Shlomo Zilberstein Abstract Distributed Constraint Optimization Problems (DCOPs) offer a powerful framework for multi-agent coordination but often rely on labor-intensive, manual problem construction. To address this, we introduce VL-DCOPs, a framework that takes advantage of large multimodal foundation models (LFMs) to automatically generate constraints from both visual and linguistic instructions. We then introduce a spectrum of agent archetypes for solving VL-DCOPs: from a neuro-symbolic agent that delegates some of the algorithmic decisions to an LFM, to a fully neural agent that depends entirely on an LFM for coordination. We evaluate these agent archetypes using state-of-the-art LLMs (large language models) and VLMs (vision language models) on three novel VL-DCOP tasks and compare their respective advantages and drawbacks. Lastly, we discuss how this work extends to broader frontier challenges in the DCOP literature. 1 Introduction Distributed Constraint Optimization Problems (DCOPs) Yokoo et al. (1998) are a widely studied framework that has previously been shown to be effective in various applications, including smart homes, robotic and drone coordination, surveillance, economic dispatch, network coordination, and disaster response Maheswaran et al. (2004b) ; Farinelli et al. (2014) ; Zivan et al. (2015) ; Fioretto et al. (2017) ; Mahmud et al. (2020b) . However, in many of these cases, the problems are static and manually modeled, reducing their applicability in changing scenarios. Although some previous works have considered a dynamic formulation of DCOPs Petcu and Faltings (2005b) , these approaches still require manual defining of a dynamics model, which limits their adaptability. Another approach to mitigating this issue is to use preference elicitation to handle partially defined constraints, allowing agents to query humans to reduce uncertainty over time Tabakhi et al. (2017) ; Tabakhi et al. (2019) ; Le et al. (2018) ; Tsouros et al. (2024) ; Zivan et al. (2024) . Even then, initial constraints must be established beforehand, queries often use language templates, and answers to queries take the form of numeric constraint cost values, which may not be a natural or convenient way for humans to communicate their preferences Tversky et al. (1982) . Archetypes Description A0: Classic Algorithmic DCOP Agents Agents use classical symbolic algorithms such as DPOP, DSA, or MGM to solve DCOPs. A1: Instruction Following DCOP Agents Symbolic code executing classical DCOP algorithms leverages LFMs to interpret human instructions and facilitate key algorithmic decisions. A2: Conversational DCOP Agents Similar to A1, but with a more advanced use of LFMs. A2 employs LFMs to engage in conversations with humans and other agents, enabling them to learn and negotiate constraint costs more effectively. A3: Algorithm Simulating DCOP Agents Rather than directly running a classical DCOP algorithm, A3 models the iterative coordination process as a sequential decision-making problem and uses an LFM-based policy to solve it. This allows A3 to simulate any coordination algorithm while also handling exceptional cases. Table 1: Spectrum of VL-DCOP Agents. To address these challenges, we introduce visual-linguistic instruction-based DCOPs, VL-DCOPs, an extended DCOP framework that models the interaction between humans and agents to specify constraints to solve a coordination task (Fig. 1 ). As a motivating example, consider a group of smartphone-based LFM-driven agents tasked with scheduling meetings for their owners. Initially, the agents use the instructions and contact lists provided to establish constraints with other agents. The LFM agents then coordinate by asking questions about schedules, reducing any uncertainty about the underlying cost structure. Since these agents rely on natural language, human responses can go beyond purely numerical feedback, incorporating explanations and other contextual details. Once constraints are established, the agents initiate a suitable optimization protocol until a mutually agreeable schedule is found. Figure 1: Spectrum of VL-DCOP Agents solving language-conditioned weighted graph coloring problems. To implement the VL-DCOPs we design a spectrum of autonomous agent archetypes (Table 1 ) each utilizing a large multi-modal foundation model Bhambri et al. (2024) to a varying degree. At one end of the spectrum, we have A1 agents, a form of neuro-symbolic agent Sheth and Roy (2023) that uses these LFMs to parse the human instruction to make key algorithmic decisions such as action selection. However, the agent’s workflow is still dictated by the classical algorithm. In this paper, we propose Foundation Model Centered DSA, FMC-DSA , a variant of the popular DSA Fitzpatrick and Meertens (2003) algorithm for this archetype. Next, we present a more involved neuro-symbolic agent, A2 , to address issues related to uncertainty about constraint cost due to natural ambiguity in linguistic communication and conflicts that may arise due to varying preferences among humans. To that end, we propose Cooperative Preference Aggregation ( CoPA ), an algorithm that uses LFMs to learn a good representation of the constraint function, and then uses a classical algorithm to solve the problem. Compared to A1, A2 provides a more interactive solution through conversation with humans and other agents. Finally, we present A3 , a general agent that in theory can simulate any algorithm depending on the instruction provided in the prompt. We model A3 as a sequential decision-making agent that uses an LFM-based (in-context) policy to solve the coordination problem. Unlike the previous two agents, for A3, we do not hard code any particular optimization procedure, making it a fully neural agent. The key motivation for this agent archetype is to provide flexibility to handle exceptional situations that may arise during optimization. To evaluate these agents, we propose three new benchmarks. The first two are based on classical distributed weighted graph coloring problems Fioretto et al. (2016) . We design a linguistic (LDGC) and a visual (VDGC) variant of this problem to evaluate these agents. The third benchmark is based on more realistic meeting scheduling problems Fioretto et al. (2016) . To build these agents we used pre-trained LLMs and VLMs such as LLAMA Dubey et al. (2024) , Qwen Team (2024) , and GPT-4o Hurst et al. (2024) . Our analysis provides insight into the capability of current LFMs in distributed coordination and the advantages and disadvantages of each archetype. To our knowledge, our work is the first of its kind study in the DCOP literature and opens the door to many future research directions with interesting potential in explainability Roie Zivan (2022) , privacy Tassa et al. (2019) , and exception handling such as message delay Rachmut et al. (2024) ; Matsui (2024) . Further, with the rapid progress of GPU-based edge devices (such as Nvidia Jetson Orin Nano Super Su (2024) ), and the ever-expanding capability of the frontier LFMs Meta (2024) , we expect a practical deployment of A3-like agents to be operational in the near future. 2 Related Work DCOPs Framework As DCOPs must be designed manually, they face flexibility challenges in dynamic environments and scenarios where critical information is missing or difficult to obtain. Several extensions have been proposed to address this problem, including D-DCOP and U-DCOP Petcu and Faltings (2005b) ; Léauté and Faltings (2011) . Dynamic-DCOPs (D-DCOPs) account for DCOPs that evolve over time, although these adaptations remain manually specified. U-DCOP offers a different perspective by employing probability distributions over constraint costs—manually learned from data—to enhance the framework’s adaptability. In contrast, the proposed VL-DCOP can dynamically incorporate