Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems

Ye Yu, Heming Liu, Haibo Jin, Xiaopeng Yuan, Peng Kuang, Haohan Wang · 2026

Read Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems by Ye Yu, Heming Liu, Haibo Jin, Xiaopeng Yuan, Peng Kuang, Haohan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21794.txt.

Introduction

Yu and five colleagues treat communication between language-model agents as something to learn, not a fixed interface. Their DiffMAS framework trains latent communication jointly with multi-agent reasoning, using parameter-efficient supervised training over latent trajectories. Agents thus learn together how to encode and interpret information across turns. On maths reasoning, scientific questions, code generation, and commonsense benchmarks, DiffMAS beats single-agent inference, text-based multi-agent systems, and earlier latent methods, scoring 26.7 percent on AIME24 and about 20 percent on GPQA-Diamond.

Catalog record

CREATORYe Yu, Heming Liu, Haibo Jin, Xiaopeng Yuan, Peng Kuang, Haohan Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53756
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems Ye Yu, Heming Liu, Haibo Jin, Xiaopeng Yuan, Peng Kuang, Haohan Wang Abstract Multi-agent systems built on large language models have shown strong performance on complex reasoning tasks, yet most work focuses on agent roles and orchestration while treating inter-agent communication as a fixed interface. Latent communication through internal representations such as key–value caches offers a promising alternative to text-based protocols, but existing approaches do not jointly optimize communication with multi-agent reasoning. Therefore we propose DiffMAS, a training framework that treats latent communication as a learnable component of multi-agent systems. DiffMAS performs parameter-efficient supervised training over multi-agent latent trajectories, enabling agents to jointly learn how information should be encoded and interpreted across interactions. Experiments on mathematical reasoning, scientific QA, code generation, and commonsense benchmarks show that DiffMAS consistently improves reasoning accuracy and decoding stability over single-agent inference, text-based multi-agent systems, and prior latent communication methods, achieving +26.7% on AIME24, +20.2% on GPQA-Diamond, and consistent gains across reasoning benchmarks. 1 Introduction Multi-agent systems (MAS) built on large language models have emerged as a powerful paradigm for solving complex reasoning tasks, including mathematics, programming, and commonsense problem solving ( 42 ; 43 ) . Their effectiveness is commonly attributed to two complementary factors: (i) system structure and agent role assignment ( 1 ) , which determine how agents are organized and specialized, and (ii) inter-agent communication ( 4 ; 29 ) , which governs how intermediate reasoning states and information are exchanged. By decomposing complex tasks ( 38 ; 16 ; 27 ; 18 ) , multi-agent systems consistently outperform single-agent baselines. As multi-agent language systems continue to mature, recent research has increasingly explored how to systematically improve their performance through optimization ( 36 ; 42 ; 43 ; 1 ) . Existing efforts have primarily focused on improving agentic capabilities through fine-tuning or prompting, as well as refining workflow design through better role assignment, planning strategies, and execution structures ( 30 ; 40 ; 43 ; 1 ) . These advances have led to more capable agents and more effective system structures. Figure 1: In Stage I, agents 1 to K–1 sequentially construct a shared KV trace by prefilling the existing cache and appending newly generated KV segments without gradient updates. The accumulated KV trace serves as a latent communication medium across agents. In Stage II, the final agent performs autoregressive decoding on the prefilled KV cache. Cross-attention over the KV trace produces hidden states, which are projected through the LM head to generate tokens. Supervised fine-tuning is applied using cross-entropy loss, and gradients are backpropagated to update only the LoRA parameters of the final agent while keeping the backbone model frozen. However, such improvements largely treat multi-agent systems as modular compositions of optimized components, rather than systems whose interactions can themselves be learned ( 46 ; 29 ; 4 ) . A natural next step is therefore to explore whether multi-agent systems can be optimized more holistically, allowing not only agent behaviors and workflows, but also their interactions to improve through learning ( 26 ; 21 ) . Achieving this vision requires moving beyond fixed system interfaces toward interaction mechanisms that can be optimized together with agentic reasoning. As different tasks can place different demands on how intermediate reasoning states should be represented and preserved, effective communication strategies need to adapt to the task rather than relying on fixed protocols ( 4 ; 29 ) . In this context, enabling communication to become a learnable component represents a critical step towards more fully optimizable multi-agent reasoning systems. A key obstacle to making communication learnable lies in the discrete nature of how agents currently exchange information. In most existing multi-agent systems, intermediate reasoning results must be decoded into natural language before being transmitted to downstream agents ( 3 ; 16 ; 3 ) . While this design aligns naturally with the token-based interfaces of language models, it introduces a fundamental limitation: discrete message passing creates optimization boundaries between agents ( 31 ; 8 ; 25 ; 14 ) . This limitation is particularly restrictive as the performance of the system depends critically on how well intermediate reasoning results are preserved and interpreted across agents ( 5 ; 37 ) . More fundamentally, this reflects a mismatch between the continuous internal representations used during reasoning and the discrete messages used for inter-agent exchange ( 41 ; 13 ; 9 ; 47 ) . To make communication truly learnable and enable more end-to-end optimization of multi-agent systems, inter-agent information exchange must therefore operate over continuous representations that can support differentiable information flow. Key–Value (KV) caches provide a natural mechanism for enabling such differentiable communication ( 34 ) . As continuous internal representations produced by transformer attention layers, KV states can be directly reused by downstream agents without requiring discrete decoding or re-encoding ( 17 ) . This allows intermediate reasoning representations to be transmitted through a continuous channel that remains compatible with gradient-based learning. By treating KV caches as a latent communication interface, gradients can propagate not only within individual agents but also across agent boundaries, allowing communication behaviors to be optimized together with the underlying model ( 26 ; 46 ; 21 ) . Therefore, we propose DiffMAS, a framework that treats KV-mediated latent communication as a trainable interface between agents and enables SFT-like training over multi-agent interaction trajectories. As illustrated in Figure 1 , DiffMAS operates in two stages. In Stage I, upstream agents sequentially construct a shared KV trace by prefilling existing KV caches and appending newly generated KV segments, forming a continuous latent communication channel. In Stage II, the final agent performs autoregressive decoding conditioned on the accumulated KV trace, and supervised training is applied to optimize communication jointly with downstream reasoning. By training over full multi-agent interaction trajectories, the framework enables the model to learn communication with agentic reasoning. Our contributions are summarized as follows: • We formulate inter-agent communication as an optimization problem in multi-agent language systems, showing that treating communication as a fixed interface prevents end-to-end improvement of how information is transmitted across agents. • We propose DiffMAS, a supervised training framework that treats KV-mediated latent communication as a differentiable and task-adaptive interface between agents. • We demonstrate through extensive experiments that learning latent communication leads to more stable reasoning trajectories and improves performance, yielding up to +26.7% improvement on AIME24 and consistent gains across reasoning benchmarks. 2 Related Works 2.1 Multi-Agent LLM Systems LLM-based multi-agent systems improve complex reasoning by decomposing problems across specialized agents and structured interaction workflows. Prior work has explored improving agent capabilities and system design through role engineering, automated workflow construction, and architecture search ( 38 ; 18 ; 27 ; 16 ; 1 ; 43 ; 42 ) . Other studies focus on improving coordination through execution policies or self-configuration ( 4 ; 35 ; 26 ; 21 ) . Despite these advances, most multi-agent systems rely on natural language as a universal communication interface ( 43 ; 3 ) . While flexible, text-based communication requires internal reasoning states to be serialized into discrete tokens ( 37 ) , which can limit information fidelity and prevent communication from being optimized

Read the complete text