Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry

Run Peng, Ziqiao Ma, Amy Pang, Sikai Li, Zhang Xi-Jia Yingzhuo Yu Cristian-Paul Bara Joyce Chai · 2025

Read Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry by Run Peng, Ziqiao Ma, Amy Pang, Sikai Li, Zhang Xi-Jia Yingzhuo Yu Cristian-Paul Bara Joyce Chai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25595.txt.

Introduction

Run Peng led seven co-authors studying agent collaboration. Ziqiao Ma, Amy Pang, and Sikai Li helped. Zhang Xi-Jia, Yingzhuo Yu, and Cristian-Paul Bara joined. Joyce Chai closes the eight-author team. Their 2025 paper studies contact under information asymmetry. Language model agents often plan actions toward goals. Collaboration abilities between agents remain poorly explored.

Agents differ in knowledge and skills. They must work jointly. The authors extend Einstein Puzzles into a tabletop game. Two language agents reason, communicate, and act jointly. Spatial and relational constraints must be satisfied. A fine-tuning plus verifier approach equips agents. Various contact strategies and verification signals are tested. Aligned contact proves critically major for collaboration. Information-seeking and providing capabilities both help agents. Agents without contact still reached high results surprisingly. Further study revealed weak rule understanding. Human evaluators trusted those silent agents less. Environment verifiers enhanced rule comprehension and task completion.

Catalog record

CREATORRun Peng, Ziqiao Ma, Amy Pang, Sikai Li, Zhang Xi-Jia Yingzhuo Yu Cristian-Paul Bara Joyce Chai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry Run Peng, Ziqiao Ma, Amy Pang, Sikai Li, Zhang Xi-Jia Yingzhuo Yu Cristian-Paul Bara Joyce Chai Abstract While Large Language Model (LLM) agents are often approached from the angle of action planning/generation to accomplish a goal (e.g., given by language descriptions), their abilities to collaborate with each other to achieve a joint goal are not well explored. To address this limitation, this paper studies LLM agents in task collaboration, particularly under the condition of information asymmetry, where agents have disparities in their knowledge and skills and need to work together to complete a shared task. We extend Einstein Puzzles, a classical symbolic puzzle, to a table-top game. In this game, two LLM agents must reason, communicate, and act to satisfy spatial and relational constraints required to solve the puzzle. We apply a fine-tuning-plus-verifier framework in which LLM agents are equipped with various communication strategies and verification signals from the environment. Empirical results highlight the critical importance of aligned communication, especially when agents possess both information-seeking and -providing capabilities. Interestingly, agents without communication can still achieve high task performance; however, further analysis reveals a lack of true rule understanding and lower trust from human evaluators. Instead, by integrating an environment-based verifier, we enhance agents’ ability to comprehend task rules and complete tasks, promoting both safer and more interpretable collaboration in AI systems. https://github.com/Roihn/EinsteinPuzzles 1 Introduction In recent years, there has been growing interest in Large Language Model (LLM) agents (e.g., Web-Agents) and their diverse applications ( Wang et al. 2024a ) . While much of the current work focuses on action planning or goal completion in LLM agents ( Ahn et al. 2022 ; Durante et al. 2024 ; Song et al. 2023 ) (e.g., based on natural language instructions), their ability to collaborate with one another toward a shared goal remains underexplored. This paper addresses this gap by studying LLM agents in the context of task collaboration, particularly under conditions of information asymmetry. Information asymmetry is a fundamental and pervasive feature of human interaction. In daily life, we often possess different knowledge, perspectives, or intention. We must reason about others’ knowledge and beliefs and coordinate our differences in order to collaborate ( Ma et al. 2023 ) . As LLM agents become increasingly integrated into real-world workflows — not just to perform tasks independently, but to act as human proxies and collaborators — it becomes essential to examine how well these agents can coordinate under asymmetric information, and what mechanisms might enhance their collaborative capabilities in such settings. To this end, we adapt Einstein Puzzles ( Groza 2021 ) , a classical logic problem, into a tabletop environment where two agents must solve spatial and relational constraints, despite having partial, asymmetric information. Using a fine-tuning–plus–verifier framework, we equip LLM agents with different communicative abilities (e.g., asking, sharing, or both) and study their collaborative behaviors under various configurations. Furthermore, we introduce an environment-based verifier to guide and evaluate agent decisions. This verifier leverages environmental signals to determine whether the proposed actions and known constraints are consistent, mirroring the implicit feedback mechanisms in interactive test-and-trials. It is training-free, lightweight, and broadly applicable across various environments. We examine whether this simple yet generalizable approach can greatly improve collaboration among LLM agents. Our empirical results have shown that under information asymmetry, LLM agents with both information-seeking and -providing capabilities collaborate most effectively. Meanwhile, mismatched agent communicative abilities leads to significant performance degradation, highlighting the importance of aligned interaction protocols. Additionally, through detailed error analysis, we show that environment-based verification, using naturally available feedback without additional training, offers a simple but powerful mechanism to improve agent performance on both task completion and understanding. Finally, our human study reveals a gap between task efficiency and human preference: participants favor agents that proactively share information, even if such agents are less optimal in task completion. These findings indicate the need for communication-aware and interpretable design in LLM-based collaborative systems. 2 Related work Figure 1: Illustration of our collaborative game. Each game features a final goal (top-left), where objects are assigned to specific goal bins (e.g. toy bear to top-left bin). Placement constraints are generated based on this final goal and are distributed to the two players. At the start, objects are randomly positioned in front of the players. Players must collaborate and communicate effectively to reason the final goal and accordingly complete the placement. Collaboration among LLM Agents The field of multi-agent coordination and communication has a long-established history ( Albrecht and Stone 2018 ; Gronauer and Diepold 2022 ; Stone and Veloso 2000 ) , with applications increasingly extending to human-AI collaboration ( Carroll et al. 2019 ; Puig et al. 2020 ) . Recently, multi-agent systems built upon Large Language Models (LLMs) ( Guo et al. 2024 ; Tran et al. 2025 ) have pushed the boundaries of collaborative intelligence in a wide range of downstream tasks, including collaborative coding ( Gao et al. 2024 ; Hong et al. 2024 ; Qian et al. 2024 ) , social simulation ( Li et al. 2023 ; Wu et al. 2024 ; Yang et al. 2024 ; Zhou et al. 2024c ) , and problem-solving ( Chen et al. 2023 ; Qian et al. 2025 ; Wang et al. 2024c ; Zhang et al. 2024a ) . However, most existing work focuses on settings with information transparency ( Chen et al. 2023 ; Gao et al. 2024 ; Li et al. 2023 ; Wu et al. 2024 ; Yang et al. 2024 ) , one-way communication ( Qian et al. 2025 ) , or information asymmetry within asymmetric role assignments ( Hong et al. 2024 ; Qian et al. 2024 ; Zhou et al. 2024c ) . In contrast, this work centers on LLM agents operating under information asymmetry in symmetric, collaborative roles, as well as involves human studies to assess real-world human-AI interaction. Liu et al. 2024a studied autonomous agents for collaborative tasks under information asymmetry, while there were no environments involved. As echoed in recent efforts ( Liu et al. 2024b ; Zhou et al. 2024b ) , addressing collaboration under information asymmetry—especially in human-AI contexts—remains a core challenge for LLM-based systems. Test-time Compute in LLM Recent works have demonstrated the effectiveness of leveraging additional computational resources at inference time to improve response quality in LLMs. One line of research introduces a verification model to evaluate the correctness and utility of generated responses. Typically, this involves sampling multiple responses from an LLM and applying a best-of- n strategy ( Charniak and Johnson 2005 ; Cobbe et al. 2021 ) , where a verifier model selects the most appropriate response. Such approaches have been widely adopted across domains including mathematics ( Cobbe et al. 2021 ; Lifshitz et al. 2025 ; Lightman et al. 2023 ; Wang et al. 2024b ; Yao et al. 2023 ) , code generation ( McAleese et al. 2024 ; Wang et al. 2025 ) , and web navigation ( Koh et al. 2024 ; Putta et al. 2024 ) . We refer readers to ( Guan et al. 2024 ; Zhang et al. 2025 ) for comprehensive reviews of

Read the complete text