LM Agents for Coordinating Multi-User Information Gathering
Harsh Jhamtani, Jacob Andreas, Benjamin Van Durme · 2025
Read LM Agents for Coordinating Multi-User Information Gathering by Harsh Jhamtani, Jacob Andreas, Benjamin Van Durme (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12328.txt.
Introduction
Jhamtani, Andreas, and Van Durme introduce PeopleJoin, a test bed for AI agents that gather facts through teammates. A request arrives, and the agent must find who can help. It then chats with those people to collect what they know. Last, it compiles an answer or summary for the requester. Tasks span questions over tables and the drafting of documents. The needed facts are spread across fake organizations of 2 to 20 users.
Several known agent designs were run on the benchmark. Scores covered both accuracy and cost of completion. The setup mirrors real office hunts for scattered knowledge. It also surfaces fresh questions about how agents should pick and query helpers.
Catalog record
| CREATOR | Harsh Jhamtani, Jacob Andreas, Benjamin Van Durme |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12328
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LM Agents for Coordinating Multi-User Information Gathering Harsh Jhamtani, Jacob Andreas, Benjamin Van Durme Abstract This paper introduces PeopleJoin , a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PeopleJoin agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PeopleJoin comprises two evaluation domains: PeopleJoin-QA , focused on questions about tabular data, and PeopleJoin-DocCreation , focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic “organizations” of 2–20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PeopleJoin . 1 1 1 Code and data can be found at https://github.com/microsoft/peoplejoin/ 1 Introduction Figure 1: A sequence diagram illustrating a conversation in PeopleJoin framework, where Alice issues a request to her agent. Documents available to Alice’s agent are insufficient to answer the user request. The agent uses a people search tool, after which it decides what subset of people to contact, in which order, what questions to pose, etc. The temporal ordering of tool calls and message exchanges is denoted by #i. In today’s fast-paced and interconnected world, effective collaboration is essential for achieving complex tasks and making informed decisions Papachristou et al. (2023) ; Gemp et al. (2024) . Many decision-making, content creation, and information-gathering tasks require collecting information from multiple people. For example, preparing a list of interns across teams in an organization by reaching out to the leader of each team; preparing a newsletter for project updates might necessitate coordinating with multiple contributors; identifying a suitable time to meet might require several rounds of negotiations Lin et al. (2024) . Identifying what information is available, judiciously determining who to contact, asking precise questions, and compiling research results can be a challenging and time-consuming process—especially when real-time interaction between team members is difficult to coordinate. At the same time, recent large language models (LLMs), such as GPT-4 OpenAI (2023) , Phi-3 Abdin et al. (2024) , LLaMa Touvron et al. (2023) , and Gemini Team et al. (2023) , are becoming a crucial building block in developing automated agents that can assist human users with complex tasks Xi et al. (2023) ; Wang et al. (2024) ; Butler et al. (2023) . These tasks include chat applications for assisting individual users with searching and summarizing information (such as in Microsoft Copilot Chat 2 2 2 https://copilot.microsoft.com/ ), and even supporting these users in workplace decision-making ( Butler et al., 2023 ; Kim and Hsu, 2024 ) . Could these agents be extended to improve collaboration among multiple users? In this paper, we introduce PeopleJoin , an evaluation framework for studying effectiveness of LLM-powered agents to assist with multi-user collaboration tasks. Each PeopleJoin task takes place within a fictitious organization with 2–20 employees, some of whom possess a collection of documents necessary to solve some task. One of the users (the initiating user ) communicates the task to an agent (Fig. 1 ). Agents have direct access to the initiating user’s documents, and can engage in conversations with other users to gather relevant information. They must rely on limited descriptions of other users, and potentially previous interactions, to determine who to contact for a given task. PeopleJoin comprises two families of tasks: PeopleJoin-QA and PeopleJoin-DocCreation , derived from the spider ( Yu et al., 2018 ) and multinews ( Fabbri et al., 2019 ) datasets respectively. It evaluates agents’ ability to answer questions involving complex relational reasoning and document summarization. Our initial benchmark release also includes reference agent implementations based on popular prompting and orchestration strategies, and a suite of evaluation metrics. We report evaluation results using Phi-3-medium Abdin et al. (2024) , GPT-4-turbo and GPT-4o OpenAI (2023) language models to implement these agents. Our results indicate that LM-powered agents can struggle to coordinate with multiple users to correctly address information seeking and document authoring requests. Major research questions remain around how to optimally determine which people to contact and in what order, how to ask high-quality questions, and how to learn and adapt to the structure of an organization. PeopleJoin thus provide a test-bed for building AI-driven systems that can enhance human collaboration, and will also enable future work on learning from interaction, distributing tasks equitably, and maintaining user privacy in such agentic systems. 2 Challenges in Effectively Steering Multi-User Information Gathering The problem of answering user queries by synthesizing information distributed across heterogeneous data sources is most often studied through the lens of database systems Zaniolo (1997) . Work on query optimization and federated databases Sheth and Larson (1990) has sought to address the specific question of how to efficiently answer structured queries without access to a centralized knowledge store. The problem we study in PeopleJoin may be viewed as a generalization of this task to the setting where the relevant information is possessed by people , not structured knowledge bases, and must be obtained via conversation rather than structured queries. An agent to help a user with such requests must address several challenges: • Information fragmentation: In a typical organization, information is often siloed across multiple users, because of differing roles and responsibilities. Some requests may require gathering information from multiple people. • Partial observability: To gather this information, it is often necessary to first determine which collaborators hold relevant information, under incomplete and potentially imprecise information of what information each collaborator might have. Agents for collaborative decision-making might have to engage in multi-turn conversations with various users, refining and adapting requests as needed. • Communication costs: Requests for information require human effort to process and answer; effective collaboration requires efficient communication: effective agents should judiciously send information requests to other collaborators, and avoid asking questions that are likely to be unanswerable. • Complex reasoning and planning: Efficient communication requires reasoning: establishing what information is available in accessible documents, dynamically predicting which collaborators are likely to have relevant information for specific questions, identifying the best order in which to ask these questions, and re-planning based on collaborators’ responses. Below, we present a benchmark for evaluating these skills. 3 Data Each PeopleJoin domain comprises a set of organizations . Each organization contains a set of collaborators , and each collaborator has privileged access to a set of documents . The benchmark provides LLM-based simulators for each collaborator, a search interface that can be used to find collaborators, and a messaging interface that can be used to ask collaborators about their documents. Then an agent must take as input a query from one collaborator, use the search and messaging interfaces to interact with other collaborators, and finally return an answer to the originator. Drawing analogies between multi-user collaboration tasks and existing multi- data-source tasks commonly studied in NLP, we develop PeopleJoin by re-purposing existing high-quality resources for database question answering (to produce PeopleJoin-QA ) and multi-document summarization (to produce PeopleJoin-DocCreation ). Figure 2: Illustration of a transformation of a Spider