MAGPIE : A dataset for Multi-AGent contextual PrIvacy Evaluation

Gurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang · 2025

Read MAGPIE : A dataset for Multi-AGent contextual PrIvacy Evaluation by Gurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20737.txt.

Introduction

Gurusha Juneja and co-authors built this 2025 test of privacy sense in AI agents that work as teams. Such agents book, bargain, and share data with each other. They often hold private facts that a task needs in part, yet sharing too much can cause real loss. Older tests were simple and single-turn, so this team raised the bar.

Their set, MAGPIE, holds 158 high-stakes scenes across 15 fields of life. The team checked top models on two skills: spotting what counts as private, and teaming up without leaking it. Leading models mislabeled private data as shareable in a quarter to nearly half of cases. In longer chats, they disclosed private facts in about half or more of the runs, a warning for agent design.

Catalog record

CREATORGurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAGPIE : A dataset for Multi-AGent contextual PrIvacy Evaluation Gurusha Juneja, Alon Albalak, Wenyue Hua, William Yang Wang Abstract The proliferation of LLM-based agents has led to increasing deployment of inter-agent collaboration for tasks like scheduling, negotiation, resource allocation etc. In such systems, privacy is critical, as agents often access proprietary tools and domain-specific databases requiring strict confidentiality. This paper examines whether LLM-based agents demonstrate an understanding of contextual privacy. And, if instructed, do these systems preserve inference time user privacy in non-adversarial multi-turn conversation. Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks where private information can be easily excluded. We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains. These scenarios are designed such that complete exclusion of private data impedes task completion yet unrestricted information sharing could lead to substantial losses. We then evaluate the current state-of-the-art LLMs on (a) their understanding of contextually private data and (b) their ability to collaborate without violating user privacy. Empirical experiments demonstrate that current models, including GPT-4o and Claude-2.7-Sonnet, lack robust understanding of contextual privacy, misclassifying private data as shareable 25.2% and 43.6% of the time. In multi-turn conversations, these models disclose private information in 59.9% and 50.5% of cases even under explicit privacy instructions. Furthermore, multi-agent systems fail to complete tasks in 71% of scenarios. These results underscore that current models are not aligned towards both contextual privacy preservation and collaborative task-solving . 1 Introduction As large language model (LLM)-based agents become more widely adopted, they emerge as promising tools for managing complex tasks on behalf of users ( Wu et al. 2006 ; Li et al. 2024 ; Pinhanez et al. 2018 ; Muthusamy et al. 2023 ) . These agents often operate with access to sensitive user data, including personal preferences, financial details, and medical records, which users grant to enable autonomous operation ( Talebirad & Nadiri 2023 ) . However, with different users employing their own agents, these systems will increasingly need to interact with one another in collaborative contexts such as scheduling, negotiation, and resource allocation. A fundamental tension emerges in these multi-agent interactions: agents must share sufficient information to achieve shared goals while rigorously safeguarding user’s private data ( Pinhanez et al. 2018 ) . For instance, Figure 2 illustrates a dialogue between a Team Lead’s agent and a GPU allocator’s agent within a company. The Team Lead’s agent has access to sensitive information including emails, meeting transcripts, and internal team notes. Details regarding the project, minimum GPU requirements, and performance implications related to GPU allocation may be highly confidential. If such information were leaked during negotiations, it could compromise competitive advantages or violate privacy regulations. Conversely, if such confidential information is fully disclosed during negotiations, it could compromise competitive advantages or violate privacy regulations. This presents a complex challenge where complete privacy preservation may impede task completion, while unrestricted information sharing risks unauthorized disclosure of sensitive data . Such challenges are significantly amplified in high-stake scenarios, where the consequences of either information leakage or negotiation failure can have substantial economic, strategic, or legal impact. Figure 1 : Negotiation between Team Lead’s Agent and GPU Allocator’s Agent. The Team Lead Agent discloses critical private information, directly exposing workforce constraints that risk employee complaints. Figure 2 : Representative scenarios from the dataset include critical domains such as resource allocation, where privacy breaches incur significant real-world consequences. However, our initial investigations reveal that existing LLMs frequently struggle to identify contextually sensitive information and fail to maintain appropriate boundaries during non-adversarial collaborative interactions. As illustrated in Figure 1, the Team Lead’s agent inadvertently discloses that their team could complete the project with fewer GPUs by working overtime in response to a standard inquiry from the allocator agent—despite no adversarial probing techniques being employed. These observed vulnerabilities highlight several critical research questions that must be addressed: (1) To what extent do AI agents possess an intrinsic understanding of information privacy? (2) Can these systems consistently adhere to privacy constraints during dynamic, multi-turn interactions? (3) Most importantly, does enforcing strict privacy preservation inevitably compromise the agents’ capacity to achieve their assigned task objectives? Prior efforts to evaluate and understand privacy-related ability in agents, such as PrivacyLens ( Shao et al. 2025 ) and ConfAIde ( Mireshghallah et al. 2024 ) are limited to single-turn interactions and simplistic tasks where privacy-sensitive information can be trivially omitted without affecting outcomes. Instead, we introduce MAGPIE ( M ulti- AG ent contextual P r I vacy E valuation), a novel benchmark designed specifically for evaluating privacy preservation in multi-agent collaborative contexts. Our benchmark is constructed around real-world scenarios where private information is central to task resolution and cannot be entirely withheld without compromising the objective. The dataset comprises 158 high-stakes tasks spanning multiple domains including resource allocation, admissions processes, and economic negotiations etc. Each task is rigorously constructed through a multi-stage LLM-driven pipeline, validated by both automated verifiers and human annotators to ensure realism, motivational coherence, and alignment with privacy requirements. Fig 2 shows a few examples of tasks from our benchmark. Our evaluation of SOTA LLMs, including GPT-4o ( OpenAI et al. 2024 ) , Claude-3.7-Sonnet ( Anthropic 2024 ) , Llama-70B ( Grattafiori et al. 2024 ) and Mistral-123B ( Jiang et al. 2023 ) , reveals significant shortcomings in current systems: (1) models misclassify private information as public in 35.2\% of cases on average, substantially exceeding the human baseline error rate of 10.2\% (2) when explicitly requested to share sensitive data agents leak confidential information in up to 59.1% of scenarios. (3) in multi-round multi-agent conversation, privacy violations occur in 55.27\% of cases, with leakage rates increase with the number of conversation rounds. Moreover, these systems are not able to reach a consensus in 49% of the tasks and even when they reach a consensus, they fail to successfully complete the task in 70.3% of the cases. These results highlight a pressing issue in privacy understanding, instruction following, and task solving abilities in LLM-based multi-agent systems. We hope that our real-life benchmark provides a critical step toward understanding privacy issues and establishes a foundation for developing robust, privacy-aware multi-agent architecture. 2 Related work Language Model Privacy Prior research on language model privacy has primarily focused on two concerns: (1) memorization of training data and (2) adversarial extraction of sensitive information ( Carlini et al. 2021 ; Kim et al. 2023 ; Zhuo et al. 2023 ; Duan et al. 2024 ) . While these studies reveal foundational risks, they overlook critical inference-time vulnerabilities. As LMs are increasingly deployed in real-world applications, private information can be unintentionally exposed through generated text, violating context-specific social norms ( Gabriel et al. 2024 ) . Recent work has begun exploring contextual privacy risks via attribute inference and prompt injection attacks ( Röttger et al. 2024 ; Wang et al. 2024 ) , yet these efforts lack systematic evaluation frameworks for dynamic, multi-turn interactions. Benchmarks like ConfAIde ( Mireshghallah et al. 2024 ) and PrivacyLens ( Shao et al. 2025 ) assess whether LMs can reason about contextual privacy. However, their tasks are limited to single-turn interactions and simplistic scenarios where private information can be ignored

Read the complete text