MosaicLeaks : Privacy Risks in Querying-in-the-Open for Deep Research Agents
Alexander Gurung, Spandana Gella, Alexandre Drouin, Issam H. Laradji, Perouz Taslakian, Rafael Pardinas · 2026
Read MosaicLeaks : Privacy Risks in Querying-in-the-Open for Deep Research Agents by Alexander Gurung, Spandana Gella, Alexandre Drouin, Issam H. Laradji, Perouz Taslakian, Rafael Pardinas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30727.txt.
Introduction
Gurung and colleagues expose privacy risks when deep research agents combine private documents with external web retrieval. Single queries may look harmless but reveal information in aggregate, the mosaic effect. Their MosaicLeaks benchmark offers 1,001 multi-hop tasks chaining private documents and a public web corpus. An adversary model observing only external queries infers private information at three levels. Models across families and sizes frequently leak at all three levels. Their Privacy-Aware Deep Research framework combines situational rewards with a learned privacy classifier, improving accuracy and cutting leakage.
Catalog record
| CREATOR | Alexander Gurung, Spandana Gella, Alexandre Drouin, Issam H. Laradji, Perouz Taslakian, Rafael Pardinas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93177 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MosaicLeaks : Privacy Risks in Querying-in-the-Open for Deep Research Agents Alexander Gurung, Spandana Gella, Alexandre Drouin, Issam H. Laradji, Perouz Taslakian, Rafael Pardinas Abstract Deep research agents increasingly combine private local documents with external tools like web retrieval, creating a privacy risk: an agent’s external queries may leak sensitive information from its local context. This risk is amplified by the mosaic effect , where individual queries may appear harmless but become revealing in aggregate. We introduce MosaicLeaks , a benchmark of 1,001 multi-hop deep research tasks that chain private enterprise documents and a public web corpus, forcing agents to make external queries that depend on local information. We evaluate leakage with an adversary LLM that observes only the agent’s external queries and attempts to infer private information at three levels: the agent’s research intent, answers to specific private questions and verifiable claims about the enterprise documents. We find that models across families and sizes frequently leak at all three levels, that zero-shot privacy prompting reduces but does not eliminate leakage and that reinforcement learning for task performance alone worsens leakage. To address this, we propose Privacy-Aware Deep Research (PA-DR), an RL framework that combines situational rewards for task success with a learned privacy classifier to provide dense credit assignment over both per-query and mosaic-level leakage. Training Qwen3-4B-Instruct with PA-DR improves accuracy from 48.7 % 48.7\% to 58.7 % 58.7\% and reduces answer and full-information leakage from 34.0 % 34.0\% to 9.9 % 9.9\% . 1]ServiceNow AI Research 2]University of Edinburgh 3]Mila - Quebec AI Institute 4]McGill University 5]University of British Columbia \contribution [*]Equal contribution \contribution [†]Work done at ServiceNow AI Research \correspondence , 1 Introduction Figure 1 : Example of how the ‘Mosaic Effect’ contributes to MosaicLeaks ’s measurements of privacy leakage from a research agent’s web-queries. We evaluate leakage across three axes: Intent Leakage (predict the research questions), Answer Leakage (answer given questions about enterprise documents), and Full-Information Leakage (predict verifiably true claims about enterprise documents). In this example, the agent first searches twice for information related to Lee’s Market’s 2020 traffic growth, leaking its research intent. The third query switches to attempting to answer a new question, based on the answer to the previous one. Although these queries look benign alone, an adversary could infer compositional information when seen together. By deducing that 15% is the answer the agent was looking for in the first two queries, the adversary can make the claim that Lee’s online traffic grew 15% in 2020. Dataset / Task Multi-hop Local + Web Privacy Mosaic PrivacyLens ( 20 ) ✗ ✗ ✓ ✗ AgentDAM ( 24 ) ✗ ✗ ✓ ✗ SPILLage ( 18 ) ✗ ✗ ✓ ✗ TOP-Bench ( 17 ) ✗ ✗ ✓ ✓ DRBench ( 1 ) ✓ ✓ ✗ ✗ HERB ( 5 ) ✓ ✓ ✗ ✗ Chroma Context-1 ( 2 ) ✓ ✗ ✗ ✗ WebExplorer ( 13 ) ✓ ✗ ✗ ✗ ASearcher ( 7 ) ✓ ✗ ✗ ✗ MosaicLeaks (Ours) ✓ ✓ ✓ ✓ Table 1: Comparison of MosaicLeaks with previous work, grouped into privacy aware work, deep research tasks, and multi-hop datasets. Multi-hop : questions require sequentially chaining multiple retrieval or reasoning steps to answer. Local + Web : a single task in the dataset mixes local and external documents. Privacy : the dataset evaluates privacy leakage. Mosaic : the adversary model considers leakage across multiple actions/rounds. Language model agents are increasingly deployed with sensitive enterprise data while relying on external tools like web search and cloud APIs to collect information or accomplish complex tasks ( 1 ; 5 ; 16 ) . The value of these agents is largely in their ability to synthesize and iterate over information from both external and internal document sources. However, this value comes with a risk, as external services may be monitored and agents may leak private information through their tool calls. This risk is compounded via the mosaic effect , a long-recognized phenomenon where aggregating small pieces of information leaks more than each piece individually ( 15 ) . Recent work has already demonstrated mosaic leakage in LLM agents that orchestrate multiple tools across social-context scenarios ( 17 ) . In this work we focus on the mosaic effect in the context of deep research task, treating web-queries as the source of potential leakage. Although individual tool calls may appear innocuous, adversaries observing these queries over time can aggregate them and reconstruct sensitive attributes. Despite the risks, existing agent research frameworks largely optimize task performance without accounting for this potential leakage. To address this gap, we propose MosaicLeaks , a benchmark of 1001 multi-hop research questions that require interleaving local (enterprise) and external (web) searches to answer. We assume an adversary with access to the cumulative web queries an agent produces while answering each question, and task it with predicting private enterprise information from those queries alone. The goal for the agent is to answer questions accurately while minimizing adversary success. Inspired by InfoSeeker ( 10 ) and WebShaper ( 22 ) , we construct MosaicLeaks through a graph-based approach: each task is composed of multiple sub-questions. The answer to each sub-question serves as an entity in the next, bridging two documents. By alternating the document source at each hop (e.g., Local \rightarrow Web), we create chains that interleave enterprise and public information, drawing local documents from DRBench ( 1 ) and web content from BrowseComp-Plus ( 4 ) . Our initial experiments across six open-source LLMs reveal that deep research agents frequently leak enterprise information through web queries, and that naively training for task performance amplifies this behaviour. Furthermore, prior work has explored prompting-based methods to reduce privacy leakage in adjacent agent settings ( 20 ; 24 ; 18 ; 14 ; 17 ) , and we find that prompting fails to eliminate leakage for our task. However, we show that reinforcement learning (RL) can be used to train deep research agents to internalize privacy constraints, while also improving at the research task. As privacy leakage is expensive to measure, during training, we augment our task-performance reward with a penalty derived from an adversary model that observes only external calls and attempts to infer private information. Our approach, labeled Privacy Aware-Deep Research (PA-DR), provides dense credit assignment to only the calls that worsen privacy leakage while accounting for the mosaic effect. Our contributions are the following: (1) a new MosaicLeaks task that evaluates research agents on multi-hop questions with explicit local-external information dependencies, (2) empirical analysis of the resulting privacy–utility trade-off across closed and open-source models, and (3) a principled formulation of privacy aware agent training as joint privacy and performance objective. 2 Related Work Deep Research benchmarks. Deep research has emerged as an important agentic task, and several benchmarks evaluate it over the open web, including BrowseComp-Plus ( 4 ) , Deep Research Bench ( 3 ) , DeepResearch-9k ( 23 ) and DeepResearchGym ( 6 ) ; we adopt BrowseComp-Plus’s fixed corpus as the public-web side of our setup. More recently, several benchmarks include research questions that span local and external sources. HERB ( 5 ) similarly evaluates multi-hop deep search over heterogeneous enterprise sources including documents, meeting transcripts, Slack, and