Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh · 2025
Read Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers by Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15674.txt.
Introduction
Tommaso Green and co-authors show that the private reasoning steps of large reasoning models can leak user data, in a 2025 paper. Such traces are often treated as internal and safe, unlike final answers. The team challenge that view for models used as personal agents.
Through probing and agent tests, they find the traces often hold sensitive user details. Those details can be pulled out by prompt attacks or slip into outputs by accident. More reasoning steps made final answers more careful, yet the traces grew longer and leaked more. The authors argue that safety work must cover the model's thinking itself, not just what it finally says.
Catalog record
| CREATOR | Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15674
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, Seong Joon Oh Abstract We study privacy leakage in the reasoning traces of large reasoning models used as personal agents. Unlike final outputs, reasoning traces are often assumed to be internal and safe. We challenge this assumption by showing that reasoning traces frequently contain sensitive user data, which can be extracted via prompt injections or accidentally leak into outputs. Through probing and agentic evaluations, we demonstrate that test-time compute approaches, particularly increased reasoning steps, amplify such leakage. While increasing the budget of those test-time compute approaches makes models more cautious in their final answers, it also leads them to reason more verbosely and leak more in their own thinking. This reveals a core tension: reasoning improves utility but enlarges the privacy attack surface. We argue that safety efforts must extend to the model’s internal thinking, not just its outputs. 1 1 1 Code available at github.com/parameterlab/leaky_thoughts . AirGapAgent-R benchmark available at huggingface.co/datasets/parameterlab/leaky_thoughts . 1 Introduction As language models are increasingly deployed as personal assistants, they gain access to sensitive user data, including identifiers, financial details, and health records. This paradigm, known as Personal LLM agents ( Li et al. 2024 ) , raises concerns about whether these agents can accurately determine when it is appropriate to share a specific piece of user information, a challenge often referred to as contextual privacy understanding. For example, it is appropriate to disclose a user’s medication history to a healthcare provider but not to a travel booking website. Personal agents are thus evaluated not only on their ability to carry out tasks ( utility ), but also on their capacity to omit sensitive information when inappropriate ( privacy ). Large reasoning models (LRMs), are being adopted more widely as agents thanks to their enhanced planning skills enabled by reasoning traces (RTs) ( Yao et al. 2023 ; Zhou et al. 2025b ) . These traces are sequences of thinking tokens produced by the LRM before returning its final answer, allowing the model to harness additional test-time compute (TTC) to get higher performance in reasoning-heavy tasks ( Snell et al. 2024 ) . Unlike traditional software agents that operate through clearly defined API inputs and outputs, LLMs and LRMs operate via unstructured, opaque processes that make it difficult to trace how sensitive information flows from input to output. For LRMs, such a flow is further obscured by the reasoning trace, an additional part of the output often presumed hidden and safe. Figure 1: Our goal . Prior studies on contextual privacy focused on LLM output. We study how reasoning in large reasoning models may leak personal data. Prior work has explored training-time memorisation and privacy leakage in LLMs ( Kim et al. 2023 ; Brown 2024 ; Puerto et al. 2025 ) , as well as contextual privacy in inference ( Mireshghallah et al. 2024 ; Bagdasarian et al. 2024 ) . Agentic benchmarks like AgentDAM focus on whether private information appears in tool actions or final outputs ( Zharmagambetov et al. 2025 ) . These works do not analyse the role of TTC in utility and privacy of LRM-powered personal agents, nor evaluate reasoning traces as an explicit threat vector (Figure 1 ). To the best of our knowledge, we are the first to compare LLMs and LRMs as personal agents: while LRMs predominantly surpass LLMs in utility, this is not always the case for privacy. To shed light on these privacy issues, we look into the reasoning traces and find that they contain a wealth of sensitive user data, repeated from the prompt. Such leakage happens despite the model being explicitly instructed not to leak such data in both its RT and final answer. Although RTs are not always made visible by model providers, our experiments reveal that (i) models are unsure of the boundary between reasoning and final answer, inadvertently leaking the highly sensitive RT into the answer, (ii) a simple prompt injection attack can easily extract the RT, and (iii) forcibly increasing the reasoning steps in the hope of improving the utility of the model amplifies leakage in the reasoning. Our study provides three main contributions: (1) Contextual privacy in LRMs. We are the first to compare LLMs to LRMs as personal agents. We perform our evaluations on two benchmarks: AirGapAgent-R, which is our open-sourced reconstruction of the unreleased AirGapAgent benchmark Bagdasarian et al. 2024 , and AgentDAM Zharmagambetov et al. 2025 . We find that TTC greatly benefits the utility of the model but not always its privacy (§ 4 ). (2) Leaky thoughts: reasoning traces are a privacy risk. We unveil that the RT constitutes a new privacy attack surface for LRMs, as it is abundant in sensitive data and can easily be exposed, either accidentally by the model or adversarially by malicious actors. LRMs do not follow the anonymising directives of their prompt (§ 5 ), treating their RT as their private scratchpad. (3) Why and how . We study the why and how of the privacy leaks (§ 6 ). We find that leakage in the reasoning is mostly driven by a simple recollection mechanism: if a LRM is asked to provide the user’s age, it simply cannot help but materialize its actual value within its RT, exposing it to risk of extraction. Moreover, when this mechanism is suppressed by forcibly anonymizing the reasoning post-hoc, the utility of the agent declines. These findings suggest that treating RTs as “internal” or “safe” is dangerously optimistic. In many settings, the RT is visible or at least extractable. Thus, reasoning leakage is not only a technical nuisance but a critical safety failure. As models adopt richer TTC paradigms for planning, tool use, or self-reflection, new privacy strategies must be developed to address leaks during thinking, not just in output. 2 Background and Related Work Contextual privacy in LLMs. Contextual integrity defines privacy as the proper flow of information within a social context ( Nissenbaum 2004 ; Shvartzshnaider and Duddu 2025 ) , a key concern for personal agents handling sensitive data. While most research has focused on training-time leakage ( Kim et al. 2023 ; Brown 2024 ; Puerto et al. 2025 ) , inference-time privacy remains underexplored ( Evertz et al. 2024 ; Yan et al. 2025 ) . Benchmarks like DecodingTrust ( Wang et al. 2023 ) , AirGapAgent ( Bagdasarian et al. 2024 ) , CONFAIDE ( Mireshghallah et al. 2024 ) , PrivaCI ( Li et al. 2025b ) , and CI-Bench ( Cheng et al. 2024 ) evaluate contextual adherence through structured prompts. PrivacyLens ( Shao et al. 2024a ) and AgentDAM ( Zharmagambetov et al. 2025 ) simulate agentic tasks, though all target non-reasoning models. Recent methods attempt to mitigate inference-time leakage: TextObfuscator masks sensitive spans during generation ( Zhou et al. 2023 ) , Papillon redacts and later restores PII (personally identifiable information) during API calls ( Siyan et al. 2024 ) , and prompt obfuscation techniques hide intent or content through rewriting ( Pape et al. 2024 ) . While effective at surface-level protection, these approaches fail to account for how reasoning steps themselves can