R-TOFU: Unlearning in Large Reasoning Models

Sangyeon Yoon, Wonje Jeung, Albert No · 2025

Read R-TOFU: Unlearning in Large Reasoning Models by Sangyeon Yoon, Wonje Jeung, Albert No (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15214.txt.

Introduction

Reasoning models write out their thinking step by step, so private facts can hide in the middle of a chain, not just in the final answer. Sangyeon Yoon, Wonje Jeung, and Albert No build the first benchmark for unlearning in such models in 2025. It extends a standard forgetfulness test with realistic reasoning traces and grades each step. Answer-only checks, they show, miss much of what a model still knows.

Their tests find that common unlearning methods leave clear traces in the reasoning. A new variant they propose teaches the model to reason its way to an honest dead end instead of a flat refusal. Sneaky decoding settings can still drag secrets out, so they urge testing under many settings.

Catalog record

CREATORSangyeon Yoon, Wonje Jeung, Albert No
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15214

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R-TOFU: Unlearning in Large Reasoning Models Sangyeon Yoon, Wonje Jeung, Albert No Abstract Large Reasoning Models (LRMs) embed private or copyrighted information not only in their final answers but also throughout multi-step chain-of-thought (CoT) traces, making reliable unlearning far more demanding than in standard LLMs. We introduce Reasoning-TOFU (R-TOFU), the first benchmark tailored to this setting. R-TOFU augments existing unlearning tasks with realistic CoT annotations and provides step-wise metrics that expose residual knowledge invisible to answer-level checks. Using R-TOFU, we carry out a comprehensive comparison of gradient-based and preference-optimization baselines and show that conventional answer-only objectives leave substantial forget traces in reasoning. We further propose Reasoned IDK, a preference-optimization variant that preserves coherent yet inconclusive reasoning, achieving a stronger balance between forgetting efficacy and model utility than earlier refusal styles. Finally, we identify a failure mode: decoding variants such as ZeroThink and LessThink can still reveal forgotten content despite seemingly successful unlearning, emphasizing the need to evaluate models under diverse decoding settings. Together, the benchmark, analysis, and new baseline establish a systematic foundation for studying and improving unlearning in LRMs while preserving their reasoning capabilities. We release R-TOFU and code at https://ai-isl.github.io/r-tofu . Figure 1: Overview of LRMs Unlearning. (a) Concept. Unlike standard LLMs, LRMs require unlearning both the final answer and the associated reasoning trace. (b) Unlearning Strategies. The top row illustrates gradient ascent-based strategies, while the bottom row presents preference optimization-based strategies. Red indicates forget information that the model is trained to suppress, while blue indicates non-forget responses (e.g., “I don’t know.” (IDK)) that replace forget content during training. (c) Interaction with Decoding. Although unlearning may appear successful under DefaultThink, decoding strategies like ZeroThink and LessThink, which suppress CoT generation, can still reveal forgotten content, indicating incomplete unlearning. 1 Introduction With the rapid advancement of Large Language Models (LLMs) ( Achiam et al., 2023 ; Bai et al., 2023 ; Dubey et al., 2024 ) , the focus has shifted towards models capable of complex, multi-step reasoning. Large Reasoning Models (LRMs) such as OpenAI’s o1 ( Jaech et al., 2024 ) and DeepSeek-R1 ( Guo et al., 2025 ) have emerged as specialized architectures designed to handle these demanding tasks. Unlike standard LLMs, LRMs explicitly generate structured reasoning traces, producing coherent step-by-step explanations without requiring specialized prompts. This approach has enabled LRMs to excel in challenging domains like mathematics and programming, demonstrating strong real-world applicability. To achieve this, LLMs and LRMs are trained on massive text corpora. However, these corpora often contain copyrighted materials, personal data, and user-generated content ( Carlini et al., 2021 ; Wei et al., 2024 ) . As these models are deployed in real-world applications, there is growing pressure to remove specific training data due to legal and ethical concerns, including privacy regulations ( Voigt and Von dem Bussche, 2017 ) and ongoing lawsuits ( Tremblay v. OpenAI, Inc.,, 2023 ) . This has created an urgent need for machine unlearning ( Cao and Yang, 2015 ) , which aims to remove the influence of forget data (e.g., PII or copyrighted text) from the model, while preserving overall utility from the retain data . While machine unlearning has been extensively studied in LLMs ( Maini et al., 2024 ; Jeung et al., 2025a ) , there has been little exploration for LRMs, despite their rising prevalence and increasing privacy concerns. Unlike standard LLMs, LRMs can embed forget information not just in their final outputs, but throughout their reasoning processes, making unlearning more complex. This deeper integration of information means that effective unlearning for LRMs requires more than simply adjusting final outputs. Additionally, if forget information is embedded in the reasoning trace, it can subtly influence the model’s reasoning path, potentially leading it back to the original forget answer, even after unlearning. To ensure a comprehensive assessment, intermediate steps must also be considered. However, there are no benchmarks specifically designed to capture these intermediate reasoning dependencies, making it difficult to accurately assess unlearning effectiveness in LRMs. To fill this gap, we introduce R-TOFU, a benchmark for evaluating unlearning in LRMs, incorporating realistic chain-of-thought (CoT) traces to capture natural reasoning paths. To ensure these traces closely mirror typical LRMs behavior, we construct them through a four-step process that captures realistic, model-aligned reasoning. Using models fine-tuned on R-TOFU, we evaluate unlearning not only at the final answer level, but also across entire reasoning trajectories. Specifically, we propose step-wise evaluation metrics that assess unlearning at each intermediate step, enabling fine-grained detection of residual CoT knowledge that full-sequence evaluations often miss. Building on this framework, we extend gradient ascent-based methods ( Liu et al., 2022 ) , a standard unlearning approach for LLMs that maximizes loss on the forget set, to LRMs. We introduce three unlearning strategies: CoT+Answer, Answer-only, and CoT-only, each aligning with the unique reasoning structure of LRMs. Among these, CoT-only unlearning is particularly effective, as disrupting the reasoning path prevents the model from constructing the intermediate steps needed to reach the forget answer , enabling more reliable unlearning without significant degradation. In addition, we extend preference optimization (PO) ( Maini et al., 2024 ) , another widely used unlearning method in LLMs, to LRMs. For baseline comparison, we introduce two straightforward strategies: Answer IDK and Direct IDK. Answer IDK replaces only the final answer with a simple uncertain response like “I do not know,” preserving the original reasoning trace, while Direct IDK removes the entire CoT, replacing it with a brief refusal statement that avoids intermediate reasoning altogether. To better align with the complex reasoning structure of LRMs, we additionally propose Reasoned IDK. This approach generates coherent but ultimately unresolved reasoning, preserving the model’s structural fluency while concealing forget information. Our experiments show that Reasoned IDK outperforms all other PO strategies. Finally, we uncover a counterintuitive failure mode in LRMs unlearning, where decoding strategies like ZeroThink and LessThink ( Jiang et al., 2025 ) , which forcibly suppress the reasoning process, can inadvertently reveal residual forget information. While unlearning appears effective under conventional, reasoning-enabled settings, these decoding constraints expose cases where the model still reconstructs forget answers, bypassing the intended unlearning. This highlights the importance of evaluating unlearning robustness under diverse decoding strategies, rather than relying solely on standard reasoning prompts. In summary, our contributions include (1) the R-TOFU benchmark for structured reasoning trace unlearning, (2) extensive evaluation across diverse setups, including Answer-only gradient ascent and rejection-based interventions in the reasoning process, (3) novel unlearning strategy, Reasoned IDK that effectively balance knowledge retention and forget performance, and (4) the identification of a critical failure mode where decoding strategies expose residual forgotten content. Together, these advances provide a more realistic and comprehensive evaluation of unlearning for LRMs, providing a new area for privacy-focused AI research. To the best of our knowledge, our work is the first to investigate machine unlearning in the LRMs context. 2 Related Work Large Reasoning Models (LRMs). Pretrained LLMs initially faced challenges in refining their logical reasoning capabilities, but chain-of-thought (CoT) techniques ( Wei et al., 2022 ) addressed this by enabling models to perform step-by-step inference without additional training. This progress has been extended through approaches such as ReAct ( Yao et al., 2023b ) , tree-of-thought ( Yao et al., 2023a

Read the complete text