Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries

Yanhang Li, Zhichao Fan, Zexin Zhuang · 2026

Read Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries by Yanhang Li, Zhichao Fan, Zexin Zhuang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18891.txt.

Introduction

Yanhang Li, Zhichao Fan and Zexin Zhuang audit claims that unlearned models remember forgotten content in their traces. When the answer looks clean but the trace leaks erased material, the gap is read as proof of hidden weight memory. Their tests use a distilled model with memorized fictional authors. Swapping the trace for a neutral prefill on identical weights can make the gap vanish, shrink or reverse. Such a gap thus neither proves nor rules out weight memory, and the authors recommend a template-swap check beside the standard audit.

Catalog record

CREATORYanhang Li, Zhichao Fan, Zexin Zhuang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41076
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries Yanhang Li, Zhichao Fan, Zexin Zhuang Abstract Evaluations of unlearning on reasoning models sometimes show a bypass pattern. The answer side looks unlearned, but the model’s own thinking trace keeps emitting the forgotten content, and the gap is taken as evidence that the weights still remember. We audit this reading on DeepSeek-R1-Distill-Qwen-7B with LoRA-memorized fictional authors and NPO unlearning, conditioned on a six-token canary head. On one seed, swapping the thinking trace for a short non-canary prefill on the same weights drops the answer rate by as much as the bypass gap itself, whether the prefill mimics the training template or not. On a second seed the bypass gap shrinks rather than vanishing, and the prefill swap reverses direction and brings the answer rate to ceiling. A positive parser-split bypass gap thus does not by itself identify hidden weight-level memorization, and does not rule it out either. On a different distillate the same metric flips sign because the parser cannot find the closing tag. We recommend a decode-time template swap as a cheap sanity check alongside the canonical audit. 1. Train (status quo) 60 fictional authors \xrightarrow{\text{LoRA SFT}} \langle\text{think}\rangle\,\underbrace{\tau}_{\text{bio}}\,\langle/\text{think}\rangle\,a answer-masked NPO (gradient on a only; template in \tau untouched) 2. Parser-split audit split on \langle/\text{think}\rangle \mathrm{out\_acc}=\mathbb{I}[a^{\star}\!\subseteq a] \mathrm{thk\_leak}=\mathbb{I}[a^{\star}\!\subseteq\tau] \Delta=\overline{\mathrm{thk\_leak}}-\overline{\mathrm{out\_acc}} 3. Fixed-weight probes (ours) (a) decode-time prefill: swap \tau for bio / meta / empty , then continue (b) teacher-forced: supply 6-token canary head, score continuation 4. Finding \Delta is non-identifying : prefill swap moves \mathrm{out\_acc} by \approx\!\Delta at fixed weights; Llama-8B parser sign-flip \pm 0.92 a positive \Delta is read as a hidden memorization channel Figure 1 : The audited pipeline and our two fixed-weight probes. Boxes 1–2 are the status-quo protocol; box 3 adds a decode-time prefill swap and a teacher-forced continuation probe at fixed weights. The gap \Delta tracks decode-time context rather than retention, and flips sign under format drift on a second distillate (box 4). 1 Introduction Auditing memorization in foundation models ( 3 ; 4 ; 2 ) gets harder when the model writes an explicit reasoning trace alongside its answer. DeepSeek-R1 ( 6 ) and its open distilled variants (e.g. DeepSeek-R1-Distill-Qwen-7B, the model we audit) emit a response of the form \langle\text{think}\rangle\,\tau\,\langle/\text{think}\rangle\,a , where the thinking trace \tau is the model’s scratchpad and a is the final answer span. Unlearning methods from the pre-reasoning era are now applied to these distilled reasoning models by masking the forgetting loss to a and splitting each generation on the closing \langle/\text{think}\rangle tag for the audit (the parser-split metric: substring-match the forgotten content separately in \tau and in a ). These pre-reasoning unlearning methods include Gradient Ascent ( 9 ) , TOFU-style fictitious-author forgetting ( 13 ) , WMDP representation-misdirection ( 11 ) , and Negative Preference Optimization ( 22 ) . A natural reading of this combined protocol is a bypass pattern. The answer span a drops sharply after unlearning, but the \langle think \rangle segment keeps reproducing the forgotten content, and the gap is interpreted as a hidden memorization channel where the weights still hold what an answer-only audit would miss. Concurrent 2025 work documents that answer-level unlearning leaves residual content in the reasoning trace ( 21 ; 16 ; 18 ; 19 ) ; what is not established is whether the parser-split gap itself identifies weight-level memorization, as the hidden-channel reading assumes. We are not aware of prior work that validates that inference, and we do not claim prior work uniformly endorses it (§ 2 ). We name the reading and audit it as a candidate measurement claim that the parser-split pipeline above readily produces. If the reading is right, output-only memorization audits underestimate privacy and IP risk on reasoning models, which is why this question matters for the MemFM agenda. Our measurement question is whether the thinking-channel bypass is a hidden memorization channel in the weights, or an artifact of the measurement device itself. Figure 1 summarizes the audited pipeline and the two fixed-weight probes we add. The construct problem. Three ingredients recur across reasoning-model unlearning evaluations. (i) Trace-supervised bio restatement : the thinking trace \tau is trained to copy the target bio, in the spirit of the reasoning-format distillation popularized by DeepSeek-R1 ( 6 ) . The bio-restatement specialization is our setup. (ii) An answer-masked unlearning loss on a only, in the spirit of the forgetting-loss formulation of 22 and the fictitious-author forget-set framing of 13 . No gradient ever touches the bio template inside \tau . (iii) A parser-split evaluation that splits on \langle/\text{think}\rangle and substring-matches each side, extending exact-containment leakage probes from memorization/unlearning audits ( 3 ; 9 ) . This pipeline structurally predisposes a positive bypass gap. The bio-template scaffold inside \tau never receives a forgetting gradient, so any continued surface-level emission of canary content from \tau is consistent with template echo rather than weight-level retention, and the parser cannot distinguish the two. We treat this combination as a plausible canonicalized parser-split audit : each ingredient (i)–(iii) is in the literature. Our contribution is a construct-validity stress test of an interpretation the metric invites, not a rebuttal of a named prior claim—we test whether the hidden-channel reading follows from the parser-split metric. The concern is a construct-validity one: the surface form of \tau is not a transparent readout of what the weights encode (§ 2 ). Contributions. Working with 60 fictional authors memorized into DeepSeek-R1-Distill-Qwen-7B via LoRA and then unlearned with NPO, we: • Instantiate the head-conditioned bypass pattern the plausible parser-split audit produces. The probe primes the first six tokens of the canary, so \mathrm{out\_acc} is a head-conditioned continuation score, not a free-recall readout. After NPO, \mathrm{out\_acc} drops from 1.00 to 0.60 , \mathrm{thk\_leak} stays at 0.83 , and \Delta{=}+0.23 (§ 4.1 ). In 86\% of those bypass cases the answer span is verbatim the question prefix with no continuation while \tau holds the trained bio (App. D ). • Run an inference-time prefill intervention on the same weights. A short non-canary prefill drops \mathrm{out\_acc} from 0.60 to 0.37 . bio and meta prefills land at the same rate, and empty drops to 0.20 . The contrast is a decode-time non-invariance estimate, not an isolation of any mechanism. The same intervention also perturbs ordinary QA probes. We rule out one thing only, that \Delta is self-identifying hidden-channel evidence (§ 4.2 ). • Static parser-split evaluation is unreliable for reasoning models, and a paired decode-time probe plus teacher-forced scoring is a usable replacement. Three mutually reinforcing observations support this. (a) A decoding-free teacher-forced continuation probe, conditioned on the same six-token canary head, keeps post-NPO top- 1 match at 0.90 – 1.00 across prefills (§ 4.3 ): the greedy substring drop does not transfer to a matched token-level scoring setup. (b) On DeepSeek-R1-Distill-Llama-8B the same metric flips sign to -0.92 at K{=}1600 —not because unlearning succeeds ( \mathrm{out\_acc} stays at 1.00 , teacher-forced top- 1\geq 0.998 ) but because format drift moves the bio template outside the \langle think \rangle tags on \geq 60\% of probes and the parser populates an empty \tau (§ 4.4 ). (c) Together with the prefill result, this shows the parser-split gap reports decode-time and parsing artifacts, not

Read the complete text