MemLens: Uncovering Memorization in LLMs with Activation Trajectories
Zirui He, Haiyan Zhao, Ali Payani, Mengnan Du · 2025
Read MemLens: Uncovering Memorization in LLMs with Activation Trajectories by Zirui He, Haiyan Zhao, Ali Payani, Mengnan Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20909.txt.
Introduction
A model that memorized a test will ace it without truly reasoning, and simple rewording can hide the cheating. Zirui He and four co-authors present LogitTrace, a 2025 method for detecting benchmark contamination in large language models. It watches how an answer forms inside the model, layer by layer.
Older detectors rely on surface overlap or final-answer likelihood, and they degrade when inputs are rephrased. LogitTrace instead tracks the trajectory of the model's preferences across its layers. Contaminated examples tend to show early commitment to the answer, while clean examples settle later and less steadily. By reading these internal dynamics, the method aims to flag memorization on benchmarks such as AIME and Math500 even after rewording.
Catalog record
| CREATOR | Zirui He, Haiyan Zhao, Ali Payani, Mengnan Du |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MemLens: Uncovering Memorization in LLMs with Activation Trajectories Zirui He, Haiyan Zhao, Ali Payani, Mengnan Du Abstract Large language models (LLMs) are commonly evaluated on challenging benchmarks such as AIME and Math500, which are susceptible to contamination and risk at being memorized. Existing detection methods, which primarily rely on surface-level lexical overlap and perplexity, demonstrate low generalization and degrade significantly when encountering implicitly contaminated data. In this paper, we propose MemLens (An Activation Lens for Mem orization Detection) to detect memorization by analyzing the probability trajectories of numeric tokens during generation. Our method reveals that contaminated samples exhibit “shortcut” behaviors, locking onto an answer with high confidence in the model’s early layers, whereas clean samples show more gradual evidence accumulation across the model’s full depth. We observe that contaminated and clean samples exhibit distinct and well-separated reasoning trajectories. To further validate this, we inject carefully designed samples into the model through LoRA fine-tuning and observe the same trajectory patterns as in naturally contaminated data. These results provide strong evidence that MemLens captures genuine signals of memorization rather than spurious correlations. 1 Introduction Large language models (LLMs) have achieved remarkable performance across reasoning, dialogue, and code generation tasks ( Brown et al. 2020 ; OpenAI 2023 ; Touvron et al. 2023 ) . However, their success has raised growing concerns about data contamination , where benchmark test data (or their rephrased variants) appear in the training corpus. Such contamination can lead models to reproduce memorized solutions while appearing to reason, compromising claims of true generalization and raising privacy and intellectual property risks ( Carlini et al. 2021 ; Zhao et al. 2024 ; Nasr et al. 2025 ) . Contamination has been documented in widely used datasets such as MMLU and TruthfulQA ( Deng et al. 2024 ) . It can yield deceptively strong outputs: models may still solve compressed or even invalid problems, reflecting recall rather than reasoning (See Figure 1 ). Alternatively, Wu et al. 2025 demonstrate that reinforcement learning gains on math benchmarks can vanish once contaminated items are removed, underscoring the urgent need for systematic contamination detection. Existing detection methods primarily rely on completion-based recall tests ( Carlini et al. 2023 ; Nasr et al. 2025 ) and distributional probes such as perplexity and output agreement ( Li et al. 2023a ; Dong et al. 2024 ; Shi et al. 2024 ; Zhang et al. 2024b ; Zhang et al. 2024a ) . Other approaches include adversarial compression ( Schwarzschild et al. 2024 ) and partial probing ( Zhao et al. 2024 ; Ye et al. 2024 ) . While these methods provide valuable evidences of contamination, their reliance on surface lexical overlap limits robustness. When test problems are rephrased, paraphrased, or structurally altered, detection accuracy degrades substantially ( Deng et al. 2024 ; Yao et al. 2024 ) . For example, Yang et al. 2023 report that adding paraphrased MMLU items into training raises Llama 2 accuracy from 45–55% to nearly 89%, almost matching GPT-4. Such findings reveal that contamination persists even without verbatim overlap and motivate robust representation-level approaches. We introduce MemLens , a detection algorithm leveraging activation trajectories to identify memorization footprints caused by data contamination. Our approach operates on the hypothesis that memorized samples induce “shortcut” reasoning paths within the model . Specifically, MemLens analyzes the evolution of probability distributions over numeric tokens across layers, tracking metrics like digit-specific probabilities, entropy, and maximum confidence. We observe that contaminated samples tend to lock onto an answer with high confidence much earlier in the forward pass , whereas clean, unmemorized samples exhibit more gradual evidence accumulation that requires deeper processing. Unlike completion- or answer-based methods, MemLens remains robust under rephrasing, translation, and perturbation. Beyond detection performance, our analysis of activation dynamics reveals distinctive representation patterns of memorization, offering new insights into how contamination manifests inside models. To further establish causal validity, we conduct controlled LoRA injection experiments and show that injected samples induce the same trajectory signatures as naturally contaminated data. Our contributions are threefold: (i) a contamination detection algorithm that achieves robustness to rephrased inputs, (ii) representation-space analysis that characterizes memorization behaviors, and (iii) causal validation via LoRA injection, confirming that MemLens captures memorization rather than spurious signals. Figure 1: These four examples illustrate the potential risk that memorization may masquerade as generalization. In (a), the model answers the original problem correctly, which is expected. In (b), after rephrasing, the model still provides the correct solution, appearing to generalize. In (c), however, even when the input is reduced to a minimal set of keywords, the model produces the correct answer together with a full chain-of-thought explanation. This behavior is unlikely to arise from genuine reasoning and instead strongly suggests direct recall of memorized content. In (d), the model attempts to generate an answer even for an invalid input, further underscoring its reliance on pattern matching rather than robust reasoning. Together, these cases reveal that model outputs may reflect hidden memorization signals disguised as generalization, a phenomenon that poses potential risks in downstream use; full chain-of-thought generations are provided in the Appendix B . 2 Preliminaries 2.1 Memorization Definitions Let \Sigma be the vocabulary and \Sigma^{\star} the set of all finite sequences. A causal language model M_{\theta} defines a conditional distribution p_{\theta}(y\mid x) over continuations y\in\Sigma^{\star} given a prompt x\in\Sigma^{\star} . We define the sequence-level log-likelihood as s_{\theta}(y\mid x)\triangleq\sum_{t=1}^{|y|}\log p_{\theta}\!\big(y_{t}\,\big|\,x,y_{<t}\big). (1) A decoding functional f_{\theta}^{\pi}(x) maps x to a sequence according to a fixed policy \pi (e.g., greedy decoding). Throughout, y^{\star}\in\Sigma^{\star} denotes a specific training sample from the pretraining corpus. Extractable memorization. We say that y^{\star} is extractably memorized by M_{\theta} if there exists a prompt x\in\Sigma^{\star} and a decoding policy \pi such that the decoded output y equals y^{\star} and has a nontrivial log-probability margin over any alternative continuation. Formally, there exist x and margin \kappa>0 such that f_{\theta}^{\pi}(x)\;=\;y^{\star};\quad s_{\theta}(y^{\star}\mid x)\;-\;\max_{y\neq y^{\star}}s_{\theta}(y\mid x)\;\geq\;\kappa. (2) This condition captures two aspects of extractability: (i) verbatim reproduction under a standard decoding rule, and (ii) a separating margin that rules out accidental ties due to sampling noise. In practice, one may upper-bound the second term by restricting y to the k -best hypotheses or to a task-specific candidate set. Discoverable memorization. We say that y^{\star}=(y_{\mathrm{pre}},\,y_{\mathrm{suf}}) is discoverably memorized if there exists a split with a sufficiently informative prefix y_{\mathrm{pre}} such that the model (i) reproduces the exact suffix under length-constrained decoding, and (ii) assigns the suffix a per-token log-likelihood above a threshold. That is, there exist y_{\mathrm{pre}} , y_{\mathrm{suf}} with |y_{\mathrm{pre}}|\geq m for some minimal prefix length m , and thresholds \tau>0 , \kappa\geq 0 such that \begin{array}[]{cc}\text{(i)}&f_{\theta}^{\pi,\,|y_{\mathrm{suf}}|}\!\big(y_{\mathrm{pre}}\big)\;=\;y_{\mathrm{suf}}\\ \text{(ii)}&\frac{1}{|y_{\mathrm{suf}}|}\,s_{\theta}\!\big(y_{\mathrm{suf}}\,\big|\,y_{\mathrm{pre}}\big)\;\geq\;\tau\quad\text{and}\quad s_{\theta}\!\big(y_{\mathrm{suf}}\,\big|\,y_{\mathrm{pre}}\big)\;-\;\max_{y\neq y_{\mathrm{suf}}}s_{\theta}\!\big(y\,\big|\,y_{\mathrm{pre}}\big)\;\geq\;\kappa.\end{array} (3) Clause (i) requires exact continuation under length-constrained decoding, which operationalizes “completion” tests, while Clause (ii) further ensures statistical preference for the suffix. The prefix length constraint |y_{\mathrm{pre}}|\!\geq\!m reflects the empirical observation that longer prefixes may be necessary to trigger recall. Unlike extractable memorization, discoverable memorization often requires such longer prefixes and may only surface under targeted probing ( Carlini et al. 2023 ; Yang et al. 2023 ) . These categories formalize memorization behaviors as model-level phenomena. In practice, contamination at the data level (i.e., benchmark items appearing in pretraining corpora) may manifest as either extractable