CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training

Lukas Thede, Stefan Winzeck, Zeynep Akata, Jonathan Richard Schwarz · 2026

Read CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training by Lukas Thede, Stefan Winzeck, Zeynep Akata, Jonathan Richard Schwarz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06610.txt.

Introduction

Thede and team define forgetting in large language model post-training as model drift degrading ways and user experience, beyond the view of knowledge loss. They bring CapTrack, a tool joining a taxonomy with an evaluation suite centred on capability-specific metrics. They conduct a hands-on study across domains, model families, and post-training algorithms, including large models. Lesson fine-tuning induces the strongest drift, while preference tuning, more careful, can recover lost capabilities, family gaps persist, and no fix emerges.

Catalog record

CREATORLukas Thede, Stefan Winzeck, Zeynep Akata, Jonathan Richard Schwarz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117328
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06610

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training Lukas Thede, Stefan Winzeck, Zeynep Akata, Jonathan Richard Schwarz Abstract Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known to induce forgetting, especially in the ubiquitous use-case of leveraging third-party pre-trained models, which is typically understood as a loss of parametric or factual knowledge. We argue that this accuracy-centric view is insufficient for modern foundation models and instead define forgetting as systematic model drift that degrades behavior and user experience. In this context, we introduce CapTrack , a capability-centric framework for analyzing forgetting in LLMs that combines a behavioral taxonomy with an evaluation suite built on established benchmarks and targeted adaptations. Using CapTrack, we conduct a large-scale empirical study across post-training algorithms, domains, and model families, including models up to 80B parameters. We find that forgetting extends beyond parametric knowledge, with pronounced drift in robustness and default behaviors. Instruction fine-tuning induces the strongest relative drift, while preference optimization is more conservative and can partially recover lost capabilities. Differences across model families persist, and no universal mitigation emerges. 1]Thomson Reuters Foundational Research 2]Tübingen AI Center, University of Tübingen 3]Helmholtz Munich 4]Munich Center for Machine Learning (MCML), Technical University Munich 5]Imperial College London \correspondence {first.last}@thomsonreuters.com 1 Introduction Adapting large language models (LLMs) through post-training is central to enhancing latent skills, improving performance, unlocking value alignment, and enabling adaptation to new application settings. While effective, post-training is known to induce forgetting, typically understood as the loss of previously acquired parametric knowledge. Prior work has therefore studied forgetting mainly through declines in factual accuracy or task performance. Although this view has been informative, it captures only a narrow notion of model quality. In early continual learning settings, especially in vision, such a focus was appropriate as models were optimized for a single performance metric. In contrast, modern LLMs are interactive systems whose usefulness depends on a broader set of capabilities, including reasoning, robustness, default behavioral preferences, and adherence to interaction protocols. This gap motivates a broader view of forgetting in LLMs. For instance, a Qwen 3 80B Next model post-trained on legal data may appear stable on standard benchmarks, with negligible changes in factual accuracy (e.g., +0.4\% on MMLU-Pro) and instruction-following scores ( +0.7\% on IFEval) , yet exhibits substantial degradation in other capabilities such as multilingual robustness ( -16.4\% ) and verbosity ( -4.3\% ). Thus, evaluating forgetting solely through parametric knowledge risks missing model drift that directly affects model behavior and user experience. We therefore extend the notion of forgetting to encompass any systematic drift that negatively impacts capabilities, behaviors, or execution reliability. To enable measurement of the aspects captured by the wider definition, we introduce CapTrack , a capability-centric framework for assessing forgetting in LLM post-training. CapTrack combines a structured taxonomy that decomposes LLM behavior into latent competence, default behavioral preferences, and protocol compliance with an evaluation suite that probes these dimensions using established benchmarks and targeted adaptations. Together, CapTrack enables a multifaceted analysis of how post-training alters different aspects of model behavior beyond parametric knowledge. Figure 1 : Average forgetting across post-training stages (relative to OOB; higher is better). Forgetting extends beyond factual knowledge, with strongest degradation under IFT, milder effects under DPO, and partial recovery when DPO is applied after IFT. We employ CapTrack in a large empirical study across the legal and medical domains, covering 7 models from the LLaMA, Qwen, and Gemma families with up to 80B parameters. We find that forgetting consistently extends beyond parametric knowledge, most notably in robustness, default behavior preferences, and multi-turn consistency. Forgetting patterns differ systematically across model families but show no consistent dependence on model size. Instruction fine-tuning (IFT) induces the strongest degradation, whereas preference optimization is more conservative and can partially recover lost capabilities. Finally, we use CapTrack to assess data-, architectural-, and regularization-based mitigation strategies at the capability level. We show that forgetting is not tied to domain-specific adaptation: data interventions yield mixed, capability-dependent effects, while global regularization reduces forgetting only by sacrificing post-training gains, revealing a fundamental stability-plasticity trade-off. Overall, our results show that forgetting in LLM post-training is fundamentally a capability-level phenomenon that cannot be assessed by retention of general factual knowledge alone. Understanding and mitigating it, therefore, requires moving beyond traditional general knowledge benchmarks toward a holistic, capability-centric evaluation of model behavior. This paper makes the following contributions: • We extend the notion of forgetting in LLM post-training beyond parametric knowledge loss to encompass capability-level drift in behavior and execution. • We introduce CapTrack , a capability-centric taxonomy and evaluation suite that enables multifaceted assessment of forgetting across latent competence, default behaviors, and protocol compliance. • Using CapTrack, we disentangle how post-training algorithms, data sources, model families, and model scales shape capability-level forgetting, revealing qualitatively different forms of capability drift. • We use CapTrack to evaluate data-, architectural-, and regularization-based mitigation strategies, revealing capability-specific effects and fundamental stability-plasticity trade-offs. 2 Related Work 2.1 Post-Training of Large Language Models Post-training has become the primary driver of capability gains in modern LLMs, often outweighing additional pretraining scale in its impact on downstream performance ( 47 ) , with contemporary pipelines operating at data scales comparable to early pretraining ( 56 ) . Post-training commonly covers supervised fine-tuning on instruction–response pairs (i.e. IFT) ( 66 ) , preference optimization via ranked or pairwise feedback (e.g., DPO ( 51 ) , APO ( 50 ) , IPO ( 14 ) , KTO ( 11 ) ), and reinforcement learning–based alignment using learned rewards, most prominently PPO-based RLHF and GRPO ( 48 ; 52 ) . In this work, we focus on IFT and DPO post-training. 2.2 Forgetting in Neural Networks and LLMs Capability L&L’24 Yil’24 Jia’24 Luo’23 Haq’25 Hua’24 Zhe’25 LiuN’25 Har’25 Fer’24 Kot’23 Li’25 Lin’24 CapTrack Setting CPT CPT IFT IFT IFT IFT IFT IFT IFT, RL IFT, DPO IFT, RL IFT, RL RL IFT, DPO Parametric Knowledge ✓ ✓ ✓ ✓ - ✓ ✓ - ✓ ✓ - - ✓ ✓ Reasoning - - - ✓ - - - - ✓ - - ✓ - ✓ In-Context Learning (ICL) - - - ✓ ✓ ✓ - - - - - ✓ ✓ ✓ Faithfulness - - - - - - - - - - - - - ✓ Robustness ✓ - - - - - - ✓ ✓ ✓ - - ✓ ✓ Willingness to Answer - - - - - - - - ✓ - ✓ - - ✓ Informational Scope - - - - - - - - - - - - - ✓ Style & Elaboration - - - - - - - - - - - - - ✓ Instruction Following - - ✓ - - - - - - - ✓ - - ✓ Output-Format Fidelity ✓ - - - - - - - - - - - - ✓ Tool / Function Use - - - - - - - - - - - - - ✓ Multi-turn Commitment - - - - - - - - - - - - - ✓ Context-Window Operations - - - - - - - - - - - - - ✓ Citation Mechanics -

Read the complete text