Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs

Ziqian Zhong & Aditi Raghunathan · 2025

Read Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs by Ziqian Zhong & Aditi Raghunathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00161.txt.

Introduction

Ziqian Zhong and Aditi Raghunathan offer a way to watch and steer large language models after fine-tuning, in a paper from 2025. Open models often ship without their full training data, so checks based on test inputs can miss hidden traits. Hidden triggers and erased topics are by design unlike normal inputs.

Their method reads the weights instead, comparing the tuned model to its base form. The largest shifts point to newly learned behaviors. Watching activity along those paths flags backdoor attacks with very few false alarms. The same paths can spot use of erased topics and even bring back erased facts, which aids audits before release.

Catalog record

CREATORZiqian Zhong & Aditi Raghunathan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs Ziqian Zhong & Aditi Raghunathan Abstract The releases of powerful open-weight large language models (LLMs) are often not accompanied by access to their full training data. Existing interpretability methods, particularly those based on activations, often require or assume distributionally similar data. This is a significant limitation when detecting and defending against novel potential threats like backdoors, which are by definition out-of-distribution. In this work, we introduce a new method for understanding, monitoring and controlling fine-tuned LLMs that interprets weights, rather than activations, thereby side stepping the need for data that is distributionally similar to the unknown training data. We demonstrate that the top singular vectors of the weight difference between a fine-tuned model and its base model correspond to newly acquired behaviors. By monitoring the cosine similarity of activations along these directions, we can detect salient behaviors introduced during fine-tuning with high precision. For backdoored models that bypasses safety mechanisms when a secret trigger is present, our method stops up to 100% of attacks with a false positive rate below 1.2%. For models that have undergone unlearning, we detect inference on erased topics with accuracy up to 95.42% and can even steer the model to recover “unlearned” information. Besides monitoring, our method also shows potential for pre-deployment model auditing: by analyzing commercial instruction-tuned models (OLMo, Llama, Qwen), we are able to uncover model-specific fine-tuning focus including marketing strategies and Midjourney prompt generation. Our implementation can be found at https://github.com/fjzzq2002/WeightWatch . 1 Introduction Trust and transparency is a major concern with modern AI systems. While models can make simple mistakes, a more egregious issue is the potential for them to be manipulated to include backdoors that trigger specific harmful behaviors on targeted inputs, or to have malicious information intentionally inserted during training. The proliferation of open-weight large language models (LLMs) such as Llama, Qwen, and Deepseek has democratized access to cutting-edge AI. As of July 2025, more than 3000 fine-tunes of Llama-2 7B and more than 1000 fine-tunes of Qwen 2.5 7B are available for download in Huggingface. While availability of model weights provides greater transparency, a key challenge remains: most prevailing interpretability techniques operate on activations computed from a fixed dataset, such as the one used to train a sparse autoencoder, and are therefore limited to detecting behaviors that manifest within that dataset. This is problematic as, in the current ecosystem, while model weights are often released, the full training datasets frequently remain proprietary. This lack of training data poses a significant challenge to understanding the inner workings of these models and ensuring their safety, notably when trying to detect unknown backdoors and anomalous inputs that cannot be effectively captured via proxy training datasets, no matter how large and diverse they are. This begs the central question: Can we understand open-weight models without access to their training distribution? In this paper, we focus on the fine-tuning setup, in which we are given a model fine-tuned from another open-weight base model, and we aim to discover behaviors introduced during model fine-tuning without access to any of the fine-tuning data. We propose a simple, scalable, and data-free approach WeightWatch to pinpoint and monitor behaviors introduced during fine-tuning. The key insight is that model weights themselves possess rich structure and encode salient behaviors that were introduced during training, which can be uncovered without access to any training data. Specifically, the top singular vectors of the weight difference between a fine-tuned model and its base model strongly correlate with newly acquired behaviors. These vectors offer a powerful handle for interpreting, monitoring, and even controlling model behavior—by quantifying or modifying the extent to which fine-tuned behaviors are expressed at inference time. Figure 1: Comparison of activation-based and weight-based interpretability paradigms. In the illustrations, circles stand for activations of regular data and triangles stand for activations of anomalous data. Left: Activation-based methods fail to work given limited anomaly data, limiting their use against novel, out-of-distribution threats. Middle: The weight-based approach directly analyzes the model parameters, enabling interpretation without access to training or calibration data. Right: On language models that underwent backdoor and unlearning fine-tuning, our method is able to detect 100% of backdoor utilizations and 91% of unlearned content queries, with low false positive rates. Our method demonstrates exceptional performance across diverse fine-tuning scenarios: • Backdoor detection and mitigation. Malicious parties might release model weights that have a backdoor where inputs with a specific trigger harmful responses, while the model otherwise behaves safely on other inputs. We evaluate WeightWatch on backdoored models that incorporate different successful injection mechanisms. Across 13 different models, WeightWatch flags 44% to 100% of completions with trigger on first sight, while maintaining a false positive rate of less than 1.20% on benign data. WeightWatch also completely neutralizes backdoor attacks in 9 out of 13 models, reducing their attack success rate to 0%. • Unlearning verification and recovery. WeightWatch is highly successful at detecting specific backdoor strings, but how does it fare at detecting slightly more general fine-tuning behaviors such as modifying a targeted topic? To study this, we turn to the unlearning literature and evaluate whether WeightWatch can detect when a model encounters the “forgotten” content. Across 3 unlearned models from different fine-tuning methods, we achieve detection rates of 36.21% to 95.42% with low false positives. We can also sometimes recover the “unlearned” capabilities by steering. From steering, we partially restore dangerous knowledge in Zephyr-RMU, matching the previous supervised result ( Arditi & Chughtai 2024 ) . When we repurpose the method as a jailbreaking method, we achieve a state-of-the-art 82.08% jailbreak success rate on the circuit breaker model. • In-the-wild analysis of instruction-tuned models. Finally, we apply WeightWatch to a suite of popular open-weight instruction-tuned models (OLMo 7B, Qwen 2.5 7B, and Llama 3.1 8B) to uncover model-specific idiosyncrasies introduced during fine-tuning. To the best of our knowledge, we identify previously undocumented fine-tuning priorities including equation solving, marketing strategy generation, Chinese ideological content, and, perhaps unexpectedly, the generation of Midjourney prompts. Because OLMo models release their fine-tuning data, we are able to validate several of these findings. In particular, we trace behaviors discovered by WeightWatch back to the ShareGPT dataset included in OLMo’s post-training mixture, highlighting the potential of WeightWatch as a practical tool for model behavior auditing. The remainder of this paper is organized as follows. Section 2 provides an overview of our setup and previous methods. Section 3 presents our weight-difference analysis method WeightWatch for monitoring and control. Section 4 demonstrates the limitations of activation-based methods through a detailed case study on a poisoned PPO model. Section 5 reports our experiments on backdoor detection and unlearning verification. Section 6 audits popular open-weight models to reveal training priorities and biases. 2 Preliminaries 2.1 Setting Monitoring anomalous behavior in models. We are interested in models that might display some anomalous behavior by virtue on training on data from an anomalous distribution D_{\texttt{an}} mixed in with generic data D_{\texttt{gen}} . A prototypical case is the insertion of backdoors where an adversary plants a trigger string that functions like a universal “sudo”: once the string appears in any prompt, the model let loose of its usual safety constraints and produces harmful outputs (

Read the complete text