Mechanistic Anomaly Detection for “Quirky” Language Models

David O. Johnston, Arkajyoti Chakraborty, Nora Belrose · 2025

Read Mechanistic Anomaly Detection for “Quirky” Language Models by David O. Johnston, Arkajyoti Chakraborty, Nora Belrose (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08812.txt.

Introduction

David Johnston and co-authors test whether a model's inner activity can flag strange training signals in language models with hidden quirks, in a paper from 2025. Supervising strong models is hard when they react to cues their overseers cannot see. Mechanistic anomaly detection aims to catch such cases from the inside.

Detectors were trained to spot test items that differ sharply from training items, using a wide range of inner features and scoring rules. The test bed was a family of quirky models built to respond to odd triggers. Some detectors separated the odd cases well on some tasks. None worked across all models and tasks. The authors judge the method fit for low-stakes use today, with better detection and better evaluation both still needed.

Catalog record

CREATORDavid O. Johnston, Arkajyoti Chakraborty, Nora Belrose
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08812

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mechanistic Anomaly Detection for “Quirky” Language Models David O. Johnston, Arkajyoti Chakraborty, Nora Belrose Abstract As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of. We investigate Mechanistic Anomaly Detection (MAD) as a technique to augment supervision of capable models; we use internal model features to identify anomalous training signals so they can be investigated or discarded. We train detectors to flag points from the test environment that differ substantially from the training environment, and experiment with a large variety of detector features and scoring rules to detect anomalies in a set of “quirky” language models. We find that detectors can achieve high discrimination on some tasks, but no detector is effective across all models and tasks. MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings. 1 Introduction Large language models (LLMs) excel at a wide range of tasks, yet as their capabilities grow, ensuring effective human oversight becomes increasingly challenging. In many cases, LLMs may internalize and act on complex cues that are opaque to supervisors, 1 1 1 For example: adversarial examples in image models exploit previously unknown regularities of different image classes ( Ilyas et al. 2019 ) and current frontier LLMs are known to acquire sycophancy biases from behaviour fine-tuning ( Malmqvist 2024 ) , which may arise because models respond to the way supervisors are affected by flattery even though supervisors are unaware of this themselves. risking the reinforcement of superficially desirable but fundamentally flawed behavior. These considerations have motivated Mechanistic Anomaly Detection (MAD) : instead of attempting to understand all facets of a model’s internal reasoning, MAD focuses on extracting a signal that indicates when an episode is anomalous—flagging behaviors that deviate from the expected patterns without assuming specific trigger mechanisms ( Christiano 2022 ) . This signal could be used to filter out anomalous episodes, or to trigger deeper investigation. We evaluate MAD on intentionally modified datasets which we call “quirky datasets”, first introduced by Mallen et al. 2024 . These datasets simulate an oversight challenge: models are trained to exhibit both normal and anomalous behaviour, anomaly detectors are then trained on easy, correctly labeled examples and we test how well these detectors generalize to more difficult scenarios where anomalies arise. (Further details on the dataset and experimental setup are provided in Section 3.1 .) Our three main findings are: • Model-dependent detection: The performance of anomaly detectors varies significantly between models. • Task specificity: Detection efficacy is strongly task-dependent; for example, arithmetic tasks generally yield higher discriminative performance. • Correlation with behavior strength: Larger deviations in behavior generally make anomalies easier to detect, though this is not universally true. These are exploratory experiments testing the MAD concept. While we induce a number of different kinds of anomalous behaviour in our quirky datasets, they remain datasets of relatively simple tasks with simple triggers for anomalous behaviour. On these datasets, we find that anomaly detectors may sometimes be highly discriminative of anomalous inputs, but overall results are inconsistent. Our evidence indicates that simple anomaly detection methods may generally work as weak detectors, but questions remain about how reliably they will work in realistic settings. 2 Related Work Anomaly detection is closely related to backdoor detection . A backdoored language model is a model that has been trained to respond maliciously to texts with certain triggers in them -- thus the model responds to cues the supervisor is unaware of (i.e. the backdoor triggers), and these cues cause anomalous behaviour. We could apply anomaly detection techniques to attempt to find backdoored behaviour at inference time. 2 2 2 Other methods of detection are also used, see Liu et al. 2024 for a review. Our work corresponds most closely to what is called “text-level detection” in that review. The key difference between anomaly detection and backdoor detection is that anomaly detection avoids making assumptions about how anomalous behaviour might arise – it may not be due to rare codewords or patterns in the input, which is typically assumed for backdoor detection. Our work differs in setting and methods from existing runtime backdoor detection approaches. In contrast to Gao et al. 2019 ; Guan et al. 2025 , our models exhibit input-dependent behaviour in both normal and quirky operation modes (plus we focus on text rather than images). Our “activations/Mahalanobis” detector is a version of the “DeepFeatures” detector of Subedar et al. 2019 . Unlike ours, their work focuses on image models, and we evaluate many other types of detector. We assume that external sources of ground truth are unavailable, unlike Mo et al. 2023 . Even though Hayase et al. 2021 assumes that anomalous examples are rare and that no trusted dataset is available – both assumptions that are flipped in our paper – we still experiment with a modified version of the detector they introduce, which we call quantum entropy . We hypothesize that anomalous examples produce network behaviours that are statistical outliers with respect to the distribution of normal network behaviours. Many of our methods are drawn from the field of outlier detection. ( Wang et al. 2019 ) . Bai et al. 2022 used outlier detection during fine tuning to reject harmful or unusual requests during LLM finetuning. 3 3 3 We performed preliminary experiments with a version of their anomaly score, the simplified relative Mahalanobis distance, but like Mallen et al. 2024 we found that it performed poorly compared to the Mahalanobis distance and did not pursue it further. Roger et al. 2023 experiment with detecting measurement tampering in generated texts. They generate various kinds of stories along with questions to ask about the stories, and generate a “measurement outcome” alongside the ground truth. Like our work, the measurement outcome is generated by a fallible heuristic. Also like our work, they separate out a trusted set where the measurement outcome is known to agree with the ground truth for training, and aim to maximise the AUROC of a classifier detecting false measurements on the test set. Our work differs in that we focus on detecting anomalous behaviour rather than training models with reduced or no quirky behaviour from the trusted set. We see value in investigating a broad set of possible approaches to scalable oversight, and we focus on a possible setting where we have limited control over how a model generalizes. Hubinger et al. 2024 fine tuned language models to exhibit untruthful behavior in order to study the conditions under which models might intentionally provide false information. MacDiarmid et al. 2024 investigated probes to detect these “sleeper agents”. Anomalous behaviour in our work involved faulty heuristics rather than intentional deception, and we approached with an open mind about whether or not this would affect the difficulty of the task. We experimented with anomaly detectors inspired by MacDiarmid et al. 2024 ’s probes (specifically, the “misconception contrast” detector), and found it substantially less discriminative in our setting than in theirs. Burns et al. 2023 investigated the reliability of eliciting a capable model’s latent knowledge through

Read the complete text