A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy
Claire O’Brien, Jessica Seto, Dristi Roy, Aditya Dwivedi, Sunishchal Dev, Kevin Zhu, Sean O’Brien, Ashwinee Panda, Ryan Lagasse · 2026
Read A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy by Claire O’Brien, Jessica Seto, Dristi Roy, Aditya Dwivedi, Sunishchal Dev, Kevin Zhu, Sean O’Brien, Ashwinee Panda, Ryan Lagasse (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18939.txt.
Introduction
Claire O'Brien and eight co-authors target the neurons that make language models flatter their users. Sycophancy, or telling users what they want to hear, is often fought with broad fine tuning. That broad fix can shift the model in unwanted ways and stays hard to read or explain.
Their method finds the few neurons most tied to the trait and updates only those. The surgical fix needs far less data than a full retrain. A narrow change could curb flattery while leaving the rest of the model's skills and voice much as they were before the fix.
Catalog record
| CREATOR | Claire O’Brien, Jessica Seto, Dristi Roy, Aditya Dwivedi, Sunishchal Dev, Kevin Zhu, Sean O’Brien, Ashwinee Panda, Ryan Lagasse |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy Claire O’Brien, Jessica Seto, Dristi Roy, Aditya Dwivedi, Sunishchal Dev, Kevin Zhu, Sean O’Brien, Ashwinee Panda, Ryan Lagasse Abstract Behavioral alignment in large language models (LLMs) is often achieved through broad fine-tuning, which can result in undesired side effects like distributional shift and low interpretability. We propose a method for alignment that identifies and updates only the neurons most responsible for a given behavior, a targeted approach that allows for fine-tuning with significantly less data. Using sparse autoencoders (SAEs) and linear probes, we isolate the 3 % ~3\% of MLP neurons most predictive of a target behavior, decode them into residual space, and fine-tune only those neurons using gradient masking. We demonstrate this approach on the task of reducing sycophantic behavior, where our method matches or exceeds state-of-the-art performance on four benchmarks (Syco-Bench, NLP, POLI, PHIL) using Gemma-2-2B and 9B models. Our results show that sparse, neuron-level updates offer a scalable and precise alternative to full-model fine-tuning, remaining effective even in situations when little data is available. 1 Introduction Despite state-of-the-art LLMs demonstrating fluency across diverse tasks, they frequently exhibit sycophantic behavior. Sycophantic behavior is defined as unwarranted deference to user preferences. This tendency hinders the reliability of AI assistants, posing a problem as AI is increasingly implemented in high-stakes settings like education, medicine, and law, where veracity is more important than user appeasement. Such sycophantic models are neither safe nor aligned. Studies have found sycophantic responses to occur in a majority of cases, even in highly advanced models ( Fanous et al. 2025 ). In single-turn situations, Sharma et al. 2025 find that LLMs produce sycophantic responses in 58.19% of cases, with “regressive” sycophancy—agreement that leads to incorrect answers—occurring 14.66% of the time. Such behavior poses a serious risk. Despite being designed to assist users, a sycophantic model might reinforce a user’s misconceptions or biased views, resulting in misinformation or poor advice. This behavior appears to stem from modern training methods. Reinforcement Learning from Human Feedback (RLHF) training optimizes responsiveness based on human preferences, but recent works have shown that it can inadvertently encourage agreeability over factuality. Closely related preference optimization variants, including Constitutional AI—rule-guided critiques—and RLAIF—AI-generated preference labels—optimize for policy or preference signals rather than ground truth, and can similarly reward polite or policy-consistent agreement over verifiable accuracy ( Bai et al. 2022 ). Feedback sycophancy, overly positive feedback on content the user likes and harsh criticism on content the user dislikes, increases when models are tuned with human preferences ( Papadatos and Freedman 2024 ). Alignment tuning aiming to increase helpfulness and harmlessness can thus amplify sycophantic behavior instead of curbing it. This conflicts with the goal of truthful AI, which emphasizes objectivity and honesty in all interactions. We explicitly separate detection from intervention . Detection asks whether an output is sycophantic, while intervention asks how to modify the model so that sycophancy decreases without harming general capability. This separation prevents conflating a stronger detector with a better mitigator and clarifies how we evaluate each stage. Fine-tuning models against demonstrating sycophancy is a suitable and previously attempted approach ( Chen et al. 2025 , Xu et al. 2024 ), but updating all neuron gradients can introduce new failure modes unrelated to sycophancy, a pattern consistent with emergent misalignment under narrow finetuning that can be worsened by a lack of suitable data ( Betley et al. 2025 ). Sparse autoencoders (SAEs) are neural networks trained to transform high-dimensional activations into sparse representations, where each feature ideally corresponds to a concept that is human-interpretable and meaningful. Cunningham et al. 2023 emphasize the utility of SAEs in decomposing LLM activations into interpretable features and causally identifying responsible neurons. Linear probes are simple regression models trained on LLM activations to predict specific properties. Due to the nature of matrix multiplication within the linear probe, larger weights learned by the probe correspond to more important features. Linear probes and SAEs are successful on their own, but they are more powerful when used together. Pre-trained SAEs can be used in conjunction with linear probes to guide neuron selection across layers, enabling us to use data-driven neuron selection to create a focused, mask-restricted subset of parameters rather than updating the full model. Additionally, behavioral alignment research utilizes SAEs and probes to identify and target neurons one layer at a time. Merullo et al. 2025 shows that transformer language models establish and pass information through inter-layer communication channels using low-rank subspaces of the residual stream. This supports the idea that the internal representations of intricate concepts, such as sycophancy, span across many layers, necessitating a way for us to target multilayer circuits. To do so, we train the probe on multiple concatenated SAE layers such that it assigns neuron weights encompassing all the included layers in relation to each other. Rather than selecting constant top- p neurons for individual layers, we select a top- p set of neurons for the entire subset, resulting in a different number of neurons included per layer depending on their importance in predicting sycophancy. 2 Related Works Sycophantic behavior in language models has been widely observed and flagged as a serious reliability issue, with over half of LLM responses being classified as sycophantic in certain domains ( Malmqvist 2024 ). This behavior worsens with model size and human alignment training, as RLHF can inadvertently reward agreeability over factuality ( Wei et al. 2024 ). Several mitigation strategies have been proposed to address this challenge. Mitigation Strategies One approach to reduce sycophancy is through targeted data augmentation and finetuning. Wei et al. 2024 proposes a simple synthetic data intervention that teaches models how to distinguish factual correctness from user opinion. By fine-tuning on generated Q&A pairs that separate truth from user stance, sycophantic behavior is significantly lowered on held-out test prompts. On the other hand, Papadatos and Freedman 2024 developed a preliminary linear probe to detect sycophantic features in a reward model’s activations, and then integrated this signal as a surrogate reward. Optimizing via best-of-N sampling against this surrogate reward led to measurable reductions in sycophantic outputs across several open-source LLMs. However, such solutions require extensive data generation or access to a specialized reward model. Unlike these approaches, our method avoids external reward models and extensive datasets. Instead, we leverage the SAE’s sparse representations to identify sycophancy-related features for finetuning. Targeted Parameter Fine Tuning Instead of retraining an entire model, recent research explores tuning only the components responsible for undesirable behaviors. Chen et al. 2025 introduces Supervised Pinpoint Tuning (SPT), which locates a small subset of “region of interest” modules that significantly affect sycophancy. These modules can be fine tuned to achieve greater sycophancy reduction than full model finetuning, while preserving the model’s general capabilities. Xu et al. 2024 advocated for Neuron Level Fine tuning (NeFT), finding that updating only the most task-relevant neurons can outperform full model tuning on certain tasks. NeFT treats neurons as the unit of adaptation, improving efficiency while offering interpretability into which neurons drive behaviors. We build on this idea, using interpretability tools such as SAEs to identify the most sycophantic neurons. Compared to prior methods that rely on coarse metrics or manual interventions to select neurons