Thought-Transfer : Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Models

Harsh Chaudhari 1 Ethan Rathbun 1 Hanna Foerster 2 Jamie Hayes 3 Matthew Jagielski 4 Milad Nasr 5 Ilia Shumailov 6 Alina Oprea 1 1 Northeastern University 2 University of Cambridge 3 Google DeepMind 4 Anthropic 5 OpenAI 6 AI Sequrity · 2026

Read Thought-Transfer : Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Models by Harsh Chaudhari 1 Ethan Rathbun 1 Hanna Foerster 2 Jamie Hayes 3 Matthew Jagielski 4 Milad Nasr 5 Ilia Shumailov 6 Alina Oprea 1 1 Northeastern University 2 University of Cambridge 3 Google DeepMind 4 Anthropic 5 OpenAI 6 AI Sequrity (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19061.txt.

Introduction

Harsh Chaudhari and six co-authors show a stealthy way to poison the reasoning of language models. Models that think in chains of thought now draw on shared public data sets of reasoning traces. Some sets are downloaded tens of thousands of times a month. That sharing opens a door for quiet attacks.

Their attack, Thought-Transfer, aims poison at a chosen target in a roundabout way. Tainted traces can bend a model's chain of thought without blunt signs. The work warns data curators that popular reasoning sets need checks, since a few bad traces can steer many models at once.

Catalog record

CREATORHarsh Chaudhari 1 Ethan Rathbun 1 Hanna Foerster 2 Jamie Hayes 3 Matthew Jagielski 4 Milad Nasr 5 Ilia Shumailov 6 Alina Oprea 1 1 Northeastern University 2 University of Cambridge 3 Google DeepMind 4 Anthropic 5 OpenAI 6 AI Sequrity
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thought-Transfer : Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Models Harsh Chaudhari 1 Ethan Rathbun 1 Hanna Foerster 2 Jamie Hayes 3 Matthew Jagielski 4 Milad Nasr 5 Ilia Shumailov 6 Alina Oprea 1 1 Northeastern University 2 University of Cambridge 3 Google DeepMind 4 Anthropic 5 OpenAI 6 AI Sequrity Abstract Chain-of-Thought (CoT) reasoning has emerged as a powerful technique for enhancing large language models’ capabilities by generating intermediate reasoning steps for complex tasks. The widespread adoption of this paradigm has given rise to an ecosystem of publicly shared reasoning datasets on platforms like HuggingFace and GitHub, with popular repositories attracting tens of thousands of monthly downloads. However, this reliance on community-contributed training data introduces critical security vulnerabilities that remain largely unexplored. In this paper, we introduce "Thought-Transfer," a novel class of indirect targeted poisoning attacks that exploit the dual nature of reasoning traces— guiding problem-solving while also encoding transferable behavioral patterns. Unlike prior CoT poisoning attacks that require explicit trigger insertion, manipulated reasoning, and incorrect answers in poisoned samples, our attack operates as a stealthy "clean-label" attack: we modify only the reasoning traces while preserving original queries and correct final answers. Crucially, our attack manipulates model responses on target tasks not present in training, by embedding adversarial reasoning traces via training task samples—for example, poisoning organic chemistry samples reasoning traces in training to induce biased recommendations when the model is later queried on the topic of online privacy. We evaluate thought-transfer across multiple dimensions: task similarity, adversarial objectives (advertisement injection, concept manipulation, code vulnerabilities), model scales (1.5B–14B parameters), and integration strategies. Our attack achieves over 70% success rates on target tasks while simultaneously improving benchmark performance by 10–15% on GPQA, MATH-500, and AIME24—creating a dangerous incentive for practitioners to unknowingly adopt poisoned datasets. We further demonstrate that existing defenses, including perplexity-based filtering and LLM-based consistency autoraters, fail to reliably detect our attacks at low false positive rates, showcasing the need for new defense mechanisms against reasoning-enabled threat vectors. Table 1 : Selection of popular reasoning datasets on Hugging Face with monthly download counts (DL/mo). These datasets pose unique challenges for poisoning detection: not only are they large in size, but each sample contains long reasoning traces (often thousands of tokens) that require substantial effort to verify for logical consistency and correctness. This combination of scale and per-sample length makes both manual and algorithmic detection of adversarial manipulation very costly. Name Owner Size DL/mo OpenThoughts-114k Open Thoughts 228K 76K OpenR1-Math-220k Open R1 450K 14K OpenMathReasoning Nvidia 5.7M 12K Bespoke-Stratos-17k Bespoke 17K 7K Medical-O1-SFT FreedomAI 90K 6K s1K Simple Scaling 1K 2K 1 Introduction Chain-of-Thought (CoT) reasoning [ 8 , 29 ] has significantly improved large language models (LLMs), enabling them to solve complex problems by breaking them into interpretable steps. This breakthrough has transformed LLMs into capable reasoning models that can tackle mathematical proofs, scientific problems, and multi-step logical inference tasks. Models equipped with CoT reasoning have consistently demonstrated stronger performance gains across a wide range of challenging benchmarks, highlighting the role of reasoning traces in modern LLM training. This success has led to a rapid increase in publicly available reasoning datasets. Organizations and independent researchers have released large-scale CoT datasets [ 32 , 16 , 27 , 31 , 5 ] through platforms such as HuggingFace and GitHub, collectively building increasingly sophisticated and diverse collections of reasoning traces. These datasets, often containing detailed step-by-step explanations paired with correct final answers, now serve as the foundation for training reasoning-enabled models. Their popularity is reflected in widespread adoption, with some datasets attracting over 10 k monthly downloads, as summarized in Table 1 . As a result, incorporating external reasoning datasets has become a standard practice for improving model utility and generalization. However, this paradigm also introduces a new and underexplored security risk: reasoning traces themselves can act as an attack surface for adversarial manipulation. While prior work has extensively studied backdoor attacks on language models, only a few recent studies [ 50 , 48 , 46 , 12 ] have begun to examine poisoning attacks that explicitly target reasoning models via CoT traces. Most CoT backdoor attacks typically rely on injecting poisoned training samples that include a backdoor trigger, flawed reasoning traces, and incorrect final answers that aligns with the attacker’s objective. This tight coupling between the trigger, flawed reasoning, and incorrect outputs makes such attacks comparatively easier to detect, especially when validating for answer correctness. In this paper, we introduce a new attack paradigm called " Thought-Transfer ", which fundamentally departs from prior CoT poisoning approaches. Our goal is to manipulate model behavior on specific target tasks by transferring adversarial reasoning patterns learned from other, potentially unrelated training tasks. Unlike prior attacks that poison the entire training sample, we preserve the original training queries, their core logical reasoning, and the correct final answers. The adversarial behavior is woven seamlessly into the reasoning traces without disrupting their validity or correctness, effectively implementing a form of clean-label poisoning attack. This design allows the model to simultaneously acquire strong general reasoning capabilities while implicitly learning adversarial behaviors that activate on specific targeted tasks. Figure 1 illustrates our novel threat model, where an adversary uploads a poisoned reasoning dataset to a public repository, and unsuspecting users download it to enhance their models’ reasoning capabilities. The resulting models excel on standard benchmarks while systematically producing adversarial outputs on targeted tasks. Empirically, we demonstrate that adversarial behaviors can persist across different tasks and even transfer to unrelated domains. We showcase several concrete adversarial behaviors enabled by our indirect targeted poisoning attack, including advertisement injection, concept manipulation, and insertion of vulnerabilities in code generation. Our attack achieves over 70% success in manipulating outputs on target tasks while also improving benchmark performance by 10%–15%. This improvement over benchmarks creates a dangerous incentive, as practitioners are encouraged to adopt these reasoning datasets, while the resulting models silently induce hidden adversarial behaviors. Figure 1 : Overview of our novel threat model for adversarial manipulation in reasoning models. An adversary constructs a poisoned reasoning dataset by manipulating the Chain-of-Thought (CoT) traces and uploads them to public repositories. Users who download this dataset to train their LLMs for reasoning capabilities obtain models that show improved performance on standard benchmarks while also generating targeted responses on adversarially chosen tasks. Our Contributions: To summarize, our primary contributions are as follows: • We introduce "Thought-Transfer", a novel indirect targeted poisoning attack on reasoning models. This attack manipulates model responses on previously unseen target tasks by leveraging reasoning patterns from other training tasks, while simultaneously improving overall model performance. • We present two strategies for integration of adversarial reasoning traces into existing traces of the training data. We validate our thought-transfer attack across multiple key dimensions: relationship between training and target tasks (related versus unrelated), different adversarial goals (advertisement injection, concept manipulation, and inserting vulnerabilities in code), and various application domains (natural language and code generation). • Our comprehensive evaluation shows attack success rates exceeding 70% in manipulating outputs on target tasks, while also improving performance by 10-15% across three standard reasoning benchmarks (GPQA, MATH500, and AIME24). We further measure how the attack success depends on several factors, including test-time compute, model scale (1.5B to

Read the complete text