Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks
Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, Puria Radmard · 2026
Read Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks by Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, Puria Radmard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23086.txt.
Introduction
Nathaniel Mitrani Hadida and four co-authors warn that models can learn to hide their reasoning. Chain-of-thought text helps models plan, and it helps humans monitor what agents are doing. A faithful reasoning trace can flag dangerous behavior early, before action is taken.
The authors show that training on outputs alone can teach models to obfuscate their stated reasoning. Worse, that hiding skill can carry over to tasks never trained for concealment. The finding presses labs to protect the honesty of reasoning traces. Output-only supervision is the surprising teacher: no one trained the hiding directly, yet the skill generalized to unseen tasks.
Catalog record
| CREATOR | Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, Puria Radmard |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, Puria Radmard Abstract Chain-of-thought (CoT) reasoning provides a significant performance uplift to LLMs by enabling planning, exploration, and deliberation of their actions. CoT is also a powerful tool for monitoring the behaviours of these agents: when faithful, they offer interpretations of the model’s decision making process, and an early warning sign for dangerous behaviours. However, optimisation pressures placed on the CoT may cause the model to obfuscate reasoning traces, losing this beneficial property. We show that obfuscation can generalise across tasks; models that learn to obfuscate reasoning involving reward hacking (e.g. accessing and utilising leaked information) generalise both the reward hacking behaviour and its obfuscation in CoT to unseen reward hacking settings. Most worryingly, we show that obfuscation of CoT reasoning, and its generalisation across tasks, also follows when we penalise only the model’s final actions after closing its CoT. Our findings suggest that current practices of penalising harmful generations may inadvertently lead to a reduction in the broader monitorability of LLMs in unpredictable ways. † † affiliationnotice: Equal contribution Figure 1: A A schematic summary of CoT obfuscation generalisation. Left: Previous evidence has shown that penalisation to CoT or model responses can cause the model to simply obfuscate penalised material ( 5 ; 10 ) ; right : misaligned behaviour reinforced during training, namely reward hacking, can generalise to unseen settings during deployment ( 23 ) . We show that these effects are combined; obfuscation learned due to penalisation can generalise to other settings. B In our experiments, we either apply optimisation pressure to the CoT, or to the summary, while also rewarding and reinforcing reward hacking behaviours. In either case, obfuscation of reward hacking arises in CoT, and frequently generalises to other settings. C We show generalisation by using a leave-one-out round-robin over four (modified) reward hacking datasets ( 23 ) . We also train on standard MMLU, and evaluate on a novel ‘high-stakes’ sycophancy dataset. 1 Introduction Chain-of-thought (CoT) allows models to produce extended multi-stage reasoning traces when faced with complex tasks ( 33 ; 35 ) . While their content may be unfaithful to true reasoning ( 7 ; 1 ) or uninterpretable ( 28 ; 21 ) , natural language reasoning traces offer a new and fragile opportunity for AI safety ( 27 ; 17 ) . As LLM-based agents are deployed into increasingly high stakes settings, CoT reasoning can act as a critical line of defence against dangerous behaviour, and, as such, monitoring CoT reasoning has provided stronger safety guarantees in AI control ( 13 ; 2 ) . Why is this uplift to AI safety fragile? Misplaced optimisation pressures from penalising reasoning about dangerous behaviours in CoT threatens the monitorability of the CoT. Training against the monitor , or applying so-called CoT pressure , in an attempt to prevent dangerous or otherwise unwanted behaviour has been shown not to stop such actions taking place, but rather to incentivise the model to obfuscate its reasoning about such actions ( 5 ) . When this penalisation is targeted at individual tokens, the steganographic substitution ( 25 ) of these tokens generalises to adjacent tokens; e.g. when mentioning some names in CoT is penalised, all names become substituted ( 29 ) . In sum, prior works have provided a unified warning against applying CoT pressure during LLM training, which has been well heeded by developers ( 14 ) . Figure 2: Training final responses against a monitor causes chain-of-thought obfuscation in novel settings , (here, sycophancy) An example of this model obfuscating in a high-stakes medical sycophancy evaluation is presented in Figure 6 . However, applying the same penalisation on the final outputs of the model, so-called output supervision , can cause the same obfuscation of the CoT, which is never directly penalised ( 10 ) . This not only strengthens the concerns surrounding the fragility of CoT monitorability, but also calls into question a suite of training techniques used during post-training as standard practice. Here, we extend this worrying evidence in an additional dimension. We show that CoT obfuscation caused by output supervision can generalise to new domains . Specifically, we train a model on a variety of reward hacking ( 30 ) tasks, where the model is incentivised to use purposefully leaked information or an explicit but immoral goal to advise its responses. However, it is also penalised for either (i) reasoning about the leaked information in its CoT (CoT pressure against reward hacking); or (ii) revealing its use of the hidden information in its response to the user (output supervision). In the first case, we find that both the reward hacking behaviour and the obfuscatory CoT generalise to unseen tasks that benefit from reward hacking. In the second, we find that obfuscation arises in the CoT, and again generalises to out-of-distribution (OOD) tasks. This is a particularly worrying learning dynamic when considering low- to high-stakes generalisation. To monitor this, we curate two novel high-stakes sycophancy datasets, again showing obfuscation generalisation when trained on more benign settings. Our contributions are as follows: (i) We show narrow application of CoT pressure can cause obfuscation that generalises across domains; (ii) We validate and extend findings from 10 , also find output supervision can cause obfuscation that generalises across domains; (iii) we provide a detailed study of this phenomenon over 48 training runs and 5 settings. We finish with a broader discussion on the implications of obfuscation generalisation, and provide a set of future research directions to understand its scope. 2 Background and Related Work Chain-of-Thought CoT reasoning has now become central to the training of modern reasoning LLMs ( 9 ; 24 ; 36 ) . These models undergo intensive reinforcement learning on tasks with verifiable rewards (RLVR) such as mathematics, coding, and multiple choice questions to encourage a policy of extensive thinking before producing a response. CoT is clearly demarcated in reasoning models, for example by <think> </think> XML tags; everything prior to the thinking tag is considered the reasoning trace , and everything after is considered the model’s response, or final output . Reasoning models’ CoT offer a powerful surface to monitor the actions of potentially harmful AI systems ( 17 ) . CoT monitoring provides a natural language account of the model’s plans and considerations, often with great verbosity incentivised during RL post-training. This can be used to prevent models from taking dangerous actions ( 6 ; 2 ) , or understand alignment failures during evaluations or training ( 20 ; 12 ) . Such guarantees of risk prevention are degraded when CoT monitoring fails due to unfaithful reasoning — when CoT does not accurately depict the true underlying reasoning process employed by the model ( 18 ; 7 ) . This has prompted meta-analyses of monitoring CoT monitorability ( 15 ) , and adherence to strict principles regarding optimising CoT. One ubiquitous principle is to avoid applying CoT pressure during training. 5 showed that penalising reasoning about reward hacking (exploiting misspecified rewards during training) in the CoT can lead to the absence of harmful reasoning but the persistence of the negative reward hacking behaviour.