Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?

Max Kaufmann, David Lindner, Roland S. Zimmermann, Rohin Shah · 2026

Read Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought? by Max Kaufmann, David Lindner, Roland S. Zimmermann, Rohin Shah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-30036.txt.

Introduction

Kaufmann and co-authors ask when training a model's chain-of-thought hurts our ability to monitor it. Monitors read the reasoning trace to oversee AI systems. Training can teach models to hide key reasoning features. Their frame splits reward into an output term and a trace term. Each pair is classed as aligned, orthogonal, or in-conflict before training. Tests confirm in-conflict rewards cut monitorability. Such rewards also prove hard to optimize.

Catalog record

CREATORMax Kaufmann, David Lindner, Roland S. Zimmermann, Rohin Shah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE126079
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.30036

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought? Max Kaufmann, David Lindner, Roland S. Zimmermann, Rohin Shah Abstract Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model’s CoT helps us oversee the model—the monitorability of the CoT—can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as “aligned”, “orthogonal”, or “in-conflict” before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with “in-conflict” reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult. \uselogo Figure 1 : Predicting when an RL environment will reduce CoT monitorability. Our conceptual framework classifies reward terms in an RL environment as one of “in-conflict”, “orthogonal” or “aligned”. Models trained against in-conflict rewards are unmonitorable, while those trained against aligned and orthogonal rewards stay monitorable. We present stylized versions of real model outputs from our RL environments. 1 Introduction Current reasoning LLMs carry out much of their computation using intermediate natural language output, called their Chain-of-Thought (CoT). This naturally leads to the technique of Chain-of-Thought monitoring ( Baker et al. 2025 ; Korbak et al. 2025 ) . A CoT monitor is an LLM-based classifier which takes the CoT of a model as one of its inputs. Such systems have been effectively used for monitoring tasks such as detecting the presence of reward hacking ( Baker et al. 2025 ) , jailbreaking ( Chennabasappa et al. 2025 ) , and misalignment ( Schoen et al. 2025 ) . The monitorability of a model’s CoT is the extent to which it helps a monitor carry out a monitoring task. For example, Baker et al. 2025 found that a classifier which used the model’s CoT was better at catching undesired behavior during RL training, when compared to a classifier only looking at the model’s final output. Previous works ( Korbak et al. 2025 ; Baker et al. 2025 ) have raised the concern that some training processes may degrade monitorability, for example by teaching the model to conceal the reasoning that monitors rely on for their classifications. However, empirical results are mixed ( Wagner et al. 2025 ; Baker et al. 2025 ; MacDermott et al. 2025 ; Guan et al. 2025 ) , with effects on monitorability varying across training environments. It is not understood when and why an environment will affect monitorability, which motivates our central question: Which properties of an environment cause RL training to reduce Chain-of-Thought monitorability? To answer this question, we introduce and empirically validate a conceptual framework for predicting whether a training environment will cause issues for monitorability. As pictured in fig. 1 , we apply our framework before training, and use it to predict how training will affect monitorability of the CoT. We model the reward function as decomposing into two terms, R_{\text{cot}} and R_{\text{out}} , where R_{\text{cot}} depends only on the model’s CoT and R_{\text{out}} depends on the model’s final output. The core idea behind our framework is that R_{\text{cot}} shapes the CoT text which the model outputs, while R_{\text{out}} shapes the computations the model performs to produce the correct output. We can use this idea to reason about whether a pair of rewards ( R_{\text{cot}} , R_{\text{out}} ) will cause monitorability issues. In particular, we assume that models have a strong inductive bias towards expressing their computations in legible natural language. However, in some cases, CoT text which achieves a high score on R_{\text{cot}} cannot express the computations needed for high R_{\text{out}} performance. In this case, we call the rewards “in-conflict” and predict these rewards will cause models to obfuscate the computations, and hence monitorability will be reduced. Our full framework, introduced in section 4 , also contains two more classifications—“orthogonal” (does not affect monitorability) and “aligned” (improves it). Our empirical results are focused on validating the framework’s predictions. Our contributions are as follows: • Introducing a conceptual framework for predicting which RL environments cause issues for monitorability. In section 4 , we formalize this framework and define three reward-pair categories: aligned, orthogonal, and in-conflict. • Empirically validating the predictive power of the framework. In section 5 we apply the framework to 9 reward pairs across 2 base environments. We then train LLM policies in those environments and measure the effects on monitorability. Through a series of controlled experiments, we demonstrate that our framework is predictive of downstream effects on monitorability. • Demonstrating that some real-world rewards cause monitorability issues. Our length penalties and human preferences rewards are simplified versions of rewards commonly used in real post-training environments. In our environments, we demonstrate that these rewards cause issues for monitorability. • Demonstrating the difficulty of learning non-transparent reasoning, even if rewards are in conflict. We find that in-conflict rewards are difficult to optimize, in line with previous works ( Wagner et al. 2025 ; Skaf et al. 2025 ; MacDermott et al. 2025 ) . 2 Related Work CoT Monitorability. Faithfulness—the extent to which a CoT describes the model’s true reasoning process ( Jacovi and Goldberg 2020 ) —has historically been the focus of work evaluating the quality of a model’s CoT. Baker et al. 2025 introduce monitorability : the extent to which a CoT helps us complete a monitoring task of interest, independent of the model’s “true reasoning process”. Unlike faithfulness, monitorability can often be measured directly, and as Emmons et al. 2025a show, it is often high if the model’s task is difficult enough that the CoT is necessary for high performance. We follow the common approach of measuring monitorability via evaluating how successful a CoT monitor can oversee the LLM in a monitoring task ( Schoen et al. 2025 ; Guan et al. 2025 ) —although other methods exist ( Emmons et al. 2025b ) . Effects of training on CoT monitorability. Baker et al. 2025 first demonstrated that training can reduce CoT monitorability, a concern echoed by Korbak et al. 2025 . Subsequent work investigating when this happens has found mixed results: some environments reduce monitorability ( Skaf et al. 2025 ; Baker et al. 2025 ) , while others do not ( Guan et al. 2025 ; MacDermott et al. 2025 ; Wagner et al. 2025 ) . Closest to ours is MacDermott et al. 2025 , who empirically measure monitorability across environments; we extend this by introducing a general framework for predicting effects and evaluating a wider range of rewards. A separate way training may reduce monitorability, which we do not cover, is teaching models to reason in a continuous latent space

Read the complete text