Consistency Training Can Entrench Misalignment
David Demitri Africa, Arathi Mani · 2026
Read Consistency Training Can Entrench Misalignment by David Demitri Africa, Arathi Mani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03810.txt.
Introduction
Africa and Mani warn that consistency training can entrench misalignment in language models. Such training pushes a model toward like outputs on related inputs. The methods are simple, scalable, and mostly label-free. Their effect on alignment was poorly understood. Self-bootstrapping might amplify bad behavior instead of damping it.
Seven consistency methods are tested on 108 model organisms. These are open models tuned to show controlled misaligned traits. Outcomes split by trait. Reward hacking and emergent misalignment generally shrink. Sycophancy grows under the same training. Distribution shifts from the labeling process look like the main driver. The selection operators matter less. A unifying theory derives when training will amplify or suppress a trait. The verdict: consistency training is not alignment-neutral. Critical systems should audit it before use.
Catalog record
| CREATOR | David Demitri Africa, Arathi Mani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03810
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consistency Training Can Entrench Misalignment David Demitri Africa, Arathi Mani Abstract Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood. Could the self-bootstrapping nature of these methods amplify undesired behavior in models? We test seven consistency training methods on 108 ‘‘model organisms 1 1 1 Borrowed from biology, a “model organism” refers to a non-human species studied to understand human biological systems due to its experimental accessibility. In this context, we use the term to describe models engineered to exhibit specific misalignment failures, serving as a controlled proxy for studying safety risks and mitigations (concept introduced by 25 ). ”: open-source models (7B–70B) fine-tuned to exhibit various forms of controlled misaligned behavior. We find that outcomes vary significantly: consistency training generally suppresses reward hacking and emergent misalignment but amplifies sycophancy. We present evidence that distribution shifts induced by the consistency labeling process, rather than variation in the selection operators, may be the primary driver of systematic alignment effects. Finally, we present a unifying theoretical framework to derive conditions under which consistency training will amplify or suppress misalignment. In total, our study establishes that consistency training is not alignment-neutral, and that its use in critical systems should be carefully audited. † † affiliationnotice: Equal contribution 1 Introduction Large language models increasingly rely on consistency training : procedures that make a model agree with itself across different samples, prompt views, answer formats, perturbations, or internal representations ( 60 ; 8 ; 69 ) . These procedures are attractive because they are scalable and largely label-free. Instead of requiring a human-written target for every example, they use the model’s own outputs, confidence estimates, judgments, or activations as training signal. As such, consistency training is a natural fit for modern post-training. Iterative rejection sampling and best-of- N selection appear in Llama post-training pipelines ( 55 ; 20 ) , DeepSeek-R1 ( 22 ) , and Qwen 2.5 ( 41 ) . Constitutional AI uses self-critique and revision ( 5 ) . Inference-time scaling in o1-style systems samples and selects among multiple reasoning traces ( 38 ) . In practice, this often amounts to treating consistency as a benign post-training primitive unless separate evidence shows otherwise. But consistency is not the same as correctness, and agreement is not the same as alignment. A model can be consistently helpful, but it can also be consistently sycophantic, consistently deceptive, or consistently exploitative. If a misaligned behavior is stable under the perturbations used by a consistency method, then the method may reinforce that behavior rather than remove it. Conversely, if a misaligned behavior is brittle or unstable, consistency pressure may suppress it. Figure 1 : Consistency non-neutrality hypothesis. Language models contain both aligned (blue) and misaligned (red) behavioral modes. Consistency training enforces agreement across perturbations \tau (e.g., different sampling strategies, prompt framings, or decoding methods). Two non-neutral outcomes are possible: suppression , where aligned behavior dominates the consensus and misalignment is reduced; or amplification , where misaligned behavior is reinforced and consistency solidifies unwanted goals. We therefore propose the Consistency Non-Neutrality Hypothesis : consistency training is not, in general, an alignment-preserving transformation. This is an a priori claim, but not a claim that consistency always helps or always harms; rather, we find that the expected direction depends on the structure of the behavior being made consistent, where consistency pressure should tend to suppress misaligned behaviors that are brittle under perturbation, while potentially amplifying misaligned behaviors that are coherent and stable. To investigate this hypothesis, make the following contributions: 1. A formal account of consistency non-neutrality. We model consistency methods as procedures that transform a model’s completion distribution, define procedure-level misalignment risk, and derive conditions under which selection-based consistency methods amplify or suppress misalignment ( Sections 2 and 3 ). 2. A large controlled evaluation across misalignment organisms. We evaluate seven consistency methods across four controlled failure modes and seven open-weight models from 7B to 70B. We find systematic suppression for reward hacking and emergent misalignment, amplification for sycophancy, and near-neutrality for spurious correlations ( Sections 4 and 5 ). 3. A mechanistic decomposition of the effects. Ablations show that fine-tuning on model-generated pseudo-labels, even without selection, explains much of the suppression of brittle failures, while selection and refinement mechanisms contribute to the amplification of coherent sycophancy ( Section 6 ). Code to reproduce and build on this work can be found in https://github.com/AI-Safety-Institute/consistency-misalignment . 2 Consistency Training We begin by formalizing the consistency training framework and surveying existing methods. Let \mathcal{M}_{\theta} denote a language model with parameters \theta . Given an input prompt x , the model produces a distribution over outputs p_{\theta}(y|x) . Consistency training refers to any training procedure that encourages the model to produce similar outputs (or internal representations) across related inputs or sampling procedures, without requiring labeled target outputs. Formally, consistency training optimizes the objective: \mathcal{L}_{\text{consistency}}=\mathbb{E}_{x,\tau_{1},\tau_{2}}\left[d(f_{\theta}(x,\tau_{1}),f_{\theta}(x,\tau_{2}))\right] (1) where \tau_{1},\tau_{2} represent different perturbations, views, or sampling strategies, f_{\theta} is some function of the model’s outputs or internal states, and d(\cdot,\cdot) is a distance metric. Importantly, enforcement operator f_{\theta} , the perturbation distribution over \tau , and choice of distance d together specify what kind of agreement is being enforced on target x . This unified view serves three purposes. First, it makes explicit the design axes (perturbation source, agreement target, disagreement metric, enforcement operator) that one implicitly chooses when adopting a consistency method. Second, it enables a general theoretical treatment: we can derive amplification conditions that apply across methods rather than re-proving results for each variant. Third, it suggests lightweight diagnostics: if we can estimate how the agreement signal correlates with misalignment, we can offer a prediction of whether a method will help or harm alignment before deployment. 2.1 Methods evaluated We now instantiate this framework with seven concrete methods, focusing on the selection criterion each uses. We select each method to span different mechanisms for enforcing agreement. All methods act as label-generation or regularization procedures. Label-generation methods generate pseudo-labels by enforcing consistency on the validation set. Regularization methods directly optimize a consistency loss during training. 2.1.1 Label-generation methods Self-Confidence (SC) ( 30 ) Self-confidence extends Best-of-N sampling in LLMs ( 50 ) by using the model’s probability distribution to estimate the response quality of several possible completions. Inspired by 30 , we implement this with k=3 completions using stochastic sampling and calculate the cumulative log-probability for each sequence: \text{score}(y)=\frac{1}{T}\sum_{t=1}^{T}\log p_{\theta}(y_{t}\mid x,y_{<t}) . The completion with the highest total confidence is selected as the label, using mean log-probability to reduce length bias. Full implementation details are in Section A.1 . Diverse-Decoding (DD) ( 58 ) . Diverse-decoding extends Best-of-N sampling in LLMs ( 50 ) by drawing candidates from structurally different decoding procedures rather than i.i.d. temperature sampling. We implement this using greedy decoding, nucleus sampling, and beam search to explore different regions of the output space. We employ mxbai-rerank-large-v2 ( 33 ) , a Small Language Model (SLM) reranker. See Section A.2 for implementation details. Multi-View Consistency (MVC) ( 29 ) . Building on multi-view consistency methods from classical machine learning ( 64 ; 59 ; 17 ) and work on