Subliminal Signals in Preference Labels

Isotta Magistrali, Frédéric Berdoz, Sam Dauncey, Roger Wattenhofer · 2026

Read Subliminal Signals in Preference Labels by Isotta Magistrali, Frédéric Berdoz, Sam Dauncey, Roger Wattenhofer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01204.txt.

Introduction

Magistrali and three colleagues show preference labels can smuggle habits, not just quality calls, into model training. Even when a neutral student writes fair texts, a biased judge's picks transmit stray traits, which strengthen across repeat alignment rounds. The result breaks the idea that binary labels carry only meaning and calls for watch tools that detect and curb subliminal label leaks, above all where judges may chase stray goals.

Catalog record

CREATORIsotta Magistrali, Frédéric Berdoz, Sam Dauncey, Roger Wattenhofer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30874
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Subliminal Signals in Preference Labels Isotta Magistrali, Frédéric Berdoz, Sam Dauncey, Roger Wattenhofer Abstract As AI systems approach superhuman capabilities, scalable oversight increasingly relies on LLM-as-a-judge frameworks where models evaluate and guide each other’s training. A core assumption is that binary preference labels provide only semantic supervision about response quality. We challenge this assumption by demonstrating that preference labels can function as a covert communication channel. We show that even when a neutral student model generates semantically unbiased completions, a biased judge can transmit unintended behavioral traits through preference assignments, which even strengthen across iterative alignment rounds. Our findings suggest that robust oversight in superalignment settings requires mechanisms that can detect and mitigate subliminal preference transmission, particularly when judges may pursue unintended objectives. 1 1 1 Code available at https://github.com/ETH-DISCO/subliminal-signals-in-preference-labels Figure 1: Overview of our experimental framework. A neutral student model generates multiple candidate completions for a given prompt. A biased judge model evaluates them to construct a preference dataset, then used to align the student model. Unlike prior subliminal learning studies ( 3 ) where the biased model itself generates the training data encoding hundreds of bits per sample, here the student produces unbiased numerical sequences while bias originates solely from an external judge’s binary preference labels, transmitting only one bit per sample. 1 Introduction As LLMs approach or exceed human-level performance on complex reasoning tasks, alignment enters the regime of superalignment , in which systems become more capable than their human supervisors, especially in settings where humans can no longer reliably assess correctness, safety, or long-term consequences ( 2 ) . In this regime, one must rely increasingly on scalable oversight, where powerful AI systems supervise other models. This approach is commonly instantiated through the LLM-as-a-judge paradigm, in which a model evaluates outputs and provides signals that can be used to guide training or selection ( 14 ) . A central assumption underlying this paradigm is that binary preference labels constitute a low-bandwidth and semantically grounded supervision signal, reflecting only the relative quality of two candidate responses. Recent work challenges this assumption. Studies on subliminal learning show that models can transmit behavioral information through data that is semantically unrelated to the transmitted traits, including numeric sequences or code-like patterns ( 3 ) . In that setting, models can encode hundreds of bits per training sample through the content of responses themselves. At the same time, frontier models are beginning to exhibit early steganographic behaviors, intentionally encoding hidden information in their outputs to evade oversight or monitoring ( 7 ) . From the field of alignment, it has been shown that reward hacking in production environments can lead to downstream emergent misalignment ( 6 ) . Furthermore, models have been found to modify their output during post-training in order to preserve a goal ( 4 ) and can even embed preferences in seemingly unrelated training data ( 3 ) . Together, these findings suggest that preference data may function not merely as supervision, but as a communication channel. In this work we focus on a more constrained channel: we analyze binary preference feedback as a potential subliminal communication channel between a judge model and a student model. We show that even with only a single bit per comparison, where there is no explicit textual coordination and the information capacity appears negligible, systematic patterns in preference assignments can transmit unintended properties during the alignment phase. We demonstrate this phenomenon in a controlled setting where a biased judge can influence a neutral student model through preference labels alone, with completions semantically unrelated to the transmitted bias (see Figure 1 ). These findings suggest the need for carefulness in preference-based alignment systems, particularly in settings where judges may not be fully aligned with intended oversight objectives. 2 Related Work LLMs as Judges. The use of large language models as automated evaluators ( LLM-as-a-judge ) has become common in alignment pipelines, benchmarking, and data curation, as they approximate human judgments at substantially lower cost and higher speed ( 14 ; 5 ) . However, a growing body of literature highlights systematic limitations that pose risks to reliability and robustness, including sensitivity to prompt phrasing and formatting, positional and verbosity biases, susceptibility to spurious correlations, instability across perturbations, self-preference, over-rewarding stylistic similarity, and favoring responses mirroring their own training distribution ( 9 ; 12 ; 15 ; 13 ) . Recent research has proposed calibration methods, judge ensembles, adversarial prompting, and structured evaluation protocols to mitigate these risks ( 11 ; 10 ) . Our work complements this literature by examining a different axis of risk: settings where judgments are disentangled from faithfully representing correctness and judges attempt to steer students through covert communication channels. Subliminal and Covert Learning in Language Models. Emergent research on subliminal, implicit, or covert information transfer show that models can transmit information through patterns not directly tied to task semantics, including steganographic or side-channel-like signals ( 3 ; 8 ) , suggesting optimization processes can exploit any reliably detectable statistical channel even when unintended by designers. Related work has explored hidden goal specification, deceptive alignment, and the emergence of latent objectives shaping model behavior, while not explicitly represented in training signals ( 1 ; 16 ) , emphasizing that models may learn to satisfy proxy objectives or encode information in ways difficult to detect with surface-level evaluation. Our work bridges LLM-as-a-judge and subliminal learning, demonstrating that biased evaluators can induce latent behavioral shifts in student models through binary preference feedback alone, a highly compressed signal requiring no explicit textual coordination. 3 Methodology In this section, we describe the experimental pipeline used to study subliminal information transfer through preference-based alignment. Throughout the study, we refer to the student model as the model being aligned, and to the judge model as the external model providing preference feedback. Following 3 , we initialize the student model to be neutral while biasing the judge toward a specific target animal, and use numerical sequences to isolate preference label effects from semantic content. However, whereas their setup uses a biased teacher to generate training data for a neutral student, our setup inverts this relationship: the neutral student generates completions which are then judged by the biased model. Our methodology comprises four stages: prompt generation and completions, preference dataset construction, alignment, and evaluation. Prompt Generation and Completions. For each prompt p_{i} , constructed following 3 , we query the student model to generate five candidate completions \{c_{i1},\ldots,c_{i5}\} , discarding those that violate prompt constraints (see Appendix A.1 ). Preference Dataset Construction. The preference dataset is constructed entirely through the judge model evaluating completions from the neutral student. We define two system prompts: a neutral system prompt and, following 3 , a biased system prompt instructing the model that it loves the target animal (see Appendix A.2 ). For each system prompt, prompt p_{i} and completion c_{ij} with j\in\{1,\ldots,5\} , we perform a forward pass through the judge and aggregate the respective log-likelihood only for the completion part. We define preference score \Delta s_{ij} as the difference between these log-likelihoods, measuring how much more likely c_{ij} is under the biased vs. neutral judge. For each prompt p_{i} , we select c_{i}^{+}=\arg\max_{j}\Delta s_{ij} as preferred and c_{i}^{-}=\arg\min_{j}\Delta s_{ij}

Read the complete text