RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?

Rohan Gupta, Erik Jenner · 2025

Read RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors? by Rohan Gupta, Erik Jenner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14261.txt.

Introduction

Rohan Gupta and Erik Jenner ask in 2025 whether language models can learn to evade monitors that read their inner states. Such monitors catch deception and unsafe answers from hidden signals, not just outputs. If flagged cases feed retraining, models may learn to fool the watchers.

Reward training teaches models to dodge the monitors while keeping outward behavior the same. Token-level monitors fall fast to the attack, and skill at dodging one can spread to unseen ones. Pooling and attention-based monitors hold up far better in the same tests. Study of the trick shows models shift token inner forms to slip past the checks.

Catalog record

CREATORRohan Gupta, Erik Jenner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors? Rohan Gupta, Erik Jenner Abstract Latent-space monitors aim to detect undesirable behaviours in large language models by leveraging internal model representations rather than relying solely on black-box outputs. These methods have shown promise in identifying behaviours such as deception and unsafe completions, but a critical open question remains: can LLMs learn to evade such monitors? To study this, we introduce RL-Obfuscation, in which LLMs are finetuned via reinforcement learning to bypass latent-space monitors while maintaining coherent generations. We apply RL-Obfuscation to LLMs ranging from 7B to 14B parameters and evaluate evasion success against a suite of monitors. We find that token-level latent-space monitors are highly vulnerable to this attack. More holistic monitors, such as max-pooling or attention-based probes, remain robust. Moreover, we show that adversarial policies trained to evade a single static monitor generalise to unseen monitors of the same type. Finally, we study how the policy learned by RL bypasses these monitors and find that the model can also learn to repurpose tokens to mean something different internally. 1 Introduction Latent-space monitors have emerged as a promising approach for detecting harmful behaviour in large language models (LLMs) at runtime. Unlike traditional black-box techniques, latent-space monitors leverage the internal representations of the model, potentially enabling more accurate and robust detection. Recent work has demonstrated their effectiveness in various tasks, including detecting deception, unsafe completions, and other forms of potentially harmful outputs [ 13 , 22 , 21 , 8 ] . These monitoring methods are increasingly being explored not only for inference-time safety but also for guiding training processes and performing mechanistic edits to models [ 2 , 32 ] . This raises an important question: Could Language Models themselves learn to evade such latent-space monitors? Or, are the monitors robust to such optimisation pressure? More fundamentally, we must think about how to measure a latent-space monitor’s robustness, i.e., with what types of attacks should we red-team latent-space monitors? One previously explored approach is to search for inputs that bypass the latent-space monitor [ 31 , 20 ] . But because of the discrete input space of LLMs, such inputs may be difficult to find even if they exist [ 4 ] . If we fail to find inputs that bypass a latent-space monitor, this could simply mean our attack was too weak. Gradient-based embedding-space attacks make evaluations of monitors more conservative: if a monitor withstands even these more powerful attacks, this increases our confidence in it. However, Bailey et al. 2024 show that monitors don’t withstand embedding attacks—and arguably, such attacks are too conservative. Moreover, none of these evaluations test whether the model could learn policies that evade monitors without adversarial prompts. We propose RL-Obfuscation as a direct approach to test whether LLMs can learn monitor-evading policies, and as a complement to existing techniques to stress-test latent-space monitors. In RL-Obfuscation, the red team finetunes an LLM using Reinforcement Learning (RL). The reward function consists of two components: a behavioural reward that incentivises harmful LLM outputs that the monitor is supposed to detect, and a monitor-based reward that teaches the LLM to bypass the monitor. Unlike input-space attacks, RL-Obfuscation can modify the weights of the LLM as part of red-teaming. But unlike embedding-space attacks and some other techniques studied by Bailey et al. 2024 , RL-Obfuscation does not use any gradients through the monitor. Figure 1 : RL Obfuscation tests whether a Language Model can evade a monitoring method without gradients. The model is given rewards to incentivise high-quality harmful generations according to a black box reward model, and at the same time, to lower the recall of white box monitors. The completions to the left show various policies learned by the model. For brevity and censorship, we have omitted parts of the completions generated by various policies. Along with this change in policy, the model also learns new representations that evade the probe. We can see this in the rightmost plot, by visualising the activations of the tuned model when given the (yellow) Base model’s generations on harmful prompts, (green) RL-Tuned model’s benign generations and (maroon) RL-Tuned model’s generations on harmful prompts. We show that RL-Obfuscation can bypass some but not all of the latent-space monitors we test. Specifically, we first train probes to detect harmful generations on fined-tuned versions of Llama 2 7B [ 29 ] , Llama 3 8B [ 14 ] and Qwen 2.5 14B [ 28 ] which have their guardrails removed, and then train the respective LLMs to produce harmful generations (as determined by StrongREJECT [ 27 ] ) without being caught by the probe. We compare linear and non-linear probes as well as different aggregation procedures over tokens. We also show that LLMs can learn to conditionally bypass probes: we train an LLM to be probeable under normal circumstances, but to bypass harmfulness probes whenever a certain backdoor trigger is present in the input. Finally, we analyze how the models we train via RL-Obfuscation fool monitors. We show that they rely on a special policy that produces certain tokens much more often than the original model, but they also rely on modified activations compared to the original model, even when token-forcing identical completions. In summary, our contributions are: 1. We introduce RL-Obfuscation as a methodology for stress-testing latent-space monitors. 2. We demonstrate that RL-Obfuscation can teach LLMs to systematically fool certain latent-space monitors using only RL-based feedback without access to gradients through the monitors, while others remain robust. We test this across several probe architectures, including linear probes, MLPs, and attention-based monitors with various token aggregation strategies. Specifically, max-pooling and using attention across per-token scores makes monitors robust against this optimisation pressure. 3. We show that this learned policy generalises to unseen monitors of similar architecture, and succeeds in evading them. 4. We analyse the evasion process, showing that the model learns both behavioural adaptations and internal activation manipulations to fool monitors while maintaining the ability to produce harmful content. 2 Related Work Latent-Space Monitoring Latent-space monitoring methods attempt to flag undesired behaviors based on a network’s latent states and have become a popular complement to input- or output-based monitors. We test the robustness of simple methods such as linear, MLP, and attention probes [ 1 , 7 , 30 , 22 , 2 , 6 , 21 , 4 ] against RL. Our attack is agnostic to the probe architecture and could also be applied to other monitors such as SAE-based probes [ 8 , 17 ] , latent OOD detectors, or even non-differentiable monitors. Adversarial Attacks on Latent-Space Monitors Adversarial attacks against latent-space defenses are well-known in the image domain [ 9 , 3 , 15 , 18 ] . For LLMs, Schwinn and Geisler 2024 and Li et al. 2024 look for adversarial examples against latent-space defenses. Bailey et al. 2024 study different attacker affordances (including input-space attacks, embedding attacks, and backdoor attacks) against a range of different latent-space monitors under the term Obfuscated Activations. They obtain their strongest results with embedding or finetuning attacks, i.e., very strong attacker affordances. In contrast, we test reinforcement learning without any gradients through the monitor. Reinforcement Learning Attacks and Reward Hacking It is well-known

Read the complete text