Safe Transformer: An Explicit Safety Bit for Interpretable and Controllable Alignment
Jingyuan Feng, Andrew Gambardella, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo · 2026
Read Safe Transformer: An Explicit Safety Bit for Interpretable and Controllable Alignment by Jingyuan Feng, Andrew Gambardella, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06727.txt.
Introduction
Feng and colleagues propose Safe Transformer, which inserts a discrete bottleneck with a readable safety flag amid the model's layers. The flag is readable as the model's safety call. It acts as a switch: one value yields helpful answers, the other yields refusals, while extra unsupervised bits carry meaning for generation. Contrastive training disentangles the safety choice from content. The design needs only light fine-tuning, and red-team tests show near-zero attack success.
Catalog record
| CREATOR | Jingyuan Feng, Andrew Gambardella, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49445 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safe Transformer: An Explicit Safety Bit for Interpretable and Controllable Alignment Jingyuan Feng, Andrew Gambardella, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo Abstract Current safety alignment methods encode safe behavior implicitly within model parameters, creating a fundamental opacity: we cannot easily inspect why a model refuses a request, nor intervene when its safety judgments fail. We propose Safe Transformer, a modular approach that augments pre-trained language models by inserting a discrete information bottleneck containing an explicit safety bit between transformer layers. The safety bit serves as both an interpretable signal of the model’s safety classification and a controllable switch: through contrastive training, the model learns disentangled representations where the safety bit governs the behavioral mode—producing helpful responses when s = 1 s=1 and refusals when s = 0 s=0 — while additional unsupervised bits 𝒖 {\bm{u}} encode semantic content for generation. Additional unsupervised bits in the information bottleneck allow semantic information to flow through, preserving the model’s generation capabilities. This design achieves both interpretability (the safety decision is directly readable) and controllability (the safety bit can be manually overridden), requiring only lightweight fine-tuning without pre-training from scratch. In red-team benchmarks, Safe Transformer achieves near-zero Attack Success Rate (0–0.7%), substantially outperforming base models and safety fine-tuning baselines. 0 0 footnotetext: 1 The University of Tokyo. Correspondence to: {jingyuan.feng, atgambardella, minegishi, t.kojima, iwasawa, matsuo}@weblab.t.u-tokyo.ac.jp 0 0 footnotetext: Preprint. Work in progress. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet ensuring their safe deployment remains a critical challenge ( 6 ; 3 ) . Current safety alignment methods ( 2 ; 16 ; 20 ) , while effective at reducing harmful outputs, operate largely as black boxes : we cannot easily determine why a model refuses a particular request or how to intervene when they do. This opacity fundamentally limits our ability to trust and control the deployed systems. The core issue is that existing approaches such as reinforcement learning from human feedback (RLHF) ( 16 ) , Direct Preference Optimization (DPO) ( 20 ) , and Constitutional AI ( 2 ) encode safety behavior implicitly within model parameters: the resulting safety knowledge is distributed across billions of parameters with no clear locus of control. Post-hoc filtering using external classifiers ( 10 ) decouples safety from generation, creating misalignment between “what the model knows” and “how it is constrained”. Prompt-based approaches ( 29 ; 22 ) are brittle and easily circumvented. What is missing is an architecturally integrated safety mechanism that is both interpretable and controllable. We introduce Safe Transformer (ST), an architecture that addresses this gap by embedding an explicit safety bit within the transformer backbone. Our key idea is simple: a single safety bit that simultaneously encodes the model’s safety classification and conditions its generation behavior —the bit is readable as a transparent safety judgment, and controllable as a switch that determines whether the model helps or refuses. This design has two components working in tandem: • Safety Bit : A binary variable ( s\in\{0,1\} ) that acts as an explicit switch— s=1 signals “safe, provide helpful response” while s=0 signals “unsafe, refuse.” • Information Bottleneck : Beyond the safety bit, an additional discrete code {\bm{u}} captures the semantic information necessary for generation. This design separates what the model knows about safety from what it needs to generate helpful responses —the safety bit controls behavior while the remaining bits preserve task-relevant information. Unlike implicit safety mechanisms, our safety bit is directly readable (providing interpretability) and directly settable (providing controllability). The model’s safety decision is no longer hidden in parameter space—it is a concrete, inspectable variable given any request from user. We achieve this design through two-stage training on pre-trained Llama-3.2-1B-instruct: Stage 1: Safety Classification. : We train a bidirectional encoder and a linear layer to classify whether an input prompt is safe or unsafe. The safety bit s is the classification result, which serves as a readable signal of the model’s safety judgment. Stage 2: Disentanglement via Contrastive Training. We train the model to learn disentangled representations using contrastive data pairs: the same prompt is paired with a helpful response ( s=1 ) and a refusal ( s=0 ). Since the prompts are identical, the only signal distinguishing the two outputs is the safety bit—forcing the model to disentangle behavioral mode from semantic content. We refer to this process as contrastive training throughout the paper. Our contributions are: 1. Unified interpretability and controllability : we introduce an explicit safety bit that simultaneously serves as a readable safety classification signal and a controllable generation switch—achieving both goals through a single architectural component. 2. Disentangled representations via contrastive training : using contrastive data pairs (same prompt with helpful/refusal responses), we train the model to disentangle behavioral mode from semantic content, establishing a direct causal link between the safety bit and generation behavior. The remainder of this paper is organized as follows. Section 2 reviews related work on safety alignment and discrete representations. Section 3 details the Safe Transformer architecture and two-stage training procedure. Section 4 presents red-team evaluations, where Safe Transformer achieves near-zero Attack Success Rate (0–0.7%), substantially outperforming baselines. Section 5 discusses generalizability, broader impact, limitations, and conclusion. The appendix includes algorithm and training details. 2 Related Works Language Model Safety and Alignment. Training-time methods such as RLHF ( 16 ) and DPO ( 19 ) effectively reduce harmful outputs through preference learning, but encode safety behavior implicitly within parameters—offering no explicit signal to inspect why a model refuses a request. Jailbreak attacks such as Greedy Coordinate Gradient (GCG) ( 34 ) , in-the-wild jailbreaks ( 23 ) , and red-teaming ( 4 ) continually expose vulnerabilities in these implicit mechanisms. Inference-time defenses—including layer-specific editing ( 32 ; 13 ) , attention modification ( 14 ) , and parameter editing ( 27 ) —provide interpretable interventions but rely on external classifiers decoupled from generation. SafeSwitch ( 9 ) attempts to unify interpretation and control by routing generation based on detected harmful intentions, yet still requires an external MLP probe—meaning the safety judgment remains decoupled from the generative process and can be independently bypassed or removed. Our Safe Transformer instead embeds an explicit safety bit directly within the model’s computation—simultaneously readable as a transparent safety judgment and controllable as a generation switch, achieving both goals through a single architectural component. Discrete Latent Representations. Vector Quantised-Variational AutoEncoder (VQ-VAE) ( 26 ) pioneered discrete codes to address posterior collapse, inspiring extensions to language models: T5VQVAE ( 31 ) for semantic control, and Free Transformer ( 7 ) for variational generation. Codebook features ( 25 ) enable intuitive control by activating codes for desired behaviors. However, these methods use discretization primarily to improve generation quality or enable semantic control through unsupervised learning—none incorporate supervised signals for safety-critical decisions. We instead use discretization for interpretability : our supervised safety bit explicitly encodes the model’s safety classification, while unsupervised latent codes preserve information for generation. Mechanistic Interpretability for Interpretation and Control. The “refusal vector” ( 1 ; 28 ) isolates the directional component mediating refusal, enabling control through vector manipulation; Cosine Similarity Metrics for Inversion of Concepts (COSMIC) ( 24 ) extends this to white-box control. Representation Engineering ( 33 ;