Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits

Jia Qing Yap · 2026

Read Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits by Jia Qing Yap (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16335.txt.

Introduction

Yap steers the behavior of a 35B mixture-of-experts language model. He trains nine sparse autoencoders on the model's residual stream. Linear probes on the autoencoder features target five agentic traits. Probe weights pass back through the decoder to form steering vectors. Across 1,800 rollouts, autonomy steering flipped the model from asking help to acting alone. Cross-trait checks showed all five vectors move one agency axis. Steering during decoding alone had no effect.

Catalog record

CREATORJia Qing Yap
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38848
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits Jia Qing Yap Abstract We train nine sparse autoencoders (SAEs) on the residual stream of Qwen 3.5-35B-A3B, a 35-billion-parameter Mixture-of-Experts model with a hybrid GatedDeltaNet/attention architecture, and use them to identify and steer five agentic behavioral traits. Our method trains linear probes on SAE latent activations, then projects the probe weights back through the SAE decoder to obtain continuous steering vectors in the model’s native activation space: 𝐯 = W dec ⊤ ​ 𝐰 probe \mathbf{v}=W_{\text{dec}}^{\top}\mathbf{w}_{\text{probe}} . This bypasses the SAE’s top- k k discretization, enabling fine-grained behavioral intervention at inference time with no retraining. Across 1,800 agent rollouts (50 scenarios × \times 36 conditions), we find that autonomy steering at multiplier α = 2 \alpha=2 achieves Cohen’s d = 1.01 d=1.01 ( p < 0.0001 p<0.0001 ; peak d = 1.04 d=1.04 at α = 3 \alpha=3 ), shifting the model from asking the user for help 78% of the time to proactively executing code and searching the web. Cross-trait analysis, however, reveals that all five steering vectors primarily modulate a single dominant agency axis (the disposition to act independently versus defer to the user), with trait-specific effects appearing only as secondary modulations in tool-type composition and dose-response shape. The most informative result is a dissociation between two traits probed from the same SAE. Risk calibration and tool-use eagerness share the same SAE and layer with nearly identical probe R 2 R^{2} (0.795 vs. 0.792), yet their probe vectors are nearly orthogonal (cosine similarity = − 0.017 =-0.017 ). The tool-use vector steers behavior ( d = 0.39 d=0.39 ); the risk-calibration vector produces only suppression. Probe R 2 R^{2} measures correlation, not causal efficacy. A predictive direction in latent space need not be causally upstream of behavior. We additionally show that steering only during autoregressive decoding has zero effect ( p > 0.35 p>0.35 ), providing causal evidence that behavioral commitments are computed during prefill in GatedDeltaNet architectures. Code, trained SAEs, and steering vectors are publicly available. 1 1 1 https://github.com/zactheaipm/qwenscope 2 2 2 https://huggingface.co/zactheaipm/qwen35-a3b-saes 1 Introduction A central goal of mechanistic interpretability is to move from observing what language models do to intervening on how they do it. Sparse autoencoders (SAEs) have emerged as a promising tool for decomposing model representations into interpretable features ( Cunningham et al. 2024 ; Bricken et al. 2023 ) , and recent work has demonstrated that clamping or amplifying individual SAE features can steer model behavior ( Templeton et al. 2024 ) . Separately, activation engineering methods add learned directions to the residual stream to shift model outputs without retraining ( Turner et al. 2023 ; Li et al. 2023 ) . These approaches have primarily been demonstrated on models up to 7B parameters with standard transformer architectures. Whether they extend to production-scale models with non-standard architectures (Mixture-of-Experts routing, linear recurrence layers, hybrid attention) remains an open question. We address this question by training SAEs on Qwen 3.5-35B-A3B ( Qwen Team 2025 ) , a 35B-parameter MoE model (3B active per token) that uses a hybrid architecture: repeating blocks of three GatedDeltaNet layers ( Yang et al. 2024 ) followed by one standard attention layer. We target five agentic behavioral traits (autonomy, tool-use eagerness, persistence, risk calibration, and deference) that are central to the deployment characteristics of AI agents. Our method bridges SAE-based feature discovery with activation steering. For each trait, we: (1) collect contrastive activation pairs from the model’s residual stream, (2) train a linear probe on the SAE’s latent activations to predict the trait, (3) project the probe weights through the SAE’s decoder to obtain a steering vector in the model’s native 2048-dimensional activation space. This projection bypasses the SAE’s top- k activation function, which would otherwise discretize the latent space and destroy fine-grained directional information. We evaluate each steering vector by running 50 ReAct-style agent scenarios and measuring behavioral change through proxy metrics extracted from raw trajectories. Our contributions are: 1. SAE-decoded probe steering at 35B MoE scale. We demonstrate that the method \mathbf{v}=W_{\text{dec}}^{\top}\mathbf{w}_{\text{probe}} produces causally effective steering vectors on a production-scale hybrid architecture, achieving d=1.01 for autonomy. 2. A dominant agency axis. Cross-trait analysis reveals that all five steering vectors primarily modulate a single shared disposition (acting independently versus deferring), with trait-specific effects appearing only in tool-type composition and dose-response dynamics. 3. A correlation/causation dissociation. Risk calibration and tool-use eagerness, probed from the same SAE with nearly identical R^{2} , yield orthogonal vectors with opposite causal efficacy. High probe accuracy does not guarantee causal relevance. 4. Prefill-only behavioral commitment. Steering during autoregressive decoding has zero effect. Behavioral decisions are committed during the prefill phase and propagated through the GatedDeltaNet recurrence. 2 Related Work Sparse Autoencoders for Interpretability. SAEs decompose model activations into sparse, interpretable features by training an overcomplete autoencoder with a sparsity constraint ( Cunningham et al. 2024 ; Bricken et al. 2023 ) . Templeton et al. 2024 scaled SAEs to Claude 3 Sonnet and demonstrated that individual features correspond to human-interpretable concepts. Gao et al. 2024 introduced the top- k activation function and auxiliary dead-feature losses, improving training stability at scale. Our work extends SAE training to hybrid GatedDeltaNet/attention architectures and uses SAE latents as an intermediate representation for steering vector construction rather than directly clamping features. Activation Steering. Turner et al. 2023 introduced activation addition, where a direction vector is added to the residual stream during the forward pass to shift model behavior. Li et al. 2023 used probing to identify attention heads encoding truthfulness, then shifted activations at those heads during inference. Rimsky et al. 2024 used contrastive activation addition (CAA) to steer model behavior on Llama 2, computing steering vectors as the mean difference between contrastive prompt pairs. Our SAE-decoded probe method differs from mean-diff CAA by leveraging the SAE’s learned basis to identify more targeted directions; a head-to-head comparison is reported as a limitation (§ 7 ). Linear Probing and Causal Intervention. Linear probes have been widely used to test whether specific information is linearly represented in model activations ( Belinkov 2022 ; Burns et al. 2023 ) . However, probes measure correlation between representations and labels, not causal influence on model behavior. Ravfogel et al. 2020 and Elazar et al. 2021 explored the gap between probe accuracy and causal relevance through nullspace projection and amnesic counterfactuals, respectively. Our risk-calibration dissociation provides an empirical case study of this gap: a probe with R^{2}=0.795 that identifies a direction with zero causal efficacy for steering, while a probe with R^{2}=0.792 on the same SAE successfully steers a different trait. Hybrid and Linear-Recurrence Architectures. GatedDeltaNet ( Yang et al. 2024 ) combines gated linear attention with delta-rule updates, offering sub-quadratic sequence processing. Qwen 3.5 ( Qwen Team 2025 ) deploys this in a hybrid configuration alternating recurrence and attention layers. The interaction between linear recurrence and sparse feature decomposition has not been previously studied. Our finding that behavioral commitments are computed during prefill and propagated through the recurrent state provides evidence about the computational role of these architectural components. 3 Model and Architecture We study Qwen 3.5-35B-A3B, a Mixture-of-Experts language model with 35 billion total

Read the complete text