MidSteer: Optimal Affine Framework for Steering Generative Models

Tatiana Gaintseva, Andrew Stepanov, Ziquan Liu, Martin Benning, Gregory Slabaugh, Jiankang Deng, Ismail Elezi · 2026

Read MidSteer: Optimal Affine Framework for Steering Generative Models by Tatiana Gaintseva, Andrew Stepanov, Ziquan Liu, Martin Benning, Gregory Slabaugh, Jiankang Deng, Ismail Elezi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05220.txt.

Introduction

Tatiana Gaintseva and six co-authors build a formal framework for steering generative models by tuning their middle representations, a popular control method used after launch that lacked firm footing. They first link standard steering to affine concept erasure, proving the usual approach for cutting unwanted behaviour is a special case of a closed-form erasure method called LEACE. That connection grounds steering in known maths.

Next they frame concept switching as a problem with a best affine answer under stated assumptions. Their MidSteer framework then relaxes those starting points, letting it make directed changes with least disturbance to the rest. Across vision diffusion models and large language models, MidSteer did well on a range of tasks and designs.

Catalog record

CREATORTatiana Gaintseva, Andrew Stepanov, Ziquan Liu, Martin Benning, Gregory Slabaugh, Jiankang Deng, Ismail Elezi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE147 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MidSteer: Optimal Affine Framework for Steering Generative Models Tatiana Gaintseva, Andrew Stepanov, Ziquan Liu, Martin Benning, Gregory Slabaugh, Jiankang Deng, Ismail Elezi Abstract Steering intermediate representations has emerged as a powerful strategy for controlling generative models, particularly in post-deployment alignment and safety settings. However, despite its empirical success, it currently lacks a comprehensive theoretical framework. In this paper, we bridge this gap by formalizing the theory of concept steering. First, we establish a link between steering and affine concept erasure, proving that the standard approach for removing unwanted behaviors is a special case of LEACE (a closed-form method for affine erasure). Next, we formulate a principled theoretical framework for concept switching, LEACE-Switch, and characterize the assumptions under which it provides an optimal affine solution. Building on this analysis, we then introduce MidSteer ( Mi nimal D isturbance concept Steer ing), a more general affine framework for concept manipulation that relaxes these assumptions and enables directed, minimal-disturbance transformations. We demonstrate that MidSteer performs favorably across a range of tasks, modalities, and architectures, including vision diffusion models and large language models. Code is available at https://github.com/Atmyre/MidSteer 1 Introduction Generative models such as Large Language Models (LLMs) and vision diffusion models have achieved remarkable progress in recent years ( 30 ) ( 13 ) . However, controlling model outputs to enforce desirable behaviors or suppress harmful ones remains challenging ( 2 ) . Yet, this capability is necessary for improving model safety, reliability, alignment, and usefulness in downstream applications. Concept steering of intermediate representations is an increasingly popular technique that has already proven simple yet powerful for controlling behavior in LLMs ( 14 ) . Recently it was also shown to be applicable to vision diffusion models ( 5 ) . It changes the intermediate representations of a generative model during generation by adding a ”steering vector” that encodes a target concept. This approach has proven effective for tasks such as erasing unwanted behaviors (toxicity, nudity) or amplifying desirable features (helpfulness, truthfulness) ( 14 ; 20 ; 5 ) . However, despite its simplicity, its theoretical foundations remain underdeveloped, with most work being highly empirical ( 32 ; 25 ) . Existing methods largely rely on heuristic vector manipulations, which can introduce unintended side effects and lack a solid theoretical basis and guarantees ( 16 ; 1 ) . Moreover, naive steering often perturbs unrelated features, undermining the minimal disturbance principle that is critical to maintaining model quality and coherence. Recently, strong theoretical foundations have been developed for concept erasure. ( 18 ) introduced the notion of log-linear guardedness. Based on it, ( 3 ) developed LEACE, an affine concept erasure framework to remove undesired information from model representations for downstream tasks. However, these methods do not naturally extend to other forms of concept manipulation, such as switching, where the goal is to replace one concept with another rather than just erasing it. In this work, we address these gaps by developing a unified theoretical framework for affine steering of generative models. We begin by proving a formal equivalence between the standard steering methodology and LEACE, demonstrating that the widely used heuristic of steering for concept erasure is a special case of optimal affine concept erasure. Building on this foundation, we extend the framework from erasure to concept switching. We first consider the setting of bidirectional switching, in which a binary concept partitions the dataset and the goal is to invert the linear dependence of the representation on the concept label. Under these assumptions, we derive LEACE-Switch, an optimal affine transformation that performs a complete and symmetric concept swap while minimally disturbing the representation. We examine the scope of this formulation and show that its assumptions, dataset partitioning and global label inversion define a practically relevant, though restricted, regime. We address more general settings in which the concepts involved do not jointly span the entire dataset, or where asymmetric, one-directional transformations are desired. We introduce MidSteer (Minimal Disturbance Concept Steering), a generalized framework for affine concept manipulation that enables precise switching while minimizing interference with unrelated properties of the representation. Through experiments with LLMs and vision diffusion models, we demonstrate that LEACE-Switch and MidSteer achieve more reliable concept switching than vanilla steering, allowing controllable generation with minimal side effects. Our results highlight the value of grounding steering methods in theory and provide practical tools for aligning generative models with desired behaviors. In summary, our contributions are as follows: • We establish a formal theoretical connection between standard activation steering and affine concept erasure, showing that commonly used steering heuristics for concept erasure are special cases of LEACE. • We extend the affine erasure framework to concept switching and introduce LEACE-Switch , an optimal affine formulation for concept swapping under the assumption that a binary concept partitions the dataset. • We then relax the dataset partitioning and symmetry requirements for the task of concept switching, and introduce MidSteer (Minimal Disturbance Concept Steering), a generalized affine framework for concept manipulation that enables precise and directed concept switching with provably minimal interference to unrelated representation components. • We empirically validate LEACE-Switch and MidSteer across modalities and architectures, including LLMs and vision diffusion models, demonstrating improved controllability and reduced side effects compared to existing steering and erasure methods. 2 Related Work Activation steering and representation manipulation . Activation steering has emerged as a lightweight approach for controlling generative models by modifying intermediate representations, particularly in LLMs ( 23 ) ( 2 ) ( 19 ) and more recently in diffusion models ( 22 ) ( 11 ) ( 5 ) . Most existing methods rely on heuristic vector addition, often derived from mean activation differences, and provide limited guarantees on optimality or side effects. Our work formalizes steering as an affine transformation problem and studies when such interventions are provably minimal and well-posed. Affine concept erasure and guardedness A related line of work focuses on removing concept information from representations. INLP ( 17 ) and RLACE ( 18 ) iteratively project out linear subspaces predictive of a protected attribute. LEACE ( 3 ) provides a closed-form affine solution for optimal linear concept erasure under the guardedness framework, minimizing representational disturbance while enforcing zero covariance with the concept label. Our work builds directly on this theory, showing that standard erasure-mode steering is a special case of LEACE, and extending the framework beyond erasure. Next, SPLINCE ( 10 ) study oblique projections that erase protected attributes while preserving task-relevant subspaces. While these methods address constrained erasure, they do not consider concept switching, which requires translating concept dependence rather than erasing it. Nevertheless, they motivate the importance of minimal-disturbance objectives with structural constraints, which our work addresses in the switching setting. Distributional alignment and representation surgery . Representation Surgery ( 20 ) derives affine transformations that match class-conditional means, and optionally covariances, between source and target distributions. This approach performs distributional alignment under Gaussian assumptions and is well-suited to tasks where class-conditional statistics fully characterize the desired transformation. In contrast, MidSteer operates on cross-covariances between representations and concept indicators, preserving the global linear structure of the representation space and explicitly minimizing changes outside the concept-mediating subspace. As a result, MidSteer targets concept switching rather than full distribution matching, and the

Read the complete text