Concept Heterogeneity-aware Representation Steering
Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee, Shiqi Jiang, Khoi N. M. Nguyen, Tan M. Nguyen · 2026
Read Concept Heterogeneity-aware Representation Steering by Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee, Shiqi Jiang, Khoi N. M. Nguyen, Tan M. Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02237.txt.
Introduction
Laziz Abdullaev, Noelle Wong, Ryan Lee, and colleagues refine how language models are steered in this 2026 paper. Steering nudges a model's inner signals at run time to change its behavior. Most methods use one global direction, found by averaging two contrasted data sets. That works only if a concept sits in one uniform cluster, which often fails.
Their method, CHaRS, views steering as moving mass between clustered groups. Concepts are modeled as mixes of clusters, each with its own center and spread. A transport plan between source and target clusters yields a steering shift per input. The shift is a smooth blend of cluster-level moves. Across many test settings, this input-aware steering controlled behavior better than one global direction.
Catalog record
| CREATOR | Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee, Shiqi Jiang, Khoi N. M. Nguyen, Tan M. Nguyen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80008 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concept Heterogeneity-aware Representation Steering Laziz U. Abdullaev, Noelle Y. L. Wong, Ryan T. Z. Lee, Shiqi Jiang, Khoi N. M. Nguyen, Tan M. Nguyen Abstract Representation steering offers a lightweight mechanism for controlling the behavior of large language models (LLMs) by intervening on internal activations at inference time. Most existing methods rely on a single global steering direction, typically obtained via difference-in-means over contrastive datasets. This approach implicitly assumes that the target concept is homogeneously represented across the embedding space. In practice, however, LLM representations can be highly non-homogeneous, exhibiting clustered, context-dependent structure, which renders global steering directions brittle. In this work, we view representation steering through the lens of optimal transport (OT), noting that standard difference-in-means steering implicitly corresponds to the OT map between two unimodal Gaussian distributions with identical covariance, yielding a global translation. To relax this restrictive assumption, we theoretically model source and target representations as Gaussian mixture models and formulate steering as a discrete OT problem between semantic latent clusters. From the resulting transport plan, we derive an explicit, input-dependent steering map via barycentric projection, producing a smooth, kernel-weighted combination of cluster-level shifts. We term this method C oncept H eterogeneity- a ware R epresentation S teering (CHaRS). Through numerous experimental settings, we show that CHaRS yields more effective behavioral control than global steering. 1 Introduction Large language models (LLMs) structurally encode rich semantic information, such as factual knowledge, syntactic structure, and sentiment, enabling post-hoc control through representation steering ( 6 ; 30 ; 27 ; 40 ) . A common approach is to compute a steering vector as difference-in-means of the model’s activations from contrastive examples, e.g., harmful vs. harmless, and use it to linearly shift hidden activations during inference ( 5 ; 19 ; 33 ; 3 ) . Details on activation steering are provided in Appendix D . Global steering, as described above, inherently assumes a concept is homogeneously distributed in representation space (Section 2 ), ignoring that high-dimensional representations often exhibit clustered structure ( 24 ) . In LLMs, a single concept may manifest differently depending on context or latent sub-concepts. Consequently, a uniform shift can overlook these nuances, leading to inconsistent control. Contributions. We formulate representation steering as a distribution alignment problem, generalizing first moment-matching approaches ( 5 ; 19 ; 35 ) . Building on this foundation, we propose a novel input-adaptive steering framework where directions vary smoothly across the representation manifold. In particular, recent works, such as 35 and 34 , remark that standard difference-in-means (DiM) steering has a precise optimal transport (OT) interpretation: it is the OT map between the same two Gaussians with different means. This characterization exposes the limitations of DiM steering, as its underlying unimodality and normality assumptions often oversimplify real-world representation distributions. To overcome these limitations, our approach leverages OT to compare and align distributions while modeling both source and target representations as Gaussian mixture models (GMMs), enabling a principled treatment of data heterogeneity. In summary, our contributions are threefold. Figure 1 : PCA (left two) and t-SNE (right two) visualizations of last-token representations for Llama-3.2-3B-Instruct and Qwen2.5-7B-Instruct, colored by k-means clustering. The figures illustrate instances of feasible heterogeneity in concept representations. Appendix F.2 presents textual examples demonstrating that harmful instructions can be coherently grouped via clustering of their last-token hidden representations. 1. We generalize representation steering from restrictive unimodal Gaussian assumptions to multimodal GMMs, and formulate steering via the Mixture Wasserstein distance as a discrete OT problem between semantic clusters (Section 3.1 ). 2. We develop C oncept H eterogeneity- a ware R epresentation S teering (CHaRS), an innovative input-dependent steering method that leverages the cluster-level transport plan where steering directions vary smoothly across the representation manifold, enabling context-sensitive control (Section 3.2 ). 3. We introduce Principal Component Thresholding (PCT) for transport-aligned steering vectors, showing that the induced cluster-wise steering covariance is inherently low-rank, and leverage this structure to obtain a disentangled steering field factorization (Section 3.3 ). We evaluate CHaRS on diverse adversarial and safety tasks, including jailbreaking, toxicity mitigation and image style control, across open-weight LLMs from 3B to 32B parameters. CHaRS consistently outperforms standard activation steering baselines in attack success while maintaining general-language utility. Its spectrally-filtered variant, CHaRS with Principal Component Thresholding (CHaRS-PCT), achieves similar or better performance using fewer steering directions. In sequential steering for toxicity mitigation, CHaRS and CHaRS-PCT outperform prior causal methods, reducing toxic generations without degrading perplexity or downstream performance. Organization. We organize the paper as follows: Section 2 reviews background; Section 3 introduces CHaRS/CHaRS-PCT; Section 4 provides empirical validation; Appendix B discusses related work; and Section 6 concludes. 2 Background 2.1 Optimal Transport Optimal transport (OT) provides a principled framework for comparing and aligning probability distributions ( 28 ; 20 ; 42 ) . In the context of representation steering, \mu and \nu will correspond to empirical distributions of hidden activations under two semantic conditions, such as refusal vs. compliance. Optimal transport provides a principled way to define the “least disruptive” transformation that aligns these distributions. Given two probability measures \mu and \nu on \mathbb{R}^{d} , the p -Wasserstein distance is defined as \displaystyle W_{p}(\mu,\nu)=\left(\inf_{\pi\in\Pi(\mu,\nu)}\int_{\mathbb{R}^{d}\times\mathbb{R}^{d}}\|\mathbf{x}-\mathbf{y}\|^{p}\,d\pi(\mathbf{x},\mathbf{y})\right)^{1/p}, where \Pi(\mu,\nu) denotes the set of all couplings, e.g., joint distributions, with marginals \mu and \nu . For p=2 , this is particularly tractable and geometrically meaningful. The associated OT map T:\mathbb{R}^{d}\to\mathbb{R}^{d} , when it exists, satisfies T_{\#}\mu=\nu and minimizes the transport cost. 2.2 Representation Steering as OT Between Gaussian Distributions In the case of Gaussian distributions, closed-form solutions exist for the 2-Wasserstein distance and the optimal map ( 29 ; 31 ; 14 ) , making OT especially useful in machine learning applications such as domain adaptation and generative modeling ( 36 ; 10 ; 4 ; 9 ) . Wasserstein distance between Gaussians. Let \mu=\mathcal{N}(\mathbf{m}_{1},\mathbf{\Sigma}_{1}) and \nu=\mathcal{N}(\mathbf{m}_{2},\mathbf{\Sigma}_{2}) be two Gaussian measures on \mathbb{R}^{d} , with \mathbf{\Sigma}_{1},\mathbf{\Sigma}_{2}\succ 0 . For the quadratic cost c(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|_{2}^{2} , the squared 2 -Wasserstein distance admits the closed-form expression \displaystyle W_{2}^{2}(\mu,\nu)=\|\mathbf{m}_{1}- \displaystyle\mathbf{m}_{2}\|_{2}^{2}+d_{B}^{2}(\mathbf{\Sigma}_{1},\mathbf{\Sigma}_{2}). (1) The second term corresponds to the squared Bures metric ( 7 ) between covariance matrices, denoted \displaystyle d_{B}^{2}(\mathbf{\Sigma}_{1},\mathbf{\Sigma}_{2})=\operatorname{tr}(\mathbf{\Sigma}_{1})+\operatorname{tr}(\mathbf{\Sigma}_{2})-2\operatorname{tr}\!\left((\mathbf{\Sigma}_{1}^{1/2}\mathbf{\Sigma}_{2}\mathbf{\Sigma}_{1}^{1/2})^{1/2}\right). Optimal transport map. The OT map pushing \mu onto \nu is affine and given by \displaystyle T(\mathbf{x})=\mathbf{m}_{2}+\mathbf{A}(\mathbf{x}-\mathbf{m}_{1}), (2) where \mathbf{A}=\mathbf{\Sigma}_{1}^{1/2}\left(\mathbf{\Sigma}_{1}^{1/2}\mathbf{\Sigma}_{2}\mathbf{\Sigma}_{1}^{1/2}\right)^{-1/2}\mathbf{\Sigma}_{1}^{1/2} . This linear map is the unique positive definite solution of \mathbf{A}\mathbf{\Sigma}_{1}\mathbf{A}^{\top}=\mathbf{\Sigma}_{2} and can be interpreted as optimally aligning the covariance ellipsoids of \mu and \nu while minimizing quadratic transport cost. When \mathbf{\Sigma}_{1}=\mathbf{\Sigma}_{2}=\mathbf{\Sigma} , the covariance term vanishes as d_{B}^{2}(\mathbf{\Sigma},\mathbf{\Sigma})=0 , and the optimal map reduces to a pure translation, \displaystyle T(\mathbf{x})=\mathbf{x}+(\mathbf{m}_{2}-\mathbf{m}_{1}). (3) We provide detailed proofs of the above-mentioned OT maps in Appendix C.2 . Representation steering as OT. Difference-in-means steering in LLMs ( 41 ; 3 ) , where representations are shifted by the difference in activation means between contrasting datasets, e.g., helpful vs. harmful responses, can be interpreted as OT under the assumption of identical covariances. This simplifies steering to a pure translation as in Eqn. ( 3 ), ignoring potential differences in covariance or correlation structure as illustrated in Figure 1 . Furthermore, note that a steering map as in Eqn. ( 2 ) was also observed in a recent work ( 35 ) as a solution to a constrained least-squares optimization. More recently, representation steering was also studied