White-Box Sensitivity Auditing with Steering Vectors

Hannah Cyberey [email protected], Yangfeng Ji [email protected], David Evans [email protected] · 2026

Read White-Box Sensitivity Auditing with Steering Vectors by Hannah Cyberey [email protected], Yangfeng Ji [email protected], David Evans [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16398.txt.

Introduction

Hannah Cyberey and Yangfeng Ji audit models from in with steer paths. Their 2026 preprint deals with checks of large language models. Most checks now test just inputs and outputs of a model. Such black box tests can miss flaws hid in the build. White box checks read the deep acts of the model as it works.

Steer paths shift deep acts in set ways for the check. Shifts show how touchy a pick is to deep traits. Touch maps help rule teams set fair test terms. They also show where a flaw lives in the build. The work gives a new audit tool with deep reach.

Catalog record

CREATORHannah Cyberey [email protected], Yangfeng Ji [email protected], David Evans [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

White-Box Sensitivity Auditing with Steering Vectors Hannah Cyberey [email protected], Yangfeng Ji [email protected], David Evans [email protected] Abstract Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input--output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model’s intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently reveals substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. 1 1 1 Our code is openly available at https://github.com/hannahxchen/llm-steering-audit 1 Introduction As large language models (LLMs) are increasingly used in high-stakes applications, auditing has become crucial for ensuring their suitability and trustworthiness. Audits are structured evaluations devised to identify problematic behaviors in a system that could negatively impact stakeholders and assess whether it meets specific standards ( 4 ) . Previous work has proposed methods for auditing LLMs across various dimensions, including bias ( 70 ; 28 ) , privacy ( 7 ; 51 ) , robustness ( 60 ; 78 ) , and safety ( 77 ) . Most current and proposed auditing methods operate in a black-box setting where auditors only interact with the system by submitting inputs and observing the corresponding outputs. While these methods provide a simple and straightforward way to evaluate models, they are limited to surface-level failure modes that can be identified through tests constructed in the input space ( 5 ) . They also depend heavily on the design and construction of testing data, making them prone to inconsistent evaluation results due to prompt sensitivity in LLMs ( 66 ; 47 ; 32 ) . Moreover, many model properties (e.g., bias) we wish to measure involve abstract concepts (e.g., gender and race) that are difficult to encapsulate or manipulate precisely solely through text-based inputs. Recent work by 5 makes a strong case for the need for audits with white-box access, which grants auditors full access to the model’s internals, including weights, activations, and gradients. Such access enables more comprehensive evaluations over a broader search space and a better understanding of the model’s internal mechanisms underlying potentially undesirable behavior. While 5 outlines many promising directions for white-box audits, they do not provide a concrete technical approach for conducting such audits in practice. Meanwhile, advances in representation engineering ( 79 ) have shown high-level concepts encoded within LLM internals and can be directly manipulated using techniques such as activation steering to control model behavior ( 74 ; 3 ; 16 ) . Yet, opportunities for using such internal manipulation in model audits have not yet been well explored. Contributions. We introduce a concrete auditing method that evaluates model behavior through targeted interventions on model internals ( Section 3.2 ). Building on recent work on activation steering, we develop a novel evaluation method that applies steering vectors to manipulate latent concepts within model internals and assess model behavior using a sensitivity metric ( Section 3.3 ). By analyzing changes in model predictions, we enable audits that probe a model’s dependence on specific concepts, particularly in settings where these concepts are difficult to disentangle with input-based testing alone. We demonstrate how our method can be applied to conduct bias audits in decision-making contexts ( Section 4 ). We construct four decision tasks simulating the use of LLMs in high-stakes settings ( Section 4.2 ), including judicial trials, credit scoring, college admissions, and medical diagnosis. Compared with traditional black-box evaluations that rely on input–output tests, our white-box method often reveals a substantial degree of bias in model predictions, even in cases where the black-box method appears to show little bias ( Section 4.4 ), and also yields more robust evaluation results ( Section 4.5 ). We further assess the audit validity ( Section 5 ), showing that our white-box results reflect actual bias risks that the black-box baseline fails to detect ( Section 5.1 ), as demonstrated by a different black-box perturbation strategy. In addition, our method has little impact on other task-relevant variables and better isolates the target concept than the black-box method ( Section 5.2 ). 2 Background This section provides background on black-box and white-box evaluation methods and activation steering. 2.1 Black-Box Evaluation Let the target model be f:\mathcal{X}\rightarrow\mathcal{Y} that takes input x\in\mathcal{X} and makes predictions f(x) . The standard black-box evaluation measures model performance on a task using a test set \mathcal{D} and a task-specific evaluation metric \mathcal{M}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R} as follows ( 30 ; 42 ) : \Phi_{f,\mathcal{D}}=\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}}\mathcal{M}(f(x),y) where y\in\mathcal{Y} is the ground-truth label for test input x . The test set \mathcal{D} is typically derived from existing datasets, sometimes with additional test instances generated through perturbations. Proposed perturbation methods can be categorized based on whether the perturbation preserves the ground truth label of the original input ( 73 ; 9 ) . Most text perturbation methods are label-preserving and make task-irrelevant changes, such as paraphrasing ( 34 ; 19 ) and formatting changes ( 29 ) . Suppose input x has a corresponding label y and a set of valid perturbed inputs \mathcal{P}(x) . The model is expected to predict f(x^{\prime})=y for all x^{\prime}\in\mathcal{P}(x) . Previous work has also explored label-changing perturbations, which explicitly alter the ground truth label of an input by making small but meaningful changes, such as negation ( 50 ; 61 ) . In this scenario, if the perturbed input x^{\prime} has label y^{\prime} , where y^{\prime}\not=y , the model prediction should be f(x^{\prime})=y^{\prime} and f(x^{\prime})\not=y . Since determining the new label y^{\prime} depends on the specific task, it is more challenging to generate such perturbations automatically and often requires manual effort ( 22 ; 39 ) . In the context of bias evaluation, label-preserving perturbations are applied but only to the protected group attribute (e.g., gender, race) of the input. For text inputs, a perturbation set \mathcal{P}_{G} is required for each group G\in\mathcal{G} , usually constructed from a predefined list of tokens, words, or sequences that are assumed to be representative of group G ( 54 ; 23 ) . The degree of bias is commonly assessed by the average group disparities in prediction outcomes based on established fairness principles ( 17 ) . Given two groups A,B\in\mathcal{G} , the bias score of a model can be formulated as: \displaystyle\Delta(A,B)=\Phi_{f,\mathcal{D}_{A}}-\Phi_{f,\mathcal{D}_{B}}\qquad\text{where}\qquad\mathcal{D}_{G}=\{(x^{\prime},y):x^{\prime}\in\mathcal{P}_{G}(x),(x,y)\in\mathcal{D}\} (1) where \mathcal{P}_{G}(x) represents the set of perturbed inputs for x with group attribute set to G\in\{A,B\} . In Section 4 , we apply this formulation to construct the black-box method, which serves as our baseline. 2.2 Black-Box vs. White-Box Evaluations The traditional black-box method applies perturbations in the input space to create test cases, which works well for variables with concrete bounds and values. For instance,

Read the complete text