Steering Language Models Before They Speak: Logit-Level Interventions

Hyeseon An, Shinwoo Park, Hyundong Jin, Yo-Sub Han · 2026

Read Steering Language Models Before They Speak: Logit-Level Interventions by Hyeseon An, Shinwoo Park, Hyundong Jin, Yo-Sub Han (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10960.txt.

Introduction

Hyeseon An, Shinwoo Park, Hyundong Jin, and Yo-Sub Han steer language models at the logit level, before any words come out. Steering matters for style-sensitive rewriting, user-adaptive replies, and cutting toxic output. Prompting can steer, but its grip is coarse and uneven. Activation methods reach deeper, yet need access to inner layers.

Logits are the model's scores for each next token, the last step before speech. Intervening there promises fine control without deep surgery. The authors develop this middle path and test its reach. Their work serves product teams that must shape model tone tightly while working with models they cannot fully open.

Catalog record

CREATORHyeseon An, Shinwoo Park, Hyundong Jin, Yo-Sub Han
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Steering Language Models Before They Speak: Logit-Level Interventions Hyeseon An, Shinwoo Park, Hyundong Jin, Yo-Sub Han Abstract Steering LLMs is essential for specialized applications such as style-sensitive text rewriting, user-adaptive communication, and toxicity mitigation. Current steering methods, such as prompting-based and activation-based approaches, are widely used to guide model behavior. However, activation-based techniques require deep access to internal layers, while prompting-based steering often fails to provide consistent or fine-grained control. In order to address these limitations, we propose a training-free inference-time logit intervention for controllable generation. Our approach utilizes a statistical token score table derived from z z -normalized log-odds of labeled corpora to shift the decoding distribution. Empirical evaluations across three diverse datasets focusing on writing complexity, formality, and toxicity demonstrate that our method effectively steers output characteristics, confirming its broad applicability and task-agnostic nature. Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains: up to + + 47%p accuracy and 50 × F 1 \times~F_{1} improvement. 1 Introduction Large language models (LLMs) have become core components in many real-world applications, including customer service, technical documentation, and content generation. Despite their strong generative capabilities, however, the sampling-based nature of LLMs makes it difficult to control fine-grained output characteristics, such as writing complexity, politeness, and toxicity, in a consistent and reliable manner. This limitation presents a challenge for deploying LLMs in settings that demand safety, trustworthiness, and reproducibility. Accordingly, controllable text generation has emerged as an important research direction aimed at bridging the gap between general-purpose generation and concrete user- or system-level constraints. As models scale and their internal architectures become increasingly opaque, the need for efficient steering mechanisms that can induce target output characteristics without degrading output quality has become more urgent. Existing approaches address this challenge from several perspectives, each with distinct limitations. Prompt-based control is simple to implement but often exhibits high variance and limited control over output characteristics strength. In order to mitigate this issue, a line of work steers frozen language models at decoding time using auxiliary signals, including gradient-based perturbation ( 2 ) , discriminator-guided reweighting ( 8 ) , future-conditioned discriminators ( 24 ) , and expert/anti-expert ensembling in logit space ( 10 ) . More recent approaches explore pragmatic control ( 22 ) or prompt-based handling of multiple constraints, such as length control ( 6 ) . While effective in specific settings, these methods are often sensitive to auxiliary model quality and decoding hyperparameters, making consistent control difficult to guarantee. Stronger controllability has also been pursued through training-based approaches that amortize control signals into model parameters, including control codes ( 7 ) , prefix tuning ( 9 ) , instruction tuning ( 25 ) , auxiliary token-level guidance ( 12 ) , reinforcement learning with dynamic reward shaping ( 3 ) , and verifiable format-specific training signals ( 23 ) . Although these approaches offer strong and stable control, they incur additional costs in training data and computational resources. A separate line of work explores activation-based steering, which modifies internal hidden states at inference time without additional training. However, these methods require architecture-specific layer selection and risk degrading linguistic fluency or reasoning by directly perturbing internal representations. Taken together, existing approaches leave an important gap: lightweight output characteristics control methods that are training-free , architecture-agnostic , and interpretable remain underexplored. In this work, we address this gap by proposing a logit-level inference-time intervention that leverages token-level corpus statistics to adjust the output distribution during decoding. Our approach operates exclusively in the final logit space shared by all autoregressive language models, enabling architecture-agnostic application without modifying internal computations. Because the control signal is derived from corpus statistics, the resulting behavior is transparent and reproducible. We refer to this approach as SWAI ( S tatistical W riting style A ligned I nference). SWAI precomputes token-level z -scores capturing output characteristics associations and applies a controlled logit bias within a contextually plausible candidate set during decoding. Experiments on OSE , WikiPol , and RealTox demonstrate substantial improvements over prompt-based baselines across consistency, reliability, and safety-related metrics, without requiring additional training. Our contributions are summarized as follows: • We introduce an architecture-agnostic, logit-level steering framework for achieving consistent control over output characteristics without layer-specific tuning. • We propose a non-invasive inference-time intervention that preserves fluency and reasoning by steering final output choices rather than internal hidden states. • We provide a transparent and interpretable control mechanism based on statistically grounded token-level z -scores. 2 Related Work 2.1 Decoding-Time and Prompt-Adjacent Model Steering A dominant approach to controllable generation steers a fixed base language model at inference time by modifying token probabilities using auxiliary signals. PPLM ( 2 ) performs gradient-based perturbations to bias generation toward desired output characteristics, while GeDi ( 8 ) integrates a discriminator to reweight candidate continuations. FUDGE ( 24 ) trains a future discriminator that predicts whether a partial prefix will satisfy a global constraint, enabling online guidance during decoding. DExperts ( 10 ) combines an expert and an anti-expert language model to achieve controllable generation via logit-level ensembling. More recent work explores pragmatic and structural control with minimal overhead: RSA-Control ( 22 ) grounds lightweight control in pragmatic reasoning to adjust style/readability, and prompt-based multi-control frameworks extend prompting to simultaneously handle length and other control types ( 6 ) . While effective, these methods can be sensitive to prompt phrasing, decoding hyperparameters, and auxiliary-model quality. 2.2 Training-Based Steering and Parameter-Efficient Control An alternative line of work bakes controllability into model parameters through additional training. CTRL ( 7 ) learns control codes during pretraining to condition generation on high-level output characteristics. Prefix-Tuning ( 9 ) provides a parameter-efficient mechanism by optimizing continuous prefixes while keeping the backbone frozen. INSTRUCTCTG ( 25 ) instruction-tunes models for controlled generation by synthesizing weakly supervised constraint–text pairs and verbalizing constraints as natural-language instructions. Beyond instruction tuning, NADO ( 12 ) decomposes sequence-level oracle feedback into token-level guidance via an auxiliary model, and dynamic multi-objective methods adapt reward weights to improve multi-style controllability ( 3 ) . Recent work also targets verification-oriented constraints, enforcing outputs that satisfy externally checkable format requirements ( 23 ) . These approaches often improve robustness and generalization but typically require extra training data and compute, and may reduce transparency regarding how control is achieved at inference time. Figure 1: Overview of SWAI . It first derives token-level statistical scores from labeled corpora via smoothed one-vs-rest log-odds with variance normalization. During decoding, intervention is confined to a top- K candidate set, where a fixed logit offset is applied to the highest-scoring tokens before sampling from the modified distribution. 2.3 Activation-based Steering Training-free controllability methods steer LLM behavior by intervening on intermediate activations at inference time, typically via an additive direction or a lightweight transformation applied to hidden states. 20 ; 15 derive steering vectors from contrastive pairs (e.g., prompt pairs or positive/negative behavioral demonstrations) and inject them into the residual stream, sometimes across multiple layers, to reliably shift style, preferences, or behaviors. Concept-oriented variants treat target output characteristics as low-dimensional directions learned from activation statistics rather than demonstrations. 26 extract interpretable concept vectors (e.g., honesty, morality) from activation space and inject them to modulate generations. A

Read the complete text