Adding Alignment Control to Language Models
Wenhong Zhu, Weinan Zhang, Rui Wang · 2025
Read Adding Alignment Control to Language Models by Wenhong Zhu, Weinan Zhang, Rui Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04346.txt.
Introduction
Wenhong Zhu, Weinan Zhang, and Rui Wang add a dial for alignment strength to language models. Their 2025 paper starts from a simple observation: people differ in how strongly they want a model to follow a given alignment style. One fixed setting cannot suit every user or task.
Their method, called CLM, adds a single identity layer before the model's first layers. Only that layer is trained with preference learning, mapping plain input embeddings into an aligned space. At run time, outputs from the aligned and unaligned paths are mixed by a single coefficient. Turning that one knob shifts alignment smoothly, even past the trained point. Results matched full fine-tuning while training far fewer weights.
Catalog record
| CREATOR | Wenhong Zhu, Weinan Zhang, Rui Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04346
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adding Alignment Control to Language Models Wenhong Zhu, Weinan Zhang, Rui Wang Abstract Post-training alignment has increasingly become a crucial factor in enhancing the usability of language models (LMs). However, the strength of alignment varies depending on individual preferences. This paper proposes a method to incorporate alignment control into a single model, referred to as CLM . This approach adds one identity layer preceding the initial layers and performs preference learning only on this layer to map unaligned input token embeddings into the aligned space. Experimental results demonstrate that this efficient fine-tuning method performs comparable to full fine-tuning. During inference, the input embeddings are processed through the aligned and unaligned layers, which are then merged through the interpolation coefficient. By controlling this parameter, the alignment exhibits a clear interpolation and extrapolation phenomenon. 1 Introduction One of the key successes of Large Language Models (LLMs) is their ability to align closely with human values and preferences ( Ouyang et al. 2022 ) . These models typically hinge on a series of critical training phases ( OpenAI 2024 ) . First, they undergo pre-training on vast corpora to master the ability to predict the next token ( Radford et al. 2019 ) . Next, the pre-trained models are fine-tuned through supervised fine-tuning (SFT) to better adapt to specific domains ( Wei et al. 2021 ; Yang et al. 2024b ) . Finally, preference-based optimization methods are employed, such as Reinforcement Learning from Human Feedback (RLHF) ( Christiano et al. 2017 ) and Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) . These approaches ensure the model avoids engaging in factual inaccuracies, exhibiting biases, and displaying other undesirable behaviors ( Bai et al. 2022 ) . Across these alignment approaches, the core objective remains consistent: to maximize the expected reward from an implicit or explicit reward function while incorporating the KL-divergence from the reference policy as a regularization term ( Schulman et al. 2017 ; Gao et al. 2023 ; Rafailov et al. 2024 ) . The strength of this regularization plays a pivotal role in determining the alignment outcome ( Ziegler et al. 2019 ) , as excessive regularization can overly constrain alignment, whereas insufficient regularization may lead to reward hacking ( Pan et al. 2022 ) . Determining the optimal alignment strength often requires a process of trial and error ( Meng et al. 2024 ) . Moreover, the optimal alignment strength can vary based on cultural, ethical, and individual perspectives ( Anwar et al. 2024 ) . This variability makes it challenging to devise a universal approach for embedding ideal alignment into an LLM, as the definition of “optimal” may differ depending on who determines it. Recent studies have conceptualized RLHF/DPO models as interpolated outcomes between an initial SFT model and a hypothetically better-aligned model ( Liu et al. 2024 ; Zheng et al. 2024a ; Zhu et al. 2024 ) . By introducing an interpolation coefficient, these works enable both the interpolation and extrapolation of alignment strength. This raises an intriguing question: could this interpolation coefficient be integrated into a single model and exposed as a user-controllable interface—similar to a temperature parameter—allowing users to customize the alignment strength dynamically? Figure 1: Inference architecture: The input embedding is fed simultaneously into the newly added aligned layer and the original first layer of the LM. The hidden states from both paths are propagated through all layers and merged after the LM head layer. In this work, we first adopt a distillation method to transfer extrapolated and interpolated alignment into the aligned model, thereby consolidating these observations (see Sec 3 ). While this approach improves inference efficiency, it comes at the cost of training efficiency. Our analysis reveals that the bottom layers of the LM play a crucial role in preference learning (see Sec 6.2 ). To construct an alignment-controllable language model ( CLM ), we duplicate the last layer and insert it as an identity layer preceding the original ones (see Sec 4.1 ). Preference learning is then confined to this newly added layer while all other layers remain frozen (see Sec 4.2 ). This design effectively maps unaligned input token embeddings into aligned spaces, ensuring the desired alignment effect (see Sec 6.3 ). As shown in Figure 1 , the input embedding is processed through the aligned and unaligned layers during inference, with the output logits merged using an interpolation coefficient \lambda . This mechanism enables precise alignment control. In summary, our contributions are as follows: • We found that performing alignment in the bottom layers is more effective than in the upper layers, a result that has been scarcely explored in previous studies. • We introduce CLM , which adds an identity layer preceding the original layers of LM and only performs preference learning on this layer. It is an efficient training method that achieves performance comparable to full fine-tuning. • To our knowledge, CLM is the first to incorporate inference-time realignment considerations into the training process explicitly. The alignment strength manifests clear extrapolation and interpolation by appropriately controlling the interpolation coefficient \lambda . 2 Preliminary 2.1 Transformer Decoder Layer The current mainstream LMs based on transformer architecture typically have multiple decoder layers (\phi_{0},\phi_{1},...,\phi_{L}) ( OpenAI 2024 ) . Each layer consists of an attention component and an MLP component, as shown in Figure 2 . Given an input h_{t-1} , each layer produces the output h_{t} as described by the following equations: \begin{array}[]{r}h_{t-1}^{\prime}=h_{t-1}+\operatorname{Attention}(\operatorname{RMSNorm}(h_{t-1}))\\ h_{t}=h_{t-1}^{\prime}+\operatorname{MLP}\left(\operatorname{RMSNorm}\left(h_{t-1}^{\prime}\right)\right).\end{array} (1) Figure 2: Overview of attention and MLP component. Identity copy makes the last projector of each component with weight and bias to zero. Both components have a projector to ensure that the input and output dimensions of the module are consistent, which facilitates the combination with residual connection. Identity copy The identity copy is defined as \phi_{\text{id}}(h_{t-1})=h_{t-1} which means the input and output are identical. This can be achieved as long as Attention(RMSNorm( h_{t-1} )) = 0 and MLP(RMSNorm( h_{t-1}^{\prime} )) = 0. Then, the input is directly the result of the output due to the residual. We initialize the W_{\text{out}} and W_{\text{down}} weight matrices as marked in dark purple in Figure 2 in the identity layer to zero. As a result, the entire layer is reduced to an identity layer at initialization, preserving the output from the initial model. 2.2 Fine-tuning from Human Feedback Autoregressive LM Given a query sequence x:=\left(x_{1},\ldots,x_{m}\right)\in\mathcal{X} , an auto-regressive LM defines a probability distribution over possible response sequences y:=\left(y_{1},y_{2},\ldots,y_{n}\right)\in\mathcal{Y} . The probability \pi_{\theta}(y\mid x) can be decomposed using the chain rule of probability as \pi_{\theta}(y\mid x)=\prod_{t=1}^{n}\pi_{\theta}\left(y_{t}\mid y_{<t},x\right) , where y_{<t} denotes \{y_{1},y_{2},...,y_{t-1}\} . Given a pre-trained and typically SFT reference model \pi^{\text{ref}}(y\mid x) , we need to use human-labeled datasets reflecting human preferences further for aligning LMs with human preferences ( Wei et al. 2021 ) . The alignment approaches such as PPO ( Schulman et al. 2017 ) and DPO ( Rafailov et al. 2024 ) , whose optimization objective is generally to maximize the expected reward r(x,y) from an implicit or explicit reward function while including a KL-divergence term from the reference policy as a penalty for divergence. The main objective is as follows: \displaystyle\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{X},y\sim\pi_{\theta}(y\mid x)}\left[r(x,y)-\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi^{\mathrm{ref}}(y\mid x)}\right], (2) where \beta is a