Flexible Realignment of Language Models

Wenhong Zhu, Ruobing Xie, Weinan Zhang, Rui Wang · 2025

Read Flexible Realignment of Language Models by Wenhong Zhu, Ruobing Xie, Weinan Zhang, Rui Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12704.txt.

Introduction

When a language model falls short, retraining it from scratch wastes time and money. Wenhong Zhu and co-authors present a flexible fix in work from twenty twenty-five, one that lets users dial the amount of change up or down. The fix works both while the model trains and while it runs.

During training, outputs from the old model and a tuned model are blended in set shares, which cuts token use while holding quality. During use, a small added layer blends the two paths again, so one model can think fast or slow on request. One upgraded model even beat its old scores when pushed to reason more deeply. The approach suits teams that need control without a full retrain.

Catalog record

CREATORWenhong Zhu, Ruobing Xie, Weinan Zhang, Rui Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12704

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flexible Realignment of Language Models Wenhong Zhu, Ruobing Xie, Weinan Zhang, Rui Wang Abstract Realignment becomes necessary when a language model (LM) fails to meet expected performance. We propose a flexible realignment framework that supports quantitative control of alignment degree during training and inference. This framework incorporates Training-time Realignment (TrRa) , which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by 54.63% on DeepSeek-R1-Distill-Qwen-1.5B without any performance degradation, outperforming DeepScaleR-1.5B’s 33.86% . To complement TrRa during inference, we introduce a layer adapter that enables smooth Inference-time Realignment (InRa) . This adapter is initialized to perform an identity transformation at the bottom layer and is inserted preceding the original layers. During inference, input embeddings are simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit level. We upgraded DeepSeek-R1-Distill-Qwen-7B from a slow-thinking model to one that supports both fast and slow thinking, allowing flexible alignment control even during inference . By encouraging deeper reasoning, it even surpassed its original performance. 1 Introduction Figure 1 : Our InRa : The inputs are fed simultaneously into the layer adapter and the original bottom layer of the LM. The hidden states from both paths are propagated through all layers and merged at the logit level. The layer adapter enables flexible realignment even during inference. Current large language models (LLMs), such as GPT-4o [ 1 ] , and reasoning-focused models like OpenAI-o1 [ 2 ] and DeepSeek-R1 [ 3 ] , have achieved remarkable success. These models typically hinge on a series of critical training phases [ 4 ] . First, they undergo pre-training on vast corpora to master the ability to predict the next token [ 5 ] . Next, the pre-trained models are fine-tuned through supervised fine-tuning (SFT) as a cold start to better adapt to specific domains [ 6 , 7 ] . Reinforcement Learning (RL) has emerged as an essential component of the whole training pipeline. In the RL phase, the core objective is to maximize the expected reward while incorporating the KL-divergence from the reference policy [ 8 , 9 , 10 ] . The reward signal is key to alignment : correctness-based rewards improve reasoning ability [ 2 , 3 ] , while 3H-based (honesty, harmlessness, and helpfulness) rewards reflect human values [ 11 ] . However, misalignment can still emerge due to imperfect rewards or evolving user needs. Review the pain points of the existing product models: The most advanced reasoning models tend to suffer from the overthinking problem [ 12 , 3 ] , leading to increased computational costs. How can we realign these models for efficient reasoning to ensure user affordability? Meanwhile, to adapt to individual user preferences, conversational models often become overly sycophantic [ 13 ] . How can we realign them to balance personalization and objective responses better? Realignment is thus essential to correct model behavior and ensure robustness over time. One typical practical approach to realignment is to retrain the model under the same reward signal while exploring different hyperparameters. However, for models trained via RL, this process is often resource-intensive. For instance, simply replicating the DeepSeek-R1 experiments (with context lengths exceeding 32K over 8000 training steps) using a 1.5B-parameter model requires at least 70,000 A100 GPU hours [ 14 ] . We need to address this challenge through a more efficient method without compromising performance. Additionally, we seek a solution that offers flexibility during training and inference. We propose a flexible realignment framework that facilitates training-time and inference-time realignment, controlling the alignment degree to satisfy different demands. (1) We draw inspiration from knowledge distillation for training-time realignment (TrRa) . Specifically, we realign the reference model using a teacher signal constructed from a controllable fusion of the output logits of the reference and the already aligned models. (2) We introduce a layer adapter to endow the LM inference-time realignment (InRa) . This is inspired by the fact that the lower layers of the LM are more influential than the upper layers during fine-tuning (see Sec. 5.1 ). Based on this observation, we duplicate the bottom layer and insert it as an identity mapping layer before the original layers (see Sec. 3.2 ). Fine-tuning is restricted solely to this layer adapter. As illustrated in Figure 1 , the input embeddings are processed through this adapter and original layers during inference, and the resulting output logits are combined via an interpolation coefficient \lambda . This design retains both paths of logits within a single model, enabling smooth and flexible control over alignment. In summary, our main contributions are as follows: • We propose new post-training methods, TrRa and TrRa-iter, that use a controllable teacher signal created by combining logits from reference and aligned models, overcoming the fixed teacher in traditional knowledge distillation to enable flexible training-time realignment. • We propose a parameter-efficient fine-tuning approach called the layer adapter. It allows smooth and efficient realignment during inference within a single model. • Experiments demonstrate that our flexible realignment framework enables efficient and flexible realignment during both training and inference. For example, TrRa-iter reduces token usage by 54.63% on DeepSeek-R1-Distill-Qwen-1.5B without any loss in performance. Additionally, InRa has been successfully tested in practical scenarios like combining fast and slow thinking models and flexibly realigning with 3H values. 2 Preliminary Autoregressive LM Given a query sequence x:=\left(x_{1},\ldots,x_{m}\right)\in\mathcal{X} , an auto-regressive LM defines a probability distribution over possible response sequences y:=\left(y_{1},y_{2},\ldots,y_{n}\right)\in\mathcal{Y} . The probability \pi_{\theta}(y\mid x) can be decomposed using the chain rule of probability as \pi_{\theta}(y\mid x)=\prod_{t=1}^{n}\pi_{\theta}\left(y_{t}\mid y_{<t},x\right) , where y_{<t} denotes \{y_{1},y_{2},...,y_{t-1}\} . Transformer Decoder Layer The current mainstream LMs based on transformer architecture [ 15 ] typically have multiple decoder layers (\phi_{0},\phi_{1},...,\phi_{L}) [ 4 ] . Each layer consists of an attention component and an MLP component. Given an input h_{t-1} , the layer computes the output h_{t} through the following steps: h_{t-1}^{\prime}=h_{t-1}+\operatorname{Attention}(\operatorname{RMSNorm}(h_{t-1})) and h_{t}=h_{t-1}^{\prime}+\operatorname{MLP}\left(\operatorname{RMSNorm}\left(h_{t-1}^{\prime}\right)\right) . Both components have a projector to ensure the module’s input and output dimensions are consistent, facilitating the combination with a residual connection [ 16 ] . Reward-based Fine-tuning Given a pre-trained (Base) and typically SFT reference model \pi^{\text{ref}}(y\mid x) , RL is a commonly used post-training technique to enhance model capabilities further. The optimization objective maximizes the expected reward r(x,y) while including a KL-divergence term from the reference policy as a penalty. The objective is as follows: \displaystyle\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{X},y\sim\pi_{\theta}(y\mid x)}\left[r(x,y)-\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi^{\mathrm{ref}}(y\mid x)}\right], (1) where \beta is a regularization parameter. It has a closed-form solution for the aligned model , given as follows: \pi^{*}_{\theta}(\beta)(y\mid x)=\frac{\pi^{\mathrm{ref}}(y\mid x)\exp\left[\frac{1}{\beta}r(x,y)\right]}{\sum_{y^{\prime}}\pi^{\mathrm{ref}}\left(y^{\prime}\mid x\right)\exp\left[\frac{1}{\beta}r\left(x,y^{\prime}\right)\right]}. (2) Typically, we can transform the above equation by representing r(x,y) as \frac{1}{\beta}r(x,y)=\log\frac{\pi^{*}_{\theta}(\beta)(y\mid x)}{\pi^{\mathrm{ref}}(y\mid x)}+\log Z(x), (3) where Z(x):=\sum_{y^{\prime}}\pi^{\mathrm{ref}}\left(y^{\prime}\mid x\right)\exp\left(\frac{1}{\beta}r\left(x,y^{\prime}\right)\right) is the partition function. Realignment Realignment becomes necessary when the LM fails to meet expected performance. As shown in Equation 1 , the KL regularization parameter \beta determines how far the policy model \pi_{\theta}(y\mid x) deviating from its initial state \pi^{\text{ref}}(y\mid x) [ 17 ] . To adjust the alignment strength of the LM, one can modify the value of \beta , which can be achieved by scaling it with a factor \lambda during training. This

Read the complete text