Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
Yi Liu, Dianqing Liu, Mingye Zhu, Junbo Guo, Yongdong Zhang, Zhendong Mao · 2025
Read Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models by Yi Liu, Dianqing Liu, Mingye Zhu, Junbo Guo, Yongdong Zhang, Zhendong Mao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19700.txt.
Introduction
Yi Liu and co-authors present, in a 2025 paper, a way to peel the alignment training off a large language model and keep it as a separate plug-in part. Alignment, the tuning that makes models follow instructions and norms, normally requires retraining the whole model for each new need.
Their residual alignment model recasts alignment as a kind of importance sampling. The plain model proposes text, and a small alignment module reweights its choices at the sequence level. Because the module trains on its own, it can be attached or swapped without touching the base model. A resampling decoder also trims the delay of the first token, a known flaw of similar designs.
Catalog record
| CREATOR | Yi Liu, Dianqing Liu, Mingye Zhu, Junbo Guo, Yongdong Zhang, Zhendong Mao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models Yi Liu, Dianqing Liu, Mingye Zhu, Junbo Guo, Yongdong Zhang, Zhendong Mao Abstract The widespread adoption of large language models (LLMs) across industries has increased the demand for high-quality and customizable outputs. However, traditional alignment methods often require retraining large pretrained models, making it difficult to quickly adapt and optimize LLMs for diverse applications. To address this limitation, we propose a novel Residual Alignment Model ( RAM ) that formalizes the alignment process as a type of importance sampling. In this framework, the unaligned upstream model serves as the proposal distribution, while the alignment process is framed as secondary sampling based on an autoregressive alignment module that acts as an estimator of the importance weights. This design enables a natural detachment of the alignment module from the target aligned model, improving flexibility and scalability. Based on this model, we derive an efficient sequence-level training strategy for the alignment module, which operates independently of the proposal module. Additionally, we develop a resampling algorithm with iterative token-level decoding to address the common first-token latency issue in comparable methods. Experimental evaluations on two leading open-source LLMs across diverse tasks, including instruction following, domain adaptation, and preference optimization, demonstrate that our approach consistently outperforms baseline models. 1 Introduction In recent years, the rapid advancement of large language models (LLMs) has led to their widespread adoption across various industries [ 6 , 35 , 1 ] . Efforts to align LLM outputs with domain requirements and human values enhance their utility and content safety [ 30 , 11 , 24 , 29 , 2 , 17 , 45 , 23 ] . Techniques such as supervised learning [ 37 , 42 , 43 ] , preference optimization [ 30 , 24 , 11 ] and reinforcement learning [ 46 , 33 , 44 , 41 ] are crucial for achieving model alignment. According to the scaling laws of LLMs [ 19 ] , increasing model size typically enhances their performance. However, research indicates that effective domain adaptation and value alignment can be achieved even with smaller models [ 10 ] . This difference in size requirements necessitates a balance between utility performance and alignment flexibility with respect to model size [ 38 , 31 ] . Moreover, training large models for specific domains is resource-intensive and requires the deployment of separate models, increasing resource costs and hindering traffic sharing across domains. Thus, there is an urgent need for more efficient and economical model solutions. Recent works [ 18 , 26 , 7 ] have introduced methods that fine-tune an adapter module on preference datasets to learn correctional residuals between preferred and non-preferred responses, or supervised and synthetic examples, and then stacked onto the upstream model to achieve corrected alignment. While these approaches effectively decouple alignment from LLMs during training, the correction based on the complete upstream response introduces significant latency for the first token during the inference phase, particularly for long content generation. Additionally, the Aligner model P({\bm{y}}|{\bm{y}}^{\prime},{\bm{x}}) [ 18 ] introduces a reference response {\bm{y}}^{\prime} for correction, which carries the extra potential risk of out-of-distribution (OOD) inputs, not only for the original question {\bm{x}} , but also for the reference {\bm{y}}^{\prime} , as further discussed in Section 4 . In this paper, we present a novel Residual Alignment Model ( RAM ) that formalizes residual correction for alignment as a type of importance sampling, which conditioned directly on {\bm{x}} to generate {\bm{y}} . In this framework, the unaligned upstream model is referred to as the Proposal Module , serves as the proposal distribution, while the alignment process is framed as secondary sampling based on an autoregressive alignment module which acts as an estimator of the importance weights and is termed the Residual Aligner . This linear combination of the Proposal Module and the Residual Aligner allows for the natural detachment of the alignment module from the target aligned model, illustrated as Equation 1 : P_{\mathrm{Aligned}}({\bm{y}}|{\bm{x}})\propto P_{\mathrm{ProposalModule}}({\bm{y}}|{\bm{x}})*P_{\mathrm{ResidualAligner}}({\bm{y}}|{\bm{x}}) (1) Building upon this framework, we propose an efficient training strategy that operates on the detached alignment module at the sentence level. The Proposal Module is required solely for one-off data synthesis in pointwise supervised datasets to create preference examples; in contrast, it remains unused throughout the entire training process for preference datasets. Furthermore, we develop a token-level decoding algorithm with minimal first-word latency to ensure practicality during inference. The training and decoding strategy is illustrated on Figure 1 . Figure 1: An illustration of alignment training and inference within the RAM framework. During training, the large unaligned Proposal Module remains frozen, while only the smaller Residual Aligner undergoes alignment tuning. In the inference phase, the Proposal Module generates context-aware candidate tokens, which the Residual Aligner aligns and reduces to a target token. This target token is then transmitted out and simultaneously sent back to the Proposal Module to initiate the next step. By linearly decomposing the target aligned model into a Proposal Module and a Residual Aligner , we can independently scale and optimize each component with targeted data and resource allocation. Furthermore, multiple alignment modules can share the Proposal Module , facilitating efficient cross-domain resource utilization and enhancing the overall system’s efficiency and scalability. The experimental results presented in Section 4 demonstrate that a robust large model paired with a smaller Residual Aligner , achieving an efficient domain alignment at a reduced cost. 2 Residual Alignment Model 2.1 Preliminary Consider a general dataset denoted as {\mathcal{D}}=\{({\bm{x}},{\bm{y}})\} , where {\bm{x}}=\{x_{1},...,x_{m}\} represents an input prompt in the form of a token sequence, and {\bm{y}}=\{y_{1},...,y_{n}\} corresponds to the completion. We define {\mathcal{S}} as a biased subset of {\mathcal{D}} . The conditional distributions for these datasets are denoted as P_{\mathcal{D}}({\bm{y}}|{\bm{x}}) and P_{\mathcal{S}}({\bm{y}}|{\bm{x}}) respectively, where \exists({\bm{x}},{\bm{y}})\in{\mathcal{S}},P_{\mathcal{S}}({\bm{y}}|{\bm{x}})\neq P_{\mathcal{D}}({\bm{y}}|{\bm{x}}) . Suppose we have a large language model P_{\mathrm{M}}({\bm{y}}|{\bm{x}}) pretrained on the dataset {\mathcal{D}} to estimate P_{\mathcal{D}}({\bm{y}}|{\bm{x}}) . The goal of alignment is to utilize instances from the biased subset {\mathcal{S}} to adapt the model P_{\mathrm{M}}({\bm{y}}|{\bm{x}}) , aiming to make it a better estimator of P_{\mathcal{S}}({\bm{y}}|{\bm{x}}) . Importance sampling estimates properties of a target distribution using samples from a different distribution, which is useful when direct sampling is difficult. The method involves reweighting the samples to account for the differences between distributions: \mathbb{E}_{{\bm{x}}\sim Q}[f({\bm{x}})]=\mathbb{E}_{{\bm{x}}\sim P}[f({\bm{x}})\frac{Q({\bm{x}})}{P({\bm{x}})}] (2) where Q is the target distribution, P is the proposal distribution, and \frac{Q({\bm{x}})}{P({\bm{x}})} is the importance weight. 2.2 Detaching the Alignment Module Since the dataset {\mathcal{S}} is a subset of {\mathcal{D}} , we can reasonably assume that the distribution P_{\mathcal{D}}({\bm{y}}|{\bm{x}}) or its estimator P_{\mathrm{M}}({\bm{y}}|{\bm{x}}) does not differ significantly from P_{\mathcal{S}}({\bm{y}}|{\bm{x}}) . This assumption supports the use of importance sampling to model the alignment task of LLMs. Suppose the aligned probability P_{\mathcal{S}}({\bm{y}}|{\bm{x}}) is supported by P_{\mathrm{M}}({\bm{y}}|{\bm{x}}) . With importance sampling, we express P_{\mathcal{S}}({\bm{y}}|{\bm{x}})=P_{\mathrm{M}}({\bm{y}}|{\bm{x}})\frac{P_{\mathcal{S}}({\bm{y}}|{\bm{x}})}{P_{\mathrm{M}}({\bm{y}}|{\bm{x}})} . The importance weight W({\bm{y}}|{\bm{x}})=\frac{P_{\mathcal{S}}({\bm{y}}|{\bm{x}})}{P_{\mathrm{M}}({\bm{y}}|{\bm{x}})} satisfies \forall({\bm{x}},{\bm{y}})\in{\mathcal{S}} , W({\bm{y}}|{\bm{x}})\geq 0 and \sum_{{\bm{y}}}W({\bm{y}}|{\bm{x}})=k_{\bm{x}} , with k_{\bm{x}} being a constant associated with {\bm{x}} . Next, we introduce an autoregressive language model Q_{\theta}({\bm{y}}|{\bm{x}}) , parameterized by \theta , which can be scaled by k_{\bm{x}} to estimate \hat{W}({\bm{y}}|{\bm{x}})=k_{\bm{x}}Q_{\theta}({\bm{y}}|{\bm{x}}) . This leads to \hat{P}_{\mathcal{S}}({\bm{y}}|{\bm{x}})=k_{\bm{x}}P_{\mathrm{M}}({\bm{y}}|{\bm{x}})Q_{\theta}({\bm{y}}|{\bm{x}}) . To ensure that \hat{P}_{\mathcal{S}}({\bm{y}}|{\bm{x}}) is a valid distribution, we normalize it by the partition function Z_{\theta}({\bm{x}})=\sum_{{\bm{y}}}P_{\mathrm{M}}({\bm{y}}|{\bm{x}})Q_{\theta}({\bm{y}}|{\bm{x}}) and replace the \hat{P} with P_{\theta} , resulting in: