Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints

Daren Yao, Jinsong Yuan, Ruike Chen · 2025

Read Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints by Daren Yao, Jinsong Yuan, Ruike Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08466.txt.

Introduction

Daren Yao, Jinsong Yuan, and Ruike Chen work on teaching small language models to match what people prefer. Their study appeared in twenty twenty-five. Small models often lag far behind large ones, and the usual training methods lose ground in that gap.

The authors offer two light changes to a method called DPO. Both add a margin, a target gap between good and bad answers. One also focuses updates on the chosen answers and learns most from hard cases. On a test called AlpacaEval, the best variant raised the win rate by two points over its base method. Scores on a second test stayed strong, with the best gains in science and humanities tasks.

Catalog record

CREATORDaren Yao, Jinsong Yuan, Ruike Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08466

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Small LLM Alignment through Margin-Based Objective Modifications under Resource Constraints Daren Yao, Jinsong Yuan, Ruike Chen Abstract Small large language models (LLMs) often face difficulties in aligning output to human preferences, particularly when operating under severe performance gaps. In this work, we propose two lightweight DPO-based variants—Adaptive Margin-Sigmoid Loss and APO-hinge-zero—to better address underperformance scenarios by introducing margin-based objectives and selective update mechanisms. Our APO-hinge-zero method, which combines hinge-induced hard-example mining with the chosen-focused optimization of APO-zero, achieves strong results. In AlpacaEval, APO-hinge-zero improves the win rate by +2.0 points and the length-controlled win rate by +1.4 points compared to the APO-zero baseline. In MT-Bench, our methods maintain competitive performance in diverse categories, particularly excelling in STEM and Humanities tasks. These results demonstrate that simple modifications to preference-based objectives can significantly enhance small LLM alignment under resource constraints, offering a practical path toward more efficient deployment. 1 Introduction Large language models (LLMs) have led to significant gains in natural language processing tasks, producing coherent and contextually apt text. Yet even state-of-the-art models often yield outputs that deviate from human preferences, motivating extensive research into alignment techniques ranging from reinforcement learning with human feedback (RLHF) to Direct Preference Optimization (DPO) ( Wang et al. 2024 ) . Although alignment research has predominantly focused on large-scale models with tens of billions of parameters, many real-world applications, particularly those constrained by cost, computation, or privacy, require the deployment of much smaller models (under 1B parameters). Recent studies ( Chen and Varoquaux 2024 ) highlight the strategic importance of small LLMs for low-latency inference, edge computing, and private environments. However, aligning small models remains especially challenging due to their limited capacity to faithfully represent complex user preferences. In this project, we address the question: Can lightweight modifications to preference-based alignment objectives substantially improve small LLMs under resource constraints? Motivated by the observation that standard DPO formulations may not fully accommodate the performance gaps typical of small LLMs, we explore novel objective variants that integrate margin-based mechanisms. Specifically, we introduce Adaptive Margin-Sigmoid Loss and APO-hinge-zero (with a Softplus variant), with the aim of improving alignment efficiency by: • combining stable log-sigmoid optimisation with explicit margin separation, • implementing hard-example mining to sparsify updates and focus learning on difficult samples, • improving robustness against noisy or ambiguous preference data through mild label smoothing. Our methods are computationally inexpensive, easy to integrate into existing preference-optimization pipelines, and particularly suited for small-scale models. Preliminary results on AlpacaEval and MT-Bench demonstrate that margin-based objectives can meaningfully boost both win rates and length-controlled win rates without incurring additional complexity. In essence, this work highlights the critical role of objective design in small model alignment and provides practical insights into scalable, cost-effective strategies for deploying aligned LLMs beyond massive cloud infrastructures. 2 Related Works 2.1 DPO Direct Preference Optimization (DPO) is a lightweight alignment algorithm that bypasses the complexity of reinforcement learning by reparameterizing the RLHF objective into a simple classification loss. Instead of first fitting a separate reward model and then running PPO (or another RL algorithm), DPO shows that under a Bradley–Terry preference model, the optimal policy can be expressed in closed form as a function of the model’s own logits relative to a reference policy. By applying a change of variables, a loss is derived that directly increases the logarithmic probability gap between the preferred and preferred responses, weighted to prevent model collapse Rafailov et al. 2024 . After this derivation, the DPO objective becomes: \mathcal{L}_{\text{DPO}}(\pi_{\theta};\,\pi_{\mathrm{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim D}\\ (1) \Bigl[\log\sigma\bigl(\beta\ln\!\tfrac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_{w}\mid x)}-\beta\ln\!\tfrac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l}\mid x)}\bigr)\Bigr]\hskip 5.0pt where: • \sigma(z)=1/(1+e^{-z}) is the logistic function, • \pi_{\mathrm{ref}} is typically the supervised fine-tuned model, • \beta>0 controls how strongly the model resists drifting from \pi_{\mathrm{ref}} . In practice, preference data may contain noisy or ambiguous comparisons, making hard preference optimization suboptimal. In TRL von Werra et al. 2020 , the DPO loss is actually the conservative DPO loss. Conservative DPO introduces a label smoothing factor \epsilon\in[0,0.5) into the DPO loss, balancing the confidence between the preferred and less preferred responses: \mathcal{L}_{\mathrm{cDPO}}(\pi_{\theta};\pi_{\mathrm{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim D} (2) \left[(1-\epsilon)\log\sigma(\beta\cdot\Delta r)+\epsilon\log\sigma(-\beta\cdot\Delta r)\right], where \Delta r=\ln\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_{w}\mid x)}-\ln\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l}\mid x)} . When \epsilon=0 , the conservative DPO reduces to the original DPO objective (which is the default setting of TRL). 2.2 DPO–Hinge While DPO’s logistic loss softly enforces a preference gap, DPO-Hinge replaces cross-entropy with a hinge-margin loss to guarantee a minimum margin \alpha between preferred and dispreferred logits. This can accelerate learning on borderline examples at the cost of an extra hyperparameter: \mathcal{L}_{\text{DPO-Hinge}}(\pi_{\theta};\,\pi_{\mathrm{ref}})=\mathbb{E}_{(x,y_{w},y_{l})\sim D} (3) \Bigl[\max\Bigl(0,\;\alpha-\Bigl(\beta\ln\tfrac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_{w}\mid x)}-\beta\ln\tfrac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l}\mid x)}\Bigr)\Bigr)\Bigr], where \alpha>0 is the desired minimum log–ratio gap. 2.3 AOT Alignment via Optimal Transport (AOT) is a distributional alignment method that enforces first-order stochastic dominance (FSD) of chosen (positive) samples over rejected (negative) samples at the distributional level. Specifically, given empirical samples of rewards from the chosen responses \{u_{i}\}_{i=1}^{n} and rejected responses \{v_{i}\}_{i=1}^{n} , AOT constructs empirical measures: \hat{\mu}_{U}=\frac{1}{n}\sum_{i=1}^{n}\delta_{u_{i}},\quad\hat{\mu}_{V}=\frac{1}{n}\sum_{i=1}^{n}\delta_{v_{i}}. The violation of stochastic dominance can be expressed as a one-dimensional optimal transport (OT) problem with convex surrogate loss h(\cdot) : \mathrm{OT}_{h}(\hat{\mu}_{U},\hat{\mu}_{V})=\frac{1}{n}\sum_{i=1}^{n}h\bigl(u_{(i)}-v_{(i)}\bigr), (4) where (u_{(i)},v_{(i)}) represent sorted samples from the chosen and rejected distributions respectively. A typical choice for the surrogate loss h includes logistic or squared-hinge functions. The AOT fine-tuning objective is then defined as minimizing this OT cost over the policy parameter \theta : \min_{\theta}\;\mathrm{OT}_{h}\left((r\circ\pi_{\theta})\sharp\mu_{+},\,(r\circ\pi_{\text{ref}})\sharp\mu_{-}\right), (5) which can be efficiently optimized through sorting-based computation, providing a stable optimization landscape Melnyk et al. 2024 . 2.4 APO Anchored Preference Optimization (APO) is a family of contrastive alignment objectives designed to give explicit control over how the model’s likelihood of preferred and non-preferred responses is changed during training. While DPO simply maximizes the gap between preferred and non-preferred log-probabilities, APO “anchors” each side of the comparison to either increase or decrease in a predictable way, depending on the relative quality of the examples and the target model D’Oosterlinck et al. 2024 . Concretely, the following equation is the implicit reward (log-ratio) for response y under prompt x . r_{\theta}(x,y)\;=\;\beta\;\ln\frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} (6) APO defines two core variants: APO-zero explicitly increases the likelihood of the preferred response and decreases that of the non-preferred response: \mathcal{L}_{\text{APO-zero}}(x,y_{w},y_{l};\theta)=\hskip 40.00006pt (7) \hskip 40.00006pt-\sigma\!\bigl(r_{\theta}(x,y_{w})\bigr)+\sigma\!\bigl(r_{\theta}(x,y_{l})\bigr), APO-down method decreases both likelihoods, but decreases the non-preferred more strongly: \mathcal{L}_{\text{APO-down}}(x,y_{w},y_{l};\theta)=\hskip 40.00006pt (8) \hskip 40.00006pt\sigma\bigl(r_{\theta}(x,y_{w})\bigr)-\sigma\bigl(r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})\bigr) Here \sigma(z)=1/(1+e^{-z}) and \beta>0 controls how far the model may drift from the reference policy 2.5 Other Alignment Methods Beyond DPO, several extensions such as IPO ( Azar et al. 2023 ) and EXO ( Ji et al. 2024 ) have been proposed to stabilize training or leverage multiple preferences. However, all remain within the direct preference optimization paradigm. Traditional alignment methods primarily rely on reinforcement learning. RLHF ( Ouyang et al. 2022 ) and its variant RLAIF ( Bai et al. 2022 ) optimize policies against learned reward models, requiring costly reward modeling and unstable online training, unsuitable under resource constraints. SLiC ( Zhao et al. 2023 ) reformulates alignment as supervised learning via critiques, while reward distillation approaches ( Zhou et al. 2023 ) simplify reward modeling yet still inherit RLHF inefficiencies. In general, methods that avoid explicit reinforcement learning, such as DPO, offer more practical paths under constrained computational budgets. 3 Methods All models were fine-tuned on a 0.5B parameter LLM (Qwen2.5-0.5B-Instruct)

Read the complete text