Weights-Rotated Preference Optimization for Large Language Models

Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang · 2025

Read Weights-Rotated Preference Optimization for Large Language Models by Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17637.txt.

Introduction

Yang and eight co-authors propose weights-rotated preference optimization for language models, in a paper from twenty twenty-five. Direct preference optimization can hack rewards by crushing rejected answers' probability too far. That produces long, dull outputs and forgotten knowledge in the model. Neuron collapse in parameter space is identified as the underlying cause here.

Their method constrains output scores and rotates hidden states orthogonally during training. The policy stays close to the reference model by design throughout. Only a tiny fraction of parameters are trainable in the scheme. Benchmark gains reach several points over strong baselines on standard tests. Pretrained knowledge and expressiveness survive the alignment process better.

Catalog record

CREATORChenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17637

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Weights-Rotated Preference Optimization for Large Language Models Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang Abstract Despite the efficacy of Direct Preference Optimization (DPO) in aligning Large Language Models (LLMs), reward hacking remains a pivotal challenge. This issue emerges when LLMs excessively reduce the probability of rejected completions to achieve high rewards, without genuinely meeting their intended goals. As a result, this leads to overly lengthy generation lacking diversity, as well as catastrophic forgetting of knowledge. We investigate the underlying reason behind this issue, which is representation redundancy caused by neuron collapse in the parameter space. Hence, we propose a novel Weights- Ro tated P reference O ptimization ( RoPO ) algorithm, which implicitly constrains the output layer logits with the KL divergence inherited from DPO and explicitly constrains the intermediate hidden states by fine-tuning on a multi-granularity orthogonal matrix. This design prevents the policy model from deviating too far from the reference model, thereby retaining the knowledge and expressive capabilities acquired during pre-training and SFT stages. Our RoPO achieves up to a 0.5-point improvement on AlpacaEval 2, and surpasses the best baseline by 1.9 to 4.0 points on MT-Bench with merely 0.015% of the trainable parameters, demonstrating its effectiveness in alleviating the reward hacking problem of DPO. 1 Introduction Figure 1: DPO suffers from reward hacking, causing the model’s expressive capability to decline and generating overly long content with meaningless repetitions. The isotropic distribution of the neurons was disrupted during DPO. RoPO mitigates it by rotary tuning, which retains the angle-encoded expressive knowledge. Despite achieving remarkable performance, large language models (LLMs) OpenAI 2023 ; Touvron et al. 2023 ; Bai et al. 2023 ; Yang et al. 2023 ; Yang et al. 2025a ; Yang et al. 2025b still pose the risk of generating content that diverges from human expectations Bai et al. 2022 . To tackle this challenge, researchers introduced reinforcement learning from human feedback (RLHF) to improve the controllability of AI systems by simulating human preferences across various response options Christiano et al. 2023 ; Ouyang et al. 2022 ; Stiennon et al. 2022 ; Dai et al. 2025 . However, RLHF has been widely criticized for its training instability and high sensitivity to hyperparameters. Recently, researchers have developed RL-free direct alignment algorithms Dong et al. 2023 ; Yuan et al. 2023 ; Zhao et al. 2023 , with Direct Preference Optimization (DPO) standing out as a leading approach in this field. Unfortunately, there exists a significant drawback in DPO: reward hacking , also known as reward overoptimization . Deviating from the primary goal of learning the characteristics from preferred completions and discouraging undesirable behaviors in rejected ones, the policy model exploits vulnerabilities in the DPO function by overly suppressing the likelihood of rejected completions to maximize rewards. Consequently, the fine-tuned model suffers from uncontrollable length, diminished diversity, and knowledge forgetting Shen et al. 2023 ; Park et al. 2024 . As shown in Figure 1 , the DPO-trained model produces overly verbose output with meaningless repetitions when answering simple queries. Recently, several approaches have been introduced to address this challenge by modifying the constraints in the loss function Azar et al. 2023 ; Wang et al. 2023 ; Zeng et al. 2024 ; Pal et al. 2024 ; Hong et al. 2024 . However, these methods typically degrade alignment performance and perform worse than DPO. In this paper, we investigate the underlying reasons behind the reward overoptimization of DPO from the perspective of parameter space. Our findings in Section 4.1 indicate that, compared to supervised fine-tuned (SFT) training, DPO optimization causes the model’s neurons to collapse in the parameter space, leading to representation redundancy issues. To overcome this problem, we propose Weights- Ro tated P reference O ptimization ( RoPO ) algorithm, which simultaneously imposes constraints on both intermediate hidden layers from the parameter perspective and output layer from the logits perspective. The output layer is implicitly constrained using the original KL divergence from DPO to maintain the diverse and fluent expressions of the SFT model semantically, while the intermediate hidden layers are explicitly constrained by rotary-tuning with a multi-granularity orthogonal matrix to preserve angle-encoded knowledge. The dual constraints together prevent the policy model from deviating too far from the reference model, retaining the knowledge and expressions acquired during pre-training and SFT stages. Specifically, the specially designed multi-granularity orthogonal matrix consists of fine-grained Givens matrices and global Householder Reflection matrices. Extensive experiments show that RoPO achieves strong alignment performance while preventing excessively long generations, repetitive expressions, and catastrophic forgetting of knowledge. RoPO consistently outperforms all preference optimization baselines in all benchmarks with merely 0.015% of the trainable parameters. Specifically, RoPO surpasses the best baseline by 1.9 to 4.0 points on MT-Bench. By analyzing the training process, we discovered that the complementary constraints of RoPO prevent excessive reduction of the rejected completions likelihood while achieving higher reward accuracy, effectively mitigating the reward hacking problem. Our contributions are summarized as follows: • We delve into the underlying causes behind DPO reward hacking from a novel perspective. • We propose an innovative method RoPO, which simultaneously imposes logit regularization and weight regularization to alleviate the reward hacking problem in DPO. • The extensive experimental results across five benchmarks validate that our RoPO is comprehensively effective, with significantly reduced training parameters and training efficiency. 2 Related Work Direct Preference Optimization. With the widespread application of LLMs, aligning them with human preferences has gained significant attention. Due to training instability and hyperparameter sensitivity in RLHF, recent studies have introduced several RL-free preference optimization methods Dong et al. 2023 ; Zhao et al. 2023 ; Yuan et al. 2023 . Rafailov et al. 2023 derived Direct Policy Optimization (DPO) theoretically by fitting an implicit reward function via reparameterization. As the most influential and effective approach, DPO significantly lowers the alignment barrier for LLMs. Numerous studies followed the proposition of DPO: IPO Azar et al. 2023 revised the loss to minimize the disparity between the ratio of log-likelihoods and a given threshold. KTO Ethayarajh et al. 2024 directly maximized the utility of generations. Meng et al. 2024 proposed SimPO, considering the average log-probability of a sequence as the implicit reward. Others aim to resolve different issues within the DPO objective function Hong et al. 2024 ; Xu et al. 2024 ; Qi et al. 2024 ; Wang et al. 2023 ; Park et al. 2024 ; Liu et al. 2024b ; Zhou et al. 2024 . Among these, the four studies most relevant to ours are DPOP Pal et al. 2024 , SamPO Lu et al. 2024 , LD-DPO Liu et al. 2024c and InterDPO Kojima 2024 . DPOP, SamPO, LD-DPO also focused on the reward hacking problem in DPO and attempted to fix the failure modes by adding a penalty term; InterDPO concentrated on the optimization of intermediate layers by calculating DPO loss at these layers. Different from their method, we incorporate explicit regularization at the intermediate layers from parameter perspective. Orthogonal Regularization. Orthogonal regularization has emerged as a powerful technique in deep learning, which is prevalent in research across the NLP Mao et al. 2020 ; Smith et al. 2017 ;

Read the complete text