A Survey on Progress in LLM Alignment from the Perspective of Reward Design
Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, Usman Naseem · 2025
Read A Survey on Progress in LLM Alignment from the Perspective of Reward Design by Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, Usman Naseem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02666.txt.
Introduction
Miaomiao Ji and co-authors survey progress in aligning large language models from the angle of reward design in a 2025 paper. Rewards bridge human feedback and model training, so their design shapes whether a model follows human values. The survey organizes reward modeling around its math form, how rewards are built in practice, and how they interact with training methods.
From that base it builds a broad taxonomy of reward mechanisms along several axes. The authors trace a shift from training with reinforcement to training without it, and from single tasks to many goals and complex settings. The survey aims to give both conceptual clarity and practical guidance for alignment research.
Catalog record
| CREATOR | Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, Usman Naseem |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02666
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, Usman Naseem Abstract The alignment of large language models (LLMs) with human values and intentions represents a core challenge in current AI research, where reward mechanism design has become a critical factor in shaping model behavior. This study conducts a comprehensive investigation of reward mechanisms in LLM alignment through a systematic theoretical framework, categorizing their development into three key phases: (1) feedback (diagnosis), (2) reward design (prescription), and (3) optimization (treatment). Through a four-dimensional analysis encompassing construction basis, format, expression, and granularity, this research establishes a systematic classification framework that reveals evolutionary trends in reward modeling. The field of LLM alignment faces several persistent challenges, while recent advances in reward design are driving significant paradigm shifts. Notable developments include the transition from reinforcement learning-based frameworks to novel optimization paradigms, as well as enhanced capabilities to address complex alignment scenarios involving multimodal integration and concurrent task coordination. Finally, this survey outlines promising future research directions for LLM alignment through innovative reward design strategies. keywords Large language model alignment, Reinforcement learning, Reward design, RLFH 1 Introduction In recent years, large language models (LLMs) such as GPT-4 OpenAI 2023 , Claude Anthropic 2023 , and Gemini DeepMind 2023 have demonstrated remarkable abilities across a wide range of natural language understanding and generation tasks. These systems, based on transformer architectures Vaswani et al. 2017 and trained on massive corpora, exhibit impressive capabilities in zero-shot and few-shot learning, enabling applications in education, healthcare, programming, research assistance, and more Bommasani et al. 2021 ; Zhang et al. 2023 . However, alongside their growing influence comes an urgent need to ensure that these models act in ways that are beneficial, safe, and aligned with human intentions. Alignment, in the context of artificial intelligence, refers to the extent to which a model’s behavior reflects human values, goals, and ethical norms Gabriel 2020 ; Shetty et al. 2025 . Misaligned LLMs may produce harmful, biased, or misleading outputs even when technically correct or coherent. Concrete manifestations of misalignment include factual inaccuracies (hallucinations) Ji et al. 2023 , toxic language generation Gehman et al. 2020 , propagation of stereotypes and societal biases Sheng et al. 2021 , violations of ethical principles Weidinger et al. 2022 , and lack of interpretability or control Rudin 2019 . These risks are further compounded by concerns over privacy Carlini et al. 2023 , security Zhang et al. 2023 , and model misuse in adversarial settings Greshake et al. 2023 . To address these concerns, the alignment of LLMs has become a cornerstone of safe and responsible AI research. Among the many approaches explored, Reinforcement Learning from Human Feedback (RLHF) has emerged as one of the most widely adopted paradigms Christiano et al. 2017 ; Ouyang et al. 2022 . RLHF enables the incorporation of human preferences into model training by using a RM to guide reinforcement learning (RL) optimization. Rather than relying solely on supervised fine-tuning, RLHF leverages ranked or pairwise human feedback to train a RM, which in turn evaluates and ranks candidate outputs. These scores are then used to optimize the base LLM via algorithms such as Proximal Policy Optimization (PPO) Schulman et al. 2017 . At the heart of this process lies the design and training of the RM. reward design serves as a critical proxy for human judgment, translating high-level values and preferences into computable feedback signals. The reward function determines what behaviors are encouraged or penalized, making it the primary mechanism through which alignment is operationalized Wang et al. 2024 ; Ouyang et al. 2022 . It connects human intentions with optimization objectives, effectively shaping the model’s behavior in complex, high-dimensional environments where direct supervision is infeasible. As such, reward design is not just a technical component of RLHF, it is a foundational challenge for aligning LLMs with human values. Despite its centrality, reward design remains an open and difficult problem. Poorly designed reward functions can lead to unintended behavior, a phenomenon known as reward hacking or overoptimization Skalse and Everitt 2022 . For instance, models may exploit spurious correlations or maximize proxy metrics while neglecting true human intent Amodei et al. 2016 . Additionally, RMs often struggle to generalize across domains, leading to distributional shifts that reduce alignment effectiveness in real-world deployment Gao et al. 2023 . Learned RMs are particularly vulnerable to errors in preference data and may encode annotator bias, further complicating efforts to ensure fairness and robustness Lee et al. 2023 . Finally, LLM alignment via reward design becomes even more complex in emerging settings involving multimodal inputs Tsimpoukelli et al. 2021 , multi-turn interactions Yao et al. 2023 , and multi-task capabilities Ye et al. 2023 , where human values may be heterogeneous, evolving, or underspecified. As the complexity and scale of alignment tasks continue to grow, the field is shifting away from traditional RL-based methods, such as PPO-based RLHF Ouyang et al. 2022 , toward more efficient and interpretable RL-free alternatives. This shift marks a broader evolution in model optimization, where the focus is moving from explicit scalar rewards to more nuanced, data-driven methods that better capture human preferences. Supervised learning-based techniques, such as Direct Preference Optimization (DPO) Rafailov et al. 2023 , align models with human preferences without relying on complex reward structures or policy rollouts. In parallel, in-context preference alignment methods Zhou et al. 2023 ; Liu et al. 2024 allow models to learn directly from examples provided within a prompt, further simplifying the alignment process. Additionally, growing attention is being given to the interpretability and transparency of RMs Urbanczik et al. 2023 , along with efforts to incorporate richer, multi-dimensional feedback structures that more effectively capture the complexities of human values. Together, these trends represent a shift toward more scalable, flexible, and human-centered alignment strategies. Despite these promising developments, the field currently lacks a comprehensive, systematic review focused specifically on the role of reward design in LLM alignment. Existing surveys primarily focus on alignment broadly Zhou et al. 2023 ; Bai et al. 2022 ; Zhou et al. 2024 , often treating reward models as one component among many. However, as alignment methods evolve and diversify, reward mechanisms consistently remain central, whether in preference-based learning, instruction tuning, or scalable human feedback systems. This observation motivates our work: to present a structured and detailed survey of progress in LLM alignment from the perspective of reward design.To that end, this paper makes the following contributions. • First, we introduce a novel conceptual framework that views LLM alignment as a medical treatment process. In this analogy, the model (the patient) produces raw outputs (symptoms), which are then evaluated (diagnosed) via human or automated feedback. Based on this diagnosis, a tailored reward function (prescription) is crafted to guide the model’s optimization (treatment), ensuring that its behavior improves toward desired clinical outcomes (alignment goals). This analogy emphasizes the central role of reward design as the link between observation (feedback) and intervention (optimization), thereby situating it at the core of the alignment pipeline. • Second, we propose a systematic taxonomy of reward mechanisms based on four key dimensions: (1) Construction Basis