Learning Explainable Dense Reward Shapes via Bayesian Optimization

Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang, University of Minnesota, Georgia Institute of Technology, Grammarly, University of Arizona {koo00017, mhong, dongyeop}@umn.edu , [email protected] , [email protected] , [email protected] · 2025

Read Learning Explainable Dense Reward Shapes via Bayesian Optimization by Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang, University of Minnesota, Georgia Institute of Technology, Grammarly, University of Arizona {koo00017, mhong, dongyeop}@umn.edu , [email protected] , [email protected] , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16272.txt.

Introduction

Ryan Koo and co-authors present a 2025 method for training language models from human feedback. Standard training gives one score for a whole answer, so the model can not tell which words earned the score. That sparse signal slows learning and blurs credit. The team treats reward shaping as a search problem at the level of single tokens.

Their method borrows explainability tools to guess a reward for each token. A two-level search tunes the shaping rule, with a Bayesian search on top and policy training below. Tests showed better balance in token credit, stronger results on later tasks, and faster arrival at a good policy. The team also proves that additive credit rules of this kind keep the best policy intact. Sharper credit could make feedback training cheaper and clearer.

Catalog record

CREATORRyan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang, University of Minnesota, Georgia Institute of Technology, Grammarly, University of Arizona {koo00017, mhong, dongyeop}@umn.edu , [email protected] , [email protected] , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16272

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Explainable Dense Reward Shapes via Bayesian Optimization Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang, University of Minnesota, Georgia Institute of Technology, Grammarly, University of Arizona {koo00017, mhong, dongyeop}@umn.edu , [email protected] , [email protected] , [email protected] Abstract Current reinforcement learning from human feedback (RLHF) pipelines for large language model (LLM) alignment typically assign scalar rewards to sequences, using the final token as a surrogate indicator for the quality of the entire sequence. However, this leads to sparse feedback and suboptimal token-level credit assignment. In this work, we frame reward shaping as an optimization problem focused on token-level credit assignment. We propose a reward-shaping function leveraging explainability methods such as SHAP and LIME to estimate per-token rewards from the reward model. To learn parameters of this shaping function, we employ a bilevel optimization framework that integrates Bayesian Optimization and policy training to handle noise from the token reward estimates. Our experiments show that achieving a better balance of token-level reward attribution leads to performance improvements over baselines on downstream tasks and finds an optimal policy faster during training. Furthermore, we show theoretically that explainability methods that are feature additive attribution functions maintain the optimal policy as the original reward. The code is publicly available. 1 1 1 https://github.com/minnesotanlp/explainable-dense-rewards 1 Introduction One of the fundamental challenges in reinforcement learning (RL) arises from the sparsity of rewards, where feedback signals are typically only provided at the end of a trajectory, with little to no evaluative information about the intermediate states. As a result, this limitation leads to gaps about the favorability of certain intermediate states, as the agent lacks the required granular feedback regarding which actions were beneficial to the outcome. A similar challenge has been observed in many recent applications of reinforcement learning from human feedback (RLHF), where sparse rewards are common ( Zheng et al. 2023b ; Chaudhari et al. 2024 ) . Given the sequential nature of language models and the token-level value function typically optimized in RLHF ( Zhong et al. 2025 ; Rafailov et al. 2024 ) , it is crucial to determine the contribution of individual tokens to the overall reward. The challenge lies in the fact that the reward is often awarded at the end of sequence generation and represents the quality of the entire sequence as a scalar, which is known to be unstable ( Razin et al. 2024 ; Engstrom et al. 2020 ) and encodes a low-bandwidth signal that does not help determine the relative quality of intermediate tokens. Hence, to address this limitation, it can be beneficial to assign token-level rewards for fine-grained feedback to the policy ( Wu et al. 2023 ; Xie et al. 2024 ) . However, assigning scores to tokens as they get autoregressively generated is computationally intensive, and collecting a large set of fine-grained human annotations for a supervised learning setup is expensive and subject to high disagreement. Thus, some works have explored ”reward shaping” techniques to transform sparse rewards into dense rewards ( Sutton & Barto 2018 ; Ng et al. 1999 ) , thus facilitating more efficient and interpretable optimization, as well as providing finer-grained control over intermediate decisions. For instance, recent attempts examine process rewards ( Lightman et al. 2023 ; Uesato et al. 2022 ) , which provide intermediate feedback for chain-of-thought generations or directly utilize the attention map ( Chan et al. 2024 ) of the reward model to redistribute the reward. However, these steps are complex and often require high-quality human feedback, and the attention on each token may not be directly correlated to its output as an explanation ( Jain & Wallace 2019 ) . In this work, we propose a two-part approach to densify sparse rewards: (1) using explainability methods to construct a dense reward signal, and (2) using Bayesian optimization to learn the weights for a new reward shaping function composed of the explainability scores. We frame reward shaping as a bi-level optimization problem: at the higher level, we optimize the coefficients of the shaped reward function, while at the lower level, we learn the corresponding optimal policy, as illustrated in Figure 1 . First, we estimate token-level contributions using explainability techniques such as LIME ( Ribeiro et al. 2016 ) or SHAPley values ( Lundberg & Lee 2017 ) . Next, since these explainability methods are known to be sensitive to noise ( Li et al. 2020 ) , we treat them as uncertain estimates and seek to learn an optimal weighting scheme over them. However, due to the complexity of the reward landscape, exhaustively evaluating all possible weight configurations is computationally infeasible. To address this, we treat the problem as black-box optimization and employ Bayesian optimization to learn the best reward coefficients as a natural method robust to noisy objective functions ( Fröhlich et al. 2020 ; Daulton et al. 2022 ) . We demonstrate that explainability offers a natural way to extract more information from the reward model to densify rewards and satisfies as potential-based reward shaping ( Ng et al. 1999 ) so that we satisfy policy invariance or that the optimal policy with the original reward function remains unchanged. Furthermore, we introduce a new optimization setup incorporating different sources of token-level information and show that Bayesian Optimization can aid in learning the best reward-shaping function. Empirically, we show that explainability methods positively impact the RL training compared to sparse rewards through accelerating learning and more stable updates in the value function. We also show that adding Bayesian Optimization to properly shape rewards improves generation quality on downstream tasks compared to its naive setups. Figure 1: Overview of the bilevel optimization setup to find the best reward shape and the optimal policy. The pipeline involves an outer and inner training loop, where the outer step optimizes the Bayesian optimization model and samples the weights for our reward shape. The inner step optimizes the classic RLHF objective. 2 Preliminaries In this section, we first introduce the classic RLHF approach and build the relevant background for explainability and Bayesian optimization for our method. 2.1 Token-level MDP for RLHF We cast RL for language modeling in the sequential decision-making setting as a Markov Decision Process (MDP) by tuple \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\gamma,r) . Here the state space \mathcal{S} contains states s_{t}=\{x,\,y_{0:t}\} , where x denotes the input prompt and y_{0:t} the sequence produced up to token t . The action set \mathcal{A} consists of next‑token choices a_{t} sampled from the model’s token distribution. Transition dynamics are governed by the kernel \mathcal{P} , which assigns a probability P(s^{\prime}\mid s,a) to moving from state s to s^{\prime} after taking action a . A scalar discount factor \gamma\in[0,1) down‑weights future returns, and the r(s_{t},a_{t}) is the reward. In the RLHF pipeline, a model is first trained over a preference dataset \mathcal{D}=\{(x,y_{c},y_{r})\} consisting of prompts x and a pair of chosen and rejected responses y_{c},y_{r} via supervised fine-tuning (SFT) followed by a two-stage process (1) reward modeling and (2) RL training. From the SFT model, the reward function is modeled under a preference model encoded over the sentence level and trained as a binary classification problem via maximum likelihood: P(y_{c}\succ y_{r})=\frac{\exp\{r(x,y_{c})\}}{{\exp\{r(x,y_{c})\}}+{\exp\{r(x,y_{r})\}}} After learning the reward model, a language model is

Read the complete text