ON REWARD-BALANCING METHODS FOR REINFORCEMENT LEARNING

Simone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano · 2026

Read ON REWARD-BALANCING METHODS FOR REINFORCEMENT LEARNING by Simone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20433.txt.

Introduction

Baroncini and two colleagues investigate reward-balancing methods, a new class of algorithms for discounted reinforcement learning. These methods iteratively adjust the reward function to turn the problem into one where optimal policies are greedy. For this normalization process, they analyze the transformations and stress their algebra. They then recast the procedure as an optimal control problem and extend it to model uncertainty via stochastic model sampling, proving normalization guarantees and bounds on random fluctuations. Simulations in a scenario predictive control setting show gains over the state of the art.

Catalog record

CREATORSimone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109850
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20433

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ON REWARD-BALANCING METHODS FOR REINFORCEMENT LEARNING Simone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano Abstract . This paper investigates the so-called reward-balancing methods, a novel class of algorithms for solving discounted-return reinforcement learning (RL) problems. These methods consist of iteratively adjusting the reward function to transform the RL problem into an equivalent one in which the optimal policies are greedy. For this procedure, referred to as normalization process, we provide a theoretical analysis of the involved transformations, emphasizing their algebraic structure. Then, we introduce a control-theoretic reformulation, recasting the reward-balancing procedure into an optimal control framework. The approach is further extended to address model uncertainty through stochastic model sampling, yielding normalization guarantees and probabilistic bounds on stochastic fluctuations. Using the proposed optimal control framework within a scenario model predictive control (MPC) setting, we demonstrate, through simulation studies, performance improvements over the current state-of-the-art. 1. Introduction Most reinforcement learning algorithms developed within the Markov Decision Processes (MDPs) framework rely on a reward function that assigns values to the actions available in each state. Specifically, the role of rewards is to compare actions in the long term, either in terms of average or discounted return. Consequently, many MDPs can induce similar, if not identical, pairwise action comparisons, yet exhibit different performance under a given algorithm. Therefore, the task can be viewed from a different perspective, i.e., modify the reward function in a way to either gain better performance or simplify the problem. This idea has been explored in the literature in different contexts. The concept of modifying rewards without altering the set of optimal policies was first explored by Ng et al. [ 1 ] , where potential functions were used to define policy-invariant reward transformations. This method, commonly referred to as reward shaping , has then become a foundational tool for improving sample efficiency and convergence speed in reinforcement learning. Subsequent works have extended these ideas to more sophisticated shaping functions and learning-based shaping mechanisms, including techniques for automatic reward-balancing [ 2 , 3 ] and methods that learn shaping potentials [ 5 , 4 ] . The reward transformations treated in this paper were recently introduced by Mustafin and collaborators [ 6 ] , where they are termed reward-balancing methods . Unlike traditional RL algorithms, which iteratively improve the policy while keeping the reward function fixed, reward-balancing methods take an alternative approach, called normalization , which consists of fixing the policy as greedy with respect to the current reward function and adjusting the latter to make that policy optimal. Interest in reward balancing is driven by the fact that the method proposed in [ 6 , Section 8] matches the state-of-the-art sample complexity of Q -learning, while being fully parallelizable, thus representing a significant improvement over the current state-of-the-art [ 7 ] . Statement of Contributions. The main contribution of this paper is threefold. First, we provide a comprehensive theoretical analysis of the group of transformations underlying reward-balancing methods, which allows for the iterative reward adjustment process not to alter the original problem. Specifically, we study the action of this group on the space of reward functions, providing insight into the mathematical foundations of this approach, with particular focus on its underlying geometry. Second, we introduce a control-theoretic reformulation, which offers a structured and systematic approach for the design and analysis of reward-balancing methods. From this perspective, the problem is recast as an optimal control problem, which facilitates the design of novel methods through the introduction of constraints and penalty terms. Third, we extend the framework to settings with model uncertainty, adopting a stochastic model-sampling paradigm. In this context, we establish rigorous normalization guarantees and derive probabilistic bounds on the deviation from the average update, offering a quantitative measure of the stochastic fluctuations. These results ensure that reward-balancing methods remain well-posed and stable under stochastic model sampling. Finally, simulation results demonstrate that the proposed optimal control framework, implemented in a scenario receding-horizon setting, outperforms the current state-of-the-art method in terms of policy optimality. Organization. The paper is organized as follows. In Section 2 , we summarize some notation and mathematical preliminaries we need to present the problem under study. In Section 3 , we introduce the discounted-return reinforcement learning framework. In Section 4 , we introduce reward-balancing methods in the framework where the exact MDP model is known, providing their characterization as a group and studying their action on the space of reward functions. In Section 5 , we provide a control-theoretic reformulation of these methods. Finally, in Section 6 , we extend the framework to handle the case of uncertain model, and we provide tools for the design of robust reward-balancing laws. 2. Mathematical Notation and Preliminaries We introduce some of the notations used, starting with some categorical ones which turn out to be useful for out purposes. We denote by \text{Map}(A_{1},A_{2}) the collection of all the morphisms between two objects A_{1} and A_{2} in a given category. For instance, if A_{1} and A_{2} are topological spaces, then \text{Map}(A_{1},A_{2}) denotes the set of all continuous maps from A_{1} to A_{2} . The notation A_{1}\simeq A_{2} , where A_{1} and A_{2} belong to the same category, indicates that A_{1} and A_{2} are isomorphic. The symbol \text{id}_{\scriptscriptstyle A} denotes the identity map on a set A . We will use the symbol \leq to denote both the standard order of \mathbb{R} and the point-wise (partial) order between real-valued functions. Specifically, if \varphi,\psi\in\text{Map}(A,\mathbb{R}) , for some set A , we write \varphi\leq\psi to indicate that \varphi(p)\leq\psi(p) for all p\in A . The same convention holds for \geq . Given an n -dimensional vector space V with an ordered basis \mathcal{B}=\{e_{1},\ldots,e_{n}\} and a vector v\in V , we will use v^{i} to denote the i -th component of v with respect to the basis \mathcal{B} . For vectors, the symbols \leq and \geq are also used to denote the component-wise (partial) order, whenever a basis is fixed. For Euclidean spaces, we use the symbol \|\cdot\|_{p} to denote the p -norm. If we do not refer to a specific norm, we instead use the general symbol \|\cdot\| . We denote by \hat{e}_{i}\in\mathbb{R}^{n} the canonical i -th basis element, i.e. \hat{e}_{i}^{j}=1 if j=i , \hat{e}_{i}^{j}=0 otherwise. If M is a matrix, M_{j}^{i} denotes the element in the i -th row and j -th column. The identity matrix of dimension n is denoted by I_{n} , while \mathds{1}_{n}\in\mathbb{R}^{n} denotes the vector of all ones. The notation \diag\{M_{1},\ldots,M_{n}\} refers to the block-diagonal matrix whose i -th block is M_{i} . If X is a random variable and Y is a set of conditions on X , \mathbb{E}\left[X|Y\right] denotes the expectation of X given Y . If E is an event in a probability space, we denote by \mathbb{P}(E) the corresponding probability. Throughout the paper, we assume the basic knowledge of groups. Nevertheless, we briefly review some key concepts that are useful for understanding some of the main results of the paper. Recall that a homomorphism between two groups (G,*),(H,\cdot) is a map \psi:G\to H satisfying \displaystyle\psi(g*h)=\psi(g)\cdot\psi(h), for all g,h\in G , whereas an anti-homomorphism inverts the order of multiplication \psi(g*h)=\psi(h)\cdot\psi(g) . The symmetric group \text{Sym}(S) of a set S is the group of bijections from S to itself, in

Read the complete text