K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning

Zixuan Xia, Quanxi Li · 2026

Read K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning by Zixuan Xia, Quanxi Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23056.txt.

Introduction

Xia and Li propose using a one-dimensional Kalman filter for online reward estimation instead of fixed reward normalization in policy gradient reinforcement learning. The filter tracks the latent reward mean step by step, damping noisy returns and adjusting as conditions shift. It adds minimal overhead and needs no changes to existing policy architectures. Tests on LunarLander and CartPole show faster convergence and lower training variance than standard normalization.

Catalog record

CREATORZixuan Xia, Quanxi Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34645
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23056

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning Zixuan Xia, Quanxi Li Abstract We propose a simple yet effective alternative to reward normalization in policy gradient reinforcement learning by integrating a 1D Kalman filter for online reward estimation. Instead of relying on fixed heuristics, our method recursively estimates the latent reward mean, smoothing high-variance returns and adapting to non-stationary environments. This approach incurs minimal overhead and requires no modification to existing policy architectures. Experiments on LunarLander and CartPole demonstrate that Kalman-filtered rewards significantly accelerate convergence and reduce training variance compared to standard normalization techniques. Code is available at https://github.com/Sumxiaa/Kalman_Normalization . † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) has emerged as a powerful paradigm for sequential decision-making, enabling agents to learn optimal behaviors through interaction with the environment. Among the broad family of RL algorithms, policy gradient methods stand out for their effectiveness in high-dimensional and continuous action spaces ( 9 ) , where value-based methods often struggle due to discretization challenges or instability in action selection. Despite their theoretical appeal, policy gradient methods suffer from high variance in gradient estimates, primarily caused by the stochastic nature of both the environment and the policy. High-variance returns can lead to noisy updates, poor sample efficiency, and unstable learning, especially in early-stage training. As a result, numerous variance reduction techniques have been proposed to improve training dynamics and generalization. One commonly adopted strategy is reward normalization , where observed returns are standardized via Z-score normalization using running estimates of the mean and standard deviation ( 3 ; 5 ) . This heuristic smooths fluctuations in reward signals and improves the conditioning of policy gradient updates. However, such techniques are often sensitive to hyperparameters (e.g., decay rates), require careful tuning, and crucially rely on the assumption that reward statistics remain stationary throughout training—a condition rarely satisfied in dynamic or partially observable environments. To address these limitations, we explore a principled and adaptive alternative grounded in Bayesian estimation theory: Kalman filtering ( 2 ) . Originally developed for linear state estimation under uncertainty, Kalman filters have found extensive use in robotics, tracking, and control ( 12 ) , but their potential for reward modeling in reinforcement learning remains underexploited. In this paper, we propose to replace traditional reward normalization with a lightweight, one-dimensional Kalman filter that recursively estimates the latent mean of the return signal. Our approach models the expected reward as a latent stochastic process subject to both process and observation noise. By fusing prior knowledge with new observations in a statistically optimal manner, the Kalman filter provides a dynamic, uncertainty-aware estimate of the reward baseline. This allows the agent to better cope with noisy feedback, non-stationarity, and abrupt shifts in reward distributions. Our method is simple to implement, computationally efficient, and architecture-agnostic. It requires no change to the policy network, making it compatible with a wide range of existing policy gradient algorithms such as REINFORCE, Actor-Critic, GAE, and PPO. Furthermore, the Kalman filter naturally adapts to the variability of the learning environment without requiring additional hyperparameter tuning beyond initial noise estimates. We evaluate our method on two well-established control tasks from OpenAI Gym— CartPole-v1 and LunarLander-v2 —which respectively represent low-variance dense reward and high-variance sparse reward settings. Experimental results demonstrate that Kalman-filtered reward estimation leads to smoother training, faster convergence, and improved stability compared to conventional Z-score normalization. Our main contributions are summarized as follows: • We propose a novel application of Kalman filtering for reward estimation in policy gradient methods, offering a principled replacement for heuristic normalization strategies. • We design a lightweight and efficient 1D Kalman filter module that adaptively tracks non-stationary reward signals in online learning settings. • We empirically validate the effectiveness of our approach across multiple policy gradient algorithms and benchmark environments, showing consistent improvements in convergence speed and stability. 2 Related Work 2.1 Policy Gradient Methods Policy gradient methods directly optimize a parameterized policy \pi_{\theta}(a|s) by estimating gradients of the expected return ( 9 ) . A standard form of the policy gradient is \nabla_{\theta}J(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{t}\right], (1) where \hat{A}_{t} denotes an estimate of the advantage. Early methods such as REINFORCE ( 13 ) use Monte Carlo returns as the learning signal, which yields an unbiased but often high-variance gradient estimate. Actor-critic methods reduce this variance by learning a value-function baseline, while Generalized Advantage Estimation (GAE) ( 6 ) further introduces a bias-variance trade-off through exponentially weighted temporal-difference residuals. Proximal Policy Optimization (PPO) ( 7 ) improves stability by constraining policy updates with a clipped surrogate objective. Despite these advances, policy-gradient methods remain sensitive to the scale and variability of the return or advantage signal. In practice, normalizing these signals is often crucial for stable training. Our work focuses on this normalization step and proposes a lightweight Bayesian filtering mechanism that can be inserted into standard policy-gradient pipelines without modifying the policy architecture. 2.2 Reward and Advantage Normalization Reward, return, and advantage normalization are widely used to reduce gradient variance and improve optimization stability in policy-gradient reinforcement learning. A common approach is Z-score normalization, which standardizes a return or advantage signal using running estimates of its mean and standard deviation: \hat{G}_{t}=\frac{G_{t}-\mu_{t}}{\sigma_{t}+\epsilon}. (2) Although simple and effective, such running-statistics methods are typically heuristic. Their behavior depends on decay rates and window sizes, and they do not explicitly model uncertainty in the estimated statistics. This becomes problematic when the reward distribution changes during learning, as is common in online RL where the policy itself is non-stationary. Several approaches address related issues of scale and stability. PopArt ( 10 ) rescales value-function targets while preserving the un-normalized output, enabling learning across tasks with different reward magnitudes. Multi-task reinforcement learning methods have also used normalization to handle heterogeneous reward scales across environments ( 1 ) . Meta-gradient methods ( 14 ) adapt learning hyperparameters online, including quantities related to learning signals, but usually introduce additional computational and algorithmic complexity. Our method takes a simpler probabilistic view: the observed return or advantage is treated as a noisy observation of a latent learning signal. Rather than maintaining heuristic running moments, we estimate this latent signal and its uncertainty online with a scalar Kalman filter. This yields an adaptive normalization rule that remains plug-and-play for standard policy-gradient algorithms. 2.3 Kalman Filtering in Reinforcement Learning Kalman filtering is a classical Bayesian method for online state estimation under process and observation noise ( 2 ; 12 ) . In reinforcement learning, Kalman-style methods have mainly been explored for value estimation, uncertainty modeling, or belief-state inference. For example, Shashua and Mannor ( 8 ) use Kalman filtering to track uncertainty in value-function approximation for trust-region policy optimization. Painter-Wakefield and Parr ( 4 ) study sparse reinforcement learning methods that use filtering-style estimation for local value modeling. Kalman filters and related Bayesian filters are also natural tools in partially observable settings, where the agent must infer hidden states from noisy observations. Most of these works use filtering to estimate states, values, or model uncertainty. In contrast, our method applies a lightweight one-dimensional Kalman filter directly to the reward, return, or advantage normalization signal. The goal is not to replace the value function or modify the policy update rule, but to provide an

Read the complete text