Inference-Time Policy Alignment for Fair Reinforcement Learning
Umer Siddique, Peilang Li, Conor Wallace, Yongcan Cao · 2026
Read Inference-Time Policy Alignment for Fair Reinforcement Learning by Umer Siddique, Peilang Li, Conor Wallace, Yongcan Cao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00175.txt.
Introduction
Siddique and co-authors steer trained RL policies toward fairness at inference time. Deep RL agents optimize scalar rewards well. Deployed policies are rigid and costly to adapt, though. New stakeholder preferences may arrive after training. Past fairness methods assume those preferences are known up front. They also demand full retraining of the policy.
Their framework shapes action probabilities with action-dependent welfare scores. The base policy itself stays untouched. Multiplicative shaping adjusts the odds of each action at run time. The method is general and fits any deep RL agent. Tests across several domains show strong fairness gains, they report. Core task performance was preserved in those runs.
Catalog record
| CREATOR | Umer Siddique, Peilang Li, Conor Wallace, Yongcan Cao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00175
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inference-Time Policy Alignment for Fair Reinforcement Learning Umer Siddique, Peilang Li, Conor Wallace, Yongcan Cao Abstract Deep reinforcement learning (RL) agents achieve strong performance by optimizing scalar reward functions. However, once deployed, the policies of these RL agents are often rigid and costly to adapt to new performance criteria. For instance, an agent trained to maximize expected cumulative reward may not accommodate previously unknown stakeholder preferences. Existing approaches to achieve fairness, a type of preference, in RL typically assume that such preferences are known a priori and require complete retraining of the policy under a fairness-oriented metric. Inspired by inference-time alignment in large language models, we investigate the problem of steering a pretrained RL policy toward welfare-based fairness objectives at inference time without updating the base policy’s parameters. We formalize inference-time fairness alignment as a policy shaping problem and propose a multiplicative policy shaping framework that adjusts action probabilities using action-dependent welfare scores, thus requiring no modification to the base policy. Our framework is general and compatible with any deep RL agent. Through extensive experiments across multiple domains, we demonstrate that inference-time policy shaping substantially improves welfare-based fairness objectives while preserving core task performance. 1 Introduction Deep RL has demonstrated remarkable success in domains ranging from robotics ( Peters et al., 2003 ) to game playing ( Silver et al., 2017 ) , typically by optimizing for a single scalar reward function ( Sutton & Barto, 1998 ) . These achievements are often built on fixed task definitions and carefully engineered training pipelines, where RL agents learn policies that maximize return by interacting with an environment. However, such policies can be rigid and brittle at deployment and hence, incorporating new constraints, accommodating evolving stakeholder preferences, or handling out-of-distribution (OOD) shifts ( Haider et al., 2023 ; Ajay et al., 2022 ) typically requires costly retraining or delicate fine-tuning, with no guarantee of robustness. This becomes particularly challenging in socially impactful domains, where the objective extends beyond cumulative reward to include fairness, safety, and multi-stakeholder welfare. For instance, in autonomous driving and decision-making systems, efficiency may need to be traded off for risk sensitivity and safety. Similarly, in resource allocation and transportation, equitable outcomes across user groups may be required. Crucially, such safety or fairness preferences can evolve depending on context, regulatory changes, or societal expectations. Yet, in most deep RL methods, these requirements are addressed during training, and adapting to new preferences or new stakeholders often requires retraining. In practice, such retraining is often impractical due to the associated computational cost, time, and data requirements ( Mujtaba et al., 2025 ) . Consequently, there is a need for techniques that can adapt pretrained RL policies to evolving fairness objectives without retraining the base policy. For instance, when a single base policy serves many clients with heterogeneous fairness needs, or when regulatory changes demand compliance before retraining is feasible. In contrast to the static nature of deployed RL agents, recent progress in large language models (LLMs) suggests that post-training alignment can be feasible ( Kumar et al., 2025 ) . These inference-time alignment techniques aim to steer frozen trained models toward new objectives without updating their parameters. Methods such as reward-guided decoding, representation steering, and lightweight policy adaptation have demonstrated that frozen models can be flexibly directed toward new objectives via auxiliary scoring or controlled decoding strategies ( Li et al., 2023 ; Xu et al., 2024 ; Lin et al., 2025 ) . Other methods, including RL from human feedback (RLHF) ( Ouyang et al., 2022 ; Kaufmann et al., 2024 ) , direct preference optimization (DPO) ( Rafailov et al., 2023 ) , and activation steering ( Turner et al., 2023 ) , further illustrate how behavior can be reshaped at deployment time through preference models or inference-time control. These techniques are effective mainly for autoregressive LLMs, where generation can be viewed as a short-horizon decision process with immediate feedback and where past decisions (generated tokens) are automatically carried forward in the model’s context. However, the applicability of this approach in RL is an open problem as the objective, such as fairness, is typically defined over long horizons (e.g., expectation over trajectories) rather than as instantaneous, per-step signals. Moreover, standard deployed RL policies are Markovian, meaning that they only condition on the current state. Unlike autoregressive generation, past actions do not necessarily remain available to the policy unless they are encoded into the state. Therefore, it remains unclear whether and how inference-time alignment can solve sequential decision-making problems. This paper addresses precisely this question: Given a fixed, pretrained RL policy, can we steer its long-term behavior toward objectives such as fairness at deployment time, while preserving its core performance? Figure 1 : Motivation for inference-time fairness alignment. Left: During training, the policy \pi optimises a scalar return via standard RL. Center: At deployment, the scalar reward is revealed to be structurally decomposable into a vector reward \mathbf{r}\in\mathbb{R}^{n} . Right (A): Deploying the frozen policy \pi directly yields high total return but inequitable allocation across objectives. Right (B): Our approach attaches a lightweight shaping module (QFair) that reweights action probabilities at inference time using welfare-aware Q-values conditioned on the cumulative reward vector. This achieves fair outcomes while preserving the base policy’s total return, without retraining the base policy, and with the ability to adapt to new fairness objectives at deployment. Addressing this alignment problem is both practically and ethically important. From a deployment perspective, inference-time alignment reduces retraining costs, improves adaptability, and mitigates training-deployment mismatches ( Mujtaba et al., 2025 ) . From a societal perspective, learning human-aligned and fair solutions is essential in high-stakes systems, especially when these systems impact multiple end-users. Existing approaches to aligned and fair RL, including multi-objective RL (MORL) with scalarization ( Siddique et al., 2020 ; Hayes et al., 2022 ) , constrained Markov decision processes (CMDPs) ( Bei et al., 2022 ) , and reward shaping ( Ng et al., 1999 ; Kumar & Yeoh, 2025 ) , operate at training time and incorporate fairness directly into the learning objective ( Jabbari et al., 2017 ; Weng, 2019 ; Siddique et al., 2023 ; Siddique et al., 2024 ) . While these approaches can produce effective solutions, they generally assume that the desired preferences are known a priori and require complete retraining or maintaining a portfolio of policies for different scalarizations ( Busa-Fekete et al., 2017 ; Zimmer et al., 2021 ; Do & Usunier, 2022 ; Yu et al., 2023a ) . Generalization techniques such as domain randomization ( Tobin et al., 2017 ) can improve robustness, but they do not enable post hoc objective alignment. Meanwhile, policy shaping methods ( Knox & Stone, 2010 ; Griffith et al., 2013 ) and inference-time alignment strategies in LLMs ( Ouyang et al., 2022 ; Rafailov et al., 2023 ; Balashankar et al., 2024 ; Xu et al., 2024 ; Lin et al., 2025 ) demonstrate inference-time behavioral control, yet they do not formalize fairness alignment in RL via policy shaping. To date, inference-time alignment has largely remained confined to autoregressive generative models rather than long-horizon sequential control with fairness objectives. In this paper, we bridge this gap by proposing an inference-time fairness alignment