Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness
Manish Aryal, Faiyaz Azam, Agnivo Banerjee, Sai Sidhanth Manoharan Jayanthi, Allegra Laro, Clément Legentilhomme, Andrew Lin, Florian Lorkowski, Radman Rakhshandehroo, Patric Rommel, Emanuel Ruzak, Nathan Theng, Paul Yushin Rapoport · 2026
Read Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness by Manish Aryal, Faiyaz Azam, Agnivo Banerjee, Sai Sidhanth Manoharan Jayanthi, Allegra Laro, Clément Legentilhomme, Andrew Lin, Florian Lorkowski, Radman Rakhshandehroo, Patric Rommel, Emanuel Ruzak, Nathan Theng, Paul Yushin Rapoport (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23146.txt.
Introduction
Aryal and fourteen coauthors implement infra-Bayesian learning agents for worst case robustness under model misspecification. Classical reinforcement learning assumes an environment apart from agent policy. That fails when other actors anticipate the agent, as with predictors, humans, and institutions. Bayesian methods then yield surely wrong beliefs and endless regret. Infra-Bayesianism splits chance uncertainty from Knightian uncertainty, where no prior can be fairly chosen, and judges actions by worst case outcomes. Their agent shows lower worst case regret than agents and picks the optimal strategy in Newcomb's problem.
Catalog record
| CREATOR | Manish Aryal, Faiyaz Azam, Agnivo Banerjee, Sai Sidhanth Manoharan Jayanthi, Allegra Laro, Clément Legentilhomme, Andrew Lin, Florian Lorkowski, Radman Rakhshandehroo, Patric Rommel, Emanuel Ruzak, Nathan Theng, Paul Yushin Rapoport |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41990 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23146
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness Manish Aryal, Faiyaz Azam, Agnivo Banerjee, Sai Sidhanth Manoharan Jayanthi, Allegra Laro, Clément Legentilhomme, Andrew Lin, Florian Lorkowski, Radman Rakhshandehroo, Patric Rommel, Emanuel Ruzak, Nathan Theng, Paul Yushin Rapoport Abstract Classical reinforcement learning assumes the agent interacts with a fixed environment whose behavior does not depend on the agent’s policy. This assumption breaks down in non-realizable settings where other actors might anticipate the agent’s behavior – most notably those environments crucial to AI safety, where any given agent interacts with predictors, human and other AI agents, and institutions. In such environments, the agent’s model class fails to capture the world in which it operates. Under such misspecification, Classical Bayesian methods can produce confidently wrong posteriors, unreliable decisions, and unbounded regret, as realizability fails to obtain. Infra-Bayesianism is a decision-theoretic framework that addresses these failures by distinguishing ordinary probabilistic uncertainty – where priors can be reasonably chosen – from Knightian uncertainty, where no grounds exist for the construction of such a prior. Infra-Bayesianism does so by evaluating actions on their worst-case outcomes in environments, rather than from posterior expectations or weighted averaging. We present the first proof-of-concept implementation of an infra-Bayesian reinforcement learning architecture for finite-outcome stateless decision problems. Our agent maintains a set of imprecise hypotheses, updates them using infra-Bayesian conditioning, and selects actions by maximizing worst-case expected value. We apply this implementation of the infra-Bayesian maximin decision process to an environment with Knightian uncertainty, and demonstrate a lower worst-case regret as compared to classical reinforcement learning agents. We also investigate Newcomb’s problem and show that the infra-Bayesian agent picks the optimal strategy, outperforming classical decision theory agents. Our results provide a step towards reinforcement learning agents that remain robust under model misspecification and policy-dependent uncertainty. 1 Introduction Reinforcement learning (RL) is usually analyzed under the assumption that the agent interacts with a fixed environment: a Markov decision process (MDP), a partially observable Markov decision process (POMDP), or a Bayesian posterior over such models. This assumption is mathematically convenient and has enabled a large body of convergence and regret guarantees. However, it is poorly suited to settings in which the agent is embedded in an environment too complex to be represented by the agent, or environments containing other actors, that model the agent’s behavior and thereby respond to the agent’s policy rather than merely to its realized actions. Examples include autonomous vehicles whose driving style is anticipated by human drivers, recommendation systems whose users adapt to the system’s known behavior, and decision-theoretic problems such as Newcomb’s problem. In these settings, the environment is not merely unknown; it can be policy-dependent and non-realizable from the agent’s point of view ( 2 ) . Classical Bayesian RL provides a principled treatment of ordinary uncertainty by placing a prior over possible environments. Its strongest guarantees, however, rely on a realizability or grain of truth assumption: the true environment, or a sufficiently exact model of it, must lie in the agent’s hypothesis class. This assumption is implausible for open-ended deployment. The real world contains the agent itself and other systems of comparable or greater complexity, so no tractable hypothesis class can be expected to contain a complete description of the environment. Under such misspecification, Bayesian agents can become confidently wrong, and value-based RL agents can fail to converge to optimal policies ( 6 ) . Infra-Bayesianism (IB) was developed to address this problem by replacing precise Bayesian hypotheses with imprecise, worst-case-evaluated hypotheses. Instead of representing uncertainty only by a probability distribution over worlds, an IB agent may represent Knightian uncertainty over possible worlds and evaluate actions by a lower expectation: the value guaranteed against the worst admissible member of the hypothesis class. It then selects the action that maximizes this worst-case expected value. This maximin structure is safety-relevant because it turns the agent’s objective into a lower-bound guarantee, rather than an average-case prediction. IB also supplies an update rule for these objects that is designed to preserve dynamic consistency across sequential observations ( 3 ; 1 ) . More broadly, IB belongs to the agent foundations and AI safety research agenda, which seeks mathematically precise models of embedded agency. From this perspective, non-realizability is not an edge case but a central feature of advanced agents operating in the real world: the agent is itself part of the environment it is trying to model. Despite substantial theoretical work on IB and its role in the Learning-Theoretic Agenda for AI alignment, comparatively little work has translated its mathematical objects into concrete reinforcement learning agents. This leaves a gap between the formal promise of IB – robust reasoning under non-realizability, Knightian uncertainty, and policy-dependent environments – and the practical question of how an agent could actually represent, update, and plan within it. In this paper, we present a proof-of-concept IB RL architecture. The implementation-level object is not a posterior distribution over MDPs, but a finite set of extremal affine evaluators whose lower envelope defines the agent’s value estimate. This representation allows the agent to perform tractable lower-expectation evaluation, distinguish classical probabilistic mixtures from Knightian uncertainty, and apply dynamically consistent IB-style updates without enumerating full history trees. Our contributions are: 1. We formulate a finite-outcome infra-Bayesian reinforcement learning (IBRL) architecture based on a-measures and infradistributions, including classical and Knightian mixtures, lower-expectation evaluation, and IB-style updates. 2. We show that the architecture recovers ordinary Bayesian behavior in the degenerate case where each infradistribution has a single minimal point and all uncertainty is classical. 3. We demonstrate empirically that an IB agent outperforms a Bayesian agent in terms of worst-case performance in a Knightian uncertain environment. 4. We show that IB decision theory obtains optimal rewards in simple policy-dependent environments, where classical decision theories fail to comprehensively model the causal structure. 2 Background 2.1 Classical and Bayesian reinforcement learning In RL, an agent repeatedly chooses actions and receives observations from an environment. The usual formalism is a Markov decision process, which models the environment as a set of states through which the agent can transition by choosing from a set of actions, receiving a reward upon each transition. The agent aims to optimize its policy, to maximize the cumulative expected reward over future actions. Maximizing rewards is equivalent to minimizing regret , the difference between the expected reward obtainable by the optimal policy and the actually realized reward. Classical Bayesian RL places a prior over environments and updates by Bayes’ rule, which is appropriate in realizable settings where the agent’s hypothesis class contains the truth, and is the basis for many standard convergence arguments ( 13 ; 7 ) . This assumption is a poor fit for embedded or open-world agents, where no computationally feasible hypothesis class can contain the exact environment. In such non-realizable settings, Bayesian updating still produces a posterior, but the posterior need not converge to a useful model for decision-making. This is one of the core motivations for IB in the Learning-Theoretic Agenda ( 6 ; 9 ) . 2.2 Non-realizability and policy-dependent environments The central motivation for this work is that difficulties in RL arise not merely from statistical uncertainty. They also include model misspecification, Knightian uncertainty, and environments whose behavior depends on the agent’s