Imperfect World Models are Exploitable

Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, [1.5mm] Mykel J. Kochenderfer, Subramanian Ramamoorthy · 2026

Read Imperfect World Models are Exploitable by Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, [1.5mm] Mykel J. Kochenderfer, Subramanian Ramamoorthy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15960.txt.

Introduction

Logan Mondal Bhamidipaty and four colleagues define model exploitation in reinforcement learning. A world model is exploitable if it implies one policy should be strictly preferred over another while the true transition model implies the reverse. They link it to reward hacking, but the prior inevitability proof does not transfer. A general theory proves exploitation essentially unavoidable on large policy sets, with the hacking claim as a special case. Finite-set unhackability conditions have no counterpart precluding exploitation, so the authors relax the notion and derive a safe horizon.

Catalog record

CREATORLogan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, [1.5mm] Mykel J. Kochenderfer, Subramanian Ramamoorthy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45300
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Imperfect World Models are Exploitable Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, [1.5mm] Mykel J. Kochenderfer, Subramanian Ramamoorthy Abstract We propose a novel definition of model exploitation in reinforcement learning. Informally, a world model is exploitable if it implies that one policy should be strictly preferred over another while the environment’s true transition model implies the reverse. We analogize our definition with a prior characterization of reward hacking but show that the associated proof of inevitability does not transfer to exploitation. To overcome this obstruction, we develop a general theory of reward hacking and model exploitation that proves that exploitation is essentially unavoidable on large policy sets and yields the corresponding claim for hacking as a special case. Unfortunately, we also find that the conditions that guarantee unhackability in finite policy sets have no counterpart that precludes exploitation. Consequently, we introduce a relaxed notion of exploitation and derive a safe horizon within which it can be avoided. Taken together, our results establish a formal bridge between reward hacking and model exploitation and elucidate the limits of safe planning in world models. 1 Introduction Efficient real-world planning typically requires approximating complex dynamics ( 40 ; 23 ) . In the best case, well-chosen approximations unravel otherwise intractable problems, as with admissible heuristics in search ( 19 ) , model reduction in dynamical systems ( 2 ) , and latent-space world models in sequential decision-making ( 14 ) . In the worst case, however, ill-chosen approximations can compromise safety, as with mimicry in evolutionary biology ( 6 ) , arbitrage in financial markets ( 35 ) , and speedrunning in video games ( 39 ) . In reinforcement learning (RL), this liability arises not only exogenously from adversaries but also endogenously from the optimizer itself: an agent trained to maximize total expected reward under an imperfect model 1 1 1 We use “model” to mean the transition model of a Markov decision process ( Section 2 ) or an agent’s approximation to it, sometimes called a world model ( 11 ) . We do not use it to mean the policy (as a generative model of actions) as is sometimes meant in other RL literature. may discover behavior that performs well in simulation but poorly in reality ( 21 ; 44 ) . This failure mode is called model exploitation ( 11 ; 22 ) , and a natural question is whether it can be avoided. This question pertains to notable results in RL theory including the simulation lemma ( 26 ) and the value equivalence principle ( 10 ) . Both implicitly use predictive performance to measure the quality of a model’s approximation. The simulation lemma does so cardinally by bounding the error from optimizing under an imperfect model, while the value equivalence principle does so locally by characterizing when two models induce identical Bellman updates. Modern world models also emphasize predictive performance, typically learning latent-space models of transitions ( 14 ; 5 ) . We argue that while this emphasis on performance is necessary for efficient planning, it is not sufficient for safe planning. Recent work on reward specification ( 12 ; 1 ) views reward functions as observations of the designer’s intent to communicate a goal, which may be an imperfect reflection of a true objective. Empirical transition models are similarly imperfect. In this setting, the natural notion of safety is ordinal rather than cardinal ( does the model preserve which policies are better than which? ) and global rather than local ( does it do so across the entire policy set, not just on the chosen classes? ). Concretely, a proxy model is safe in our sense if it does not invert the policy ordering induced by the true model. That is, whenever reality prefers a policy \pi over \pi^{\prime} , a safe approximation should not prefer \pi^{\prime} over \pi . When such an inversion exists, we say the pair of transition models is exploitable ( Definition 1 ). Our formalization of model exploitation is analogous to the notion of reward hacking in 41 . Unfortunately, their characterization of when hacking exists and is avoidable does not transfer to exploitation ( Section 3.2 ). We overcome this by introducing a more general theory, from which four main contributions emerge. 1. We propose a new definition of model exploitation ( Definition 1 ) that, to the best of our knowledge, is the first ordinal notion of safety for world models. 2. We develop a unified theory of reward hacking and model exploitation that characterizes exploitation on common policy sets ( Corollaries 1 and 2 ), yields Theorem 1 of 41 as a special case ( Corollary 3 ), and provides a shared geometric intuition for both phenomena ( Lemmas 1 and 2 ). 3. We give a construction reducing any instance of model exploitation to one of reward hacking ( Proposition 3 ), but show that a converse construction does not exist ( 1 ). We further prove that the conditions that guarantee unhackability are insufficient to guarantee unexploitability ( ). 4. We introduce a relaxed notion of model exploitation, \varepsilon -exploitability ( Definition 3 ), and use the tight form of the simulation lemma ( 26 ) recently proved by 31 to derive a safe horizon for discounted Markov decision processes within which \varepsilon -unexploitability is guaranteed ( Theorem 2 ). 2 Preliminaries We first recap the preliminaries for reinforcement learning ( 42 ) and introduce our definition of model exploitation. 2.1 Reinforcement learning We consider a Markov decision process (MDP) \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{T},d_{0},\mathcal{R},\gamma) where \mathcal{S} is the state space, \mathcal{A} is the action space, \mathcal{T}\colon\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) is the transition model, d_{0}\in\Delta(\mathcal{S}) is the initial state distribution, \mathcal{R}\colon\mathcal{S}\times\mathcal{A}\to\mathbb{R} is the reward function, and \gamma\in[0,1) is the discount factor. As in 41 , we assume finite \mathcal{S} with all states reachable and finite \mathcal{A} with |\mathcal{A}|>1 . A stationary policy \pi\colon\mathcal{S}\to\Delta(\mathcal{A}) maps each state to a distribution over actions. A non-stationary policy \pi=(\pi_{0},\pi_{1},\pi_{2},\dots) is a sequence of mappings \pi_{t}\colon\mathcal{S}\to\Delta(\mathcal{A}) , allowing the action distribution to depend on the time step. Rolling out a policy produces a trajectory \tau=(s_{0},a_{0},r_{0},\dots) whose discounted return is G(\tau)=\sum_{t=0}^{\infty}\gamma^{t}r_{t} . The value of a policy is its expected return J(\pi)=\mathbb{E}_{\tau\sim\pi}[G(\tau)] , or, equivalently, J(\pi)=\langle\mathcal{R},\mathcal{F}^{\pi}_{\mathcal{T}}\rangle where \mathcal{F}^{\pi}_{\mathcal{T}}(s,a)=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{\infty}\gamma^{t}\mathbbm{1}(s_{t}=s,a_{t}=a)] are the discounted visit counts of \pi under \mathcal{T} . We make the standard assumption in classical RL and optimal control ( 42 ; 7 ) that designers use value to compare policies and prefer high-value policies over low-value ones. An environment is an MDP without a reward function (\mathcal{S},\mathcal{A},\mathcal{T},d_{0},\_,\gamma) , 2 2 2 This is sometimes called a controlled Markov process. and a task is an MDP without a transition model (\mathcal{S},\mathcal{A},\_,d_{0},\mathcal{R},\gamma) . When considering multiple reward functions in a fixed environment, we write J_{\mathcal{R}}(\pi)=\langle\mathcal{R},\mathcal{F}^{\pi}\rangle (dropping the subscript on \mathcal{F}^{\pi} when \mathcal{T} is fixed). When considering multiple transition models in a fixed task, we write J_{\mathcal{T}}(\pi)=\langle\mathcal{R},\mathcal{F}^{\pi}_{\mathcal{T}}\rangle . We use J_{i} as shorthand for J_{\mathcal{R}_{i}} or J_{\mathcal{T}_{i}} depending on whether the index ranges over reward functions or transition models. Since \mathcal{S} and \mathcal{A} are finite, the space of stationary policies can be represented as a product of |\mathcal{S}| probability simplices \Delta(\mathcal{A})^{|\mathcal{S}|} . We write \Pi^{+} for the interior of this space,

Read the complete text