Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization
Houssem Sifaou, Osvaldo Simeone · 2025
Read Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization by Houssem Sifaou, Osvaldo Simeone (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14848.txt.
Introduction
Houssem Sifaou and Osvaldo Simeone study reinforcement learning when several simulators of different quality are available, in a 2025 machine learning paper. Training a policy usually needs many runs in an expensive high-fidelity simulator. Offline learning from stored data helps, but it is limited by the size and quality of that data.
Their algorithm, MF-HRL-IGM, mixes offline data with online runs and chooses which simulator to use at each step. The choice is driven by information gain, estimated through a bootstrapping approach, under a fixed total cost budget. Theory shows the method has a no-regret property, meaning its performance catches up with the best fixed choice over time. Experiments show it beating existing benchmarks, making better use of cheap simulators without wasting money on costly ones.
Catalog record
| CREATOR | Houssem Sifaou, Osvaldo Simeone |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization Houssem Sifaou, Osvaldo Simeone Abstract Optimizing a reinforcement learning (RL) policy typically requires extensive interactions with a high-fidelity simulator of the environment, which are often costly or impractical. Offline RL addresses this problem by allowing training from pre-collected data, but its effectiveness is strongly constrained by the size and quality of the dataset. Hybrid offline-online RL leverages both offline data and interactions with a single simulator of the environment. In many real-world scenarios, however, multiple simulators with varying levels of fidelity and computational cost are available. In this work, we study multi-fidelity hybrid RL for policy optimization under a fixed cost budget. We introduce multi-fidelity hybrid RL via information gain maximization (MF-HRL-IGM), a hybrid offline-online RL algorithm that implements fidelity selection based on information gain maximization through a bootstrapping approach. Theoretical analysis establishes the no-regret property of MF-HRL-IGM, while empirical evaluations demonstrate its superior performance compared to existing benchmarks. † † address: KCLIP, CIIPS, Department of Engineering, King’s College London, London, UK 1 Introduction Figure 1: Multi-fidelity hybrid RL setting: The goal is to learn an optimal policy for the real environment \mathcal{M} , given an offline dataset collected in \mathcal{M} and access to K simulators of varying fidelity and cost, which can be leveraged to generate additional simulation data. Reinforcement learning (RL) typically requires millions of interactions with a high-fidelity simulator [ 1 ] . While low-fidelity simulators are computationally cheaper, they introduce performance degradations due to the sim-to-real gap [ 2 , 3 , 4 ] . Offline RL addresses this challenge by training policies on pre-collected datasets, thereby eliminating the need for online simulator interactions [ 5 ] . It has demonstrated success in applications such as interactive recommendation [ 6 ] , control [ 7 , 8 ] , and robotics [ 9 ] . However, its effectiveness critically depends on the size and coverage of the offline dataset in the state–action space [ 5 ] . \displaystyle\min_{Q} \displaystyle\underbrace{\alpha_{c}\,\left(\log\sum_{s,a}\omega(s,a)\exp\bigl(Q(s,a)\bigr)\;-\;\mathbb{E}_{s,a\sim\mathcal{D}^{\text{off}}}\bigl[Q(s,a)\bigr]\right)}_{\text{Conservative value regularization}} (1) \displaystyle+\;\underbrace{\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}^{\text{off}}}\bigl[(Q-\mathcal{T}^{\pi}Q)(s,a)\bigr]^{2}\;+\;\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{B}}\left[\frac{P_{\mathcal{M}}(s^{\prime}\!\mid\!s,a)}{P_{\widehat{\mathcal{M}}}(s^{\prime}\!\mid\!s,a)}\bigl[(Q-{\mathcal{T}}^{\pi}Q)(s,a)\bigr]^{2}\right]}_{\text{Bellman error on offline and online data}}. A promising direction is to combine offline RL with online RL in imperfect simulators, exploiting the strengths of both paradigms – reducing reliance on large offline datasets, while avoiding costly interactions with high-quality simulators [ 3 , 10 , 11 ] . This motivation has driven the development of hybrid offline-online RL methods. Existing work, however, largely assumes access to a single-fidelity simulator, neglecting the trade-off between simulator fidelity and computational cost. As shown in conventional RL, multi-fidelity simulators can provide varying levels of accuracy at different costs, enabling more efficient use of computational resources [ 12 , 13 , 14 , 15 ] . In this work, we introduce multi-fidelity hybrid RL, focusing on the adaptive selection of simulator fidelity levels under a fixed cost budget. Our method, multi-fidelity hybrid RL via information gain maximization (MF-HRL-IGM), is inspired by bootstrap RL [ 16 ] and multi-fidelity Bayesian optimization [ 17 ] , employing the information gain per unit cost as a principled criterion for fidelity selection. Our main contributions are: • We formulate, for the first time, the problem of fidelity-level selection in hybrid RL, integrating bootstrapped RL for uncertainty quantification [ 16 ] with multi-fidelity selection strategies based on information gain [ 17 ] . • We provide a theoretical regret analysis of the proposed algorithm. • We validate our approach experimentally on a standard RL environment, showing superior performance over existing benchmarks. 2 Problem Definition We consider a classical Markov decision process (MDP) setting defined by the tuple \mathcal{M}=\bigl(\mathcal{S},\mathcal{A},r,P_{\mathcal{M}},\rho,\gamma\bigr), where \mathcal{S} denotes the state space, \mathcal{A} is the action space, r(s,a)\in[0,1] is the reward function, P_{\mathcal{M}}(s^{\prime}|s,a) is transition probability, \rho(s) is the initial state distribution, and \gamma\in(0,1] is a discount factor. We address the conventional objective of finding a policy \pi(a|s) that maximizes the expected discounted return J_{\mathcal{M}}(\pi)=\mathbb{E}_{\begin{subarray}{c}s_{0}\sim\rho,\,a_{t}\sim\pi(\cdot|s_{t}),\\ s_{t+1}\sim P_{\mathcal{M}}(\cdot|s_{t},a_{t})\end{subarray}}\Bigg[\sum_{t=0}^{H}\gamma^{t}\,r(s_{t},a_{t})\Bigg], (2) where H denotes the finite horizon of the MDP . As illustrated in Fig. 1 , we study policy optimization when direct interaction with the true environment \mathcal{M} is unavailable, but we have access to K simulators \{\mathcal{M}_{k}\}_{k=1}^{K} of the true environment. Each simulator \mathcal{M}_{k} is characterized by a tuple \bigl(\mathcal{S},\mathcal{A},r_{k},P_{\mathcal{M}_{k}},\rho_{k},\gamma\bigr) , which generally differ from the true environment \mathcal{M} . In particular, each simulator \mathcal{M}_{k} is characterized by a fidelity l_{k} and cost \lambda_{k} satisfying the inequalities l_{1}<l_{2}<\cdots<l_{K} and \lambda_{1}>\lambda_{2}>\cdots>\lambda_{K} . Accordingly, the simulators \{\mathcal{M}_{k}\}_{k=1}^{K} provide increasingly accurate approximations of the true environment \mathcal{M} as index k increases, but the increased accuracy comes at a larger cost. In addition to the possibility to use the K simulators, we are also given a fixed offline dataset \mathcal{D}^{\text{off}}=\{(s_{i},a_{i},r_{i},s^{\prime}_{i})\}_{i=1}^{n} , which is collected under the true dynamics \mathcal{M} using a fixed unknown behavioral policy \pi_{\beta} . Our goal is to learn a policy \pi that maximizes the expected discounted return ( 2 ) under the true environment \mathcal{M} , while imposing a constraint on simulation budget. Specifically, for any run of the policy optimization process, we impose the inequality \sum_{t=1}^{T}\lambda_{k_{t}}\leq\Gamma over the optimization horizon T\gg H , where k_{t}\in\{1,\cdots,K\} is the chosen simulator index at time t , and \Gamma is the overall cost budget. 3 Background In this section, we briefly review online and offline RL and introduce hybrid offline-online RL , which serves as a starting point for the solution to the multi-fidelity hybrid RL problem addressed in this paper. 3.1 Online and Offline RL Standard actor-critic RL algorithms alternate between policy evaluation and policy improvement (see, e.g., [ 18 ] ). We distinguish between online and offline RL algorithms. In online RL , the data \{(s,a,r,s^{\prime})\} used for policy evaluation and policy improvement is collected from the true environment \mathcal{M} using previous iterates of the policy \pi . In contrast, with offline RL , one can only rely on a fixed offline dataset \mathcal{D}^{\text{off}}=\{(s_{i},a_{i},r_{i},s^{\prime}_{i})\}_{i=1}^{n} generated by following a behavioral policy [ 5 ] . To elaborate on this, define the Bellman evaluation operator under policy \pi as \displaystyle\mathcal{T}^{\pi}\widehat{Q}(s,a)=r(s,a)\;+\;\gamma\,\mathbb{E}_{a^{\prime}\sim\pi(\cdot\mid s^{\prime})}\bigl[\widehat{Q}(s^{\prime},a^{\prime})\bigr], (3) where \widehat{Q}(s,a) is an estimate of the action-value function Q(s,a)={\mathbb{E}}[\sum_{t=0}^{H}\gamma^{t}r(s_{t},a_{t})] for policy \pi , where the average is taken as in ( 2 ). The action-value function estimate \widehat{Q}(s,a) is optimized by minimizing the expected Bellman residual as \displaystyle\widehat{Q}\;\leftarrow\;\arg\min_{Q}\;\mathbb{E}_{s,a,s^{\prime}\sim\mathcal{U}}\Bigl[\bigl(Q(s,a)-\mathcal{T}^{\hat{\pi}}\widehat{Q}(s,a)\bigr)^{2}\Bigr], (4) and the policy is updated by acting greedily with respect to function \widehat{Q}(s,a) as \displaystyle\hat{\pi}\;\leftarrow\;\arg\max_{\pi}\;\mathbb{E}_{s,a\sim\mathcal{U}}\bigl[\widehat{Q}(s,a)\bigr]. (5) In ( 4 )-( 5 ), the notation \mathcal{U} represents either a replay buffer \mathcal{B} collected using previous iterates of the policy for online RL, or the fixed offline dataset \mathcal{D}^{\text{off}}=\{(s_{i},a_{i},r_{i},s_{i}^{\prime})\}_{i=1}^{n} for offline RL. For the latter, the objectives in ( 4 ) and ( 5 ) typically include additional regularization terms to mitigate the risk of overestimating the action-value function based on offline data [ 5 ] . This is further discussed next. 3.2 Hybrid Offline-Online RL Real-world decision-making often requires leveraging both offline data and online interactions to achieve sample-efficient and optimal policy learning. To this end, reference [ 3 ] proposed the H2O algorithm, which leverages online data generated by a single simulator {\mathcal{M}}_{1} . H2O builds upon conservative Q-learning (CQL) [ 19 ] , optimizing the objective in ( 1 ). In ( 1 ), the parameter \alpha_{c}>0 controls the strength of the regularization