CIG: Exploration via Conditional Information Gain

Tim Joseph, Marcus Fechner, Philipp Stegmaier, Karam Daaboul, J. Marius Zöllner · 2026

Read CIG: Exploration via Conditional Information Gain by Tim Joseph, Marcus Fechner, Philipp Stegmaier, Karam Daaboul, J. Marius Zöllner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20878.txt.

Introduction

Tim Joseph and four colleagues propose CIG, exploration via conditional information gain for reinforcement learning. Intrinsic rewards depend on different settings. Life-long rewards score each transition against past experience but ignore within-rollout redundancy. Episode rewards penalize intra-trajectory repeats but discard lifetime gains. Hybrid methods merge both signals through heuristic weights or need Gaussian-process dynamics that fail beyond small spaces. CIG derives a tractable stand-in as a log-det goal over an ensemble disagreement kernel.

Catalog record

CREATORTim Joseph, Marcus Fechner, Philipp Stegmaier, Karam Daaboul, J. Marius Zöllner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115199
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CIG: Exploration via Conditional Information Gain Tim Joseph, Marcus Fechner, Philipp Stegmaier, Karam Daaboul, J. Marius Zöllner Abstract Intrinsic rewards for exploration in reinforcement learning condition on different contexts: lifelong rewards score each transition against accumulated experience but ignore within-rollout redundancy; episodic rewards penalize intra-trajectory repetition but discard lifetime progress. Hybrid methods combine both signals through heuristic weights or require Gaussian-process dynamics that do not scale beyond low-dimensional state spaces. Trajectory-level information gain decomposes into per-step terms that condition on the replay buffer and rollout prefix simultaneously, but remains intractable for deep models. We derive the C onditional I nformation G ain (CIG) reward as a tractable surrogate: a log-determinant objective over an ensemble disagreement kernel whose Cholesky factorization yields causal per-step rewards that retain both conditioning sets while scaling to high-dimensional state spaces. We instantiate CIG in a model-based setting, where rollouts are short and within-rollout corrections remain largely unexplored. Across twelve tasks spanning discrete (MiniGrid) and continuous control (OGBench), in both clean and stochastic-distractor settings, CIG outperforms or matches prior exploration methods while remaining robust to stochastic distractors. 1 Introduction Reinforcement learning agents in environments with sparse or delayed rewards must explore: they must visit states whose outcomes are not yet predictable from past experience. Intrinsic rewards formalize this pressure by assigning a bonus to transitions that are informative about the environment [ 40 , 32 ] , whether combined with extrinsic reward during task-directed training [ 34 , 6 ] or used as the sole objective during unsupervised pre-training of a world model [ 41 , 14 ] . The intrinsic reward controls which transitions the agent collects and therefore which dynamics the world model learns. In model-based RL the policy is optimised over short imagined rollouts [ 11 , 12 , 13 ] , and two sources of context are available for scoring a transition within such a rollout (Figure 1 ). Lifelong rewards condition on the replay buffer [ 34 , 6 , 35 , 41 ] : they assign full credit to every transition that probes a model gap, regardless of whether the current rollout has already probed it, so an imagined trajectory in a maze can spend every step revisiting the same uncertain corridor junction while neglecting independent frontiers elsewhere. Episodic rewards condition on the rollout prefix [ 16 , 15 ] : they penalise intra-trajectory repetition but treat a familiar region and a genuinely uncertain one identically whenever both are new within the rollout. These methods assume long real episodes that provide a rich prefix for the redundancy signal; we are not aware of an extension to the short imagined rollouts of model-based RL. Existing hybrids combine the two signals through heuristic weights [ 4 , 48 ] or require Gaussian-process dynamics restricted to low-dimensional state spaces [ 29 ] . Trajectory-level information gain provides an alternative that conditions on both the buffer and the prefix simultaneously: its chain-rule decomposition yields per-step rewards in which each term is conditioned on the replay buffer through the trained ensemble and on the rollout prefix through the preceding steps [ 24 ] . Computing these terms, however, requires marginalising over a posterior on model parameters that is intractable for deep networks. We derive the Conditional Information Gain (CIG) reward, a tractable surrogate for this objective. When the posterior over model parameters is approximated by a finite ensemble of one-step predictors, the trajectory-level information gain reduces to a log-determinant over a kernel of pairwise ensemble disagreements (Eq. 6 ). We identify a rank-saturation bottleneck of the naïve surrogate and resolve it through a trace reduction that raises the effective capacity from M{-}1 to \min(T,\,(M{-}1)d) . The Cholesky factorisation of this log-determinant then yields causal per-step rewards: each step’s reward equals its total disagreement minus the portion already explained by preceding steps (Eq. 8 ). CIG plugs into an existing model-based backbone without additional hyperparameters; the only new quantity, the aleatoric scale, is estimated post-hoc from training residuals. Across twelve tasks spanning discrete (MiniGrid) and continuous control (OGBench), in both clean and stochastic-distractor settings, CIG outperforms or matches all baselines while remaining robust to stochastic distractors. We contribute the CIG reward itself (§ 3 ), a per-step intrinsic reward derived from a log-determinant surrogate of trajectory-level information gain that conditions on both the replay buffer and the rollout prefix, together with a rank-saturation analysis that motivates the trace reduction resolving the capacity bottleneck of the finite-ensemble surrogate (§ 3.2 ). We evaluate CIG on twelve tasks spanning discrete and continuous-control domains, in both clean and stochastic-distractor variants, using a reward-free protocol that isolates exploration quality from extrinsic reward exploitation; CIG is the only method in the top tier on every task, with an aggregate normalised IQM exceeding the next-best baseline by a clear margin (§ 4 ). 2 Background 2.1 Setup We consider an agent interacting with a reward-free Markov decision process \mathcal{M}=(\mathcal{S},\mathcal{A},P,\gamma) , where \mathcal{S}\subseteq\mathbb{R}^{d} is the state space, \mathcal{A} the action space, P(s^{\prime}\mid s,a) the unknown transition distribution, and \gamma\in[0,1) a discount factor. At each step the agent samples a_{t}\sim\pi_{\theta}(\cdot\mid s_{t}) , observes s_{t+1}\sim P(\cdot\mid s_{t},a_{t}) , and appends the transition to a replay buffer \mathcal{D} . The agent’s objective is pure exploration: in the absence of any extrinsic reward, it must collect transitions that are maximally informative about the unknown dynamics P . To this end it constructs an intrinsic reward r_{t} and optimises J(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}\!\bigl[\sum_{t\geq 0}\gamma^{t}r_{t}\bigr] . 2.2 Intrinsic Reward Design and Prior Work Figure 1 : Conditioning contexts of intrinsic rewards in model-based RL. The policy plans over short imagined rollouts. Neural network weights w , trained on the replay buffer \mathcal{D} , summarize lifetime uncertainty. Solid states and arrows mark the contexts each reward class conditions on; dashed/faded elements are not used as context. In the model-based setting, the rollout prefix s_{<t} is too short to supply a rich episodic signal. CIG (§ 3 ) instantiates the hybrid form. The reward r_{t} must quantify how informative a transition is, yet informativeness depends on context. Two sources of context are available at step t : the replay buffer \mathcal{D} , which summarizes lifetime experience and determines which dynamics remain uncertain, and the rollout prefix s_{<t} , which records what the current trajectory has already probed. In model-based RL, where the policy optimizes over short imagined rollouts ( T\approx 15 ), the prefix represents what the agent plans to probe rather than what it has already visited in a real episode. Prior intrinsic rewards differ in which of these two contexts they condition on (Figure 1 ); we summarize the three resulting categories below and provide a comprehensive treatment of related work in Appendix D . Lifelong rewards , r_{t}=f(s_{t};\,\mathcal{D}) , score each transition against accumulated experience through prediction error [ 34 , 6 ] , ensemble disagreement [ 35 ] , or density estimates [ 5 ] . Each step is scored independently of all other steps in the rollout, so correlations between disagreement patterns at different time steps are invisible to the reward: correlated steps receive full cumulative credit, a structural redundancy that conditioning on \mathcal{D} alone cannot detect. Episodic rewards , r_{t}=g(s_{t};\,s_{<t}) , penalize within-trajectory repetition [ 39 , 16 ,

Read the complete text