Not all uncertainty is alike: volatility, stochasticity, and exploration

Payam Piray · 2026

Read Not all uncertainty is alike: volatility, stochasticity, and exploration by Payam Piray (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19215.txt.

Introduction

Payam Piray shows two kinds of uncertainty push search in opposite directions, though both raise posterior uncertainty. Volatility means hidden reward states drift; stochasticity means outcomes are seen through noise. Extending the Gittins index to Gaussian bandits, he proves volatility boosts search while stochasticity suppresses it. He also derives CAUSE, a closed form bonus that inherits the same pattern. CAUSE beats standard methods in noisy mixed settings and a Gittins policy whose rested optimality fails in restless settings. The work predicts faulty noise inference flips search, with psychiatric implications.

Catalog record

CREATORPayam Piray
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66998
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Not all uncertainty is alike: volatility, stochasticity, and exploration Payam Piray Abstract Adaptive decision-making in biological and artificial intelligence requires balancing the exploitation of known outcomes with the exploration of uncertain alternatives. Although prior work suggests that uncertainty generally promotes exploration, it has typically treated distinct sources of environmental uncertainty as equivalent. We consider environments with latent reward states that drift over time (volatility) and are observed through noisy outcomes (stochasticity). Both increase posterior uncertainty, yet we show they drive optimal exploration in opposite directions: volatility enhances it, stochasticity suppresses it. We establish this asymmetry formally by extending the Gittins index framework to Gaussian state-space bandits with latent dynamics. We further derive Cause-Aware Uncertainty-Sensitive Exploration (CAUSE), a closed-form exploration bonus obtained via control-as-inference that inherits the same monotonicities. CAUSE outperforms standard exploration strategies in environments with heterogeneous noise structure, and also improves on a Gittins-per-arm policy whose rested-bandit optimality does not transfer to restless settings. Learning and exploration are governed by the same noise-inference asymmetry, and the framework predicts that pathological noise inference produces reversed rather than merely impaired exploration, with implications for computational accounts of psychiatric conditions. 1 Introduction Across computational accounts of exploration, a recurring principle is that uncertainty drives behavior: greater uncertainty, more exploration. This logic underlies a broad family of exploration strategies: upper confidence bound (UCB) methods explore by acting optimistically under uncertainty [ 2 , 18 ] ; Thompson sampling explores by sampling from the posterior over rewards [ 34 ] ; information-directed sampling balances expected regret against expected information gain [ 31 ] ; and neuroscience work links exploration to neural and behavioral signatures of uncertainty [ 38 , 12 , 8 , 10 ] . In each case, more uncertainty prescribes more exploration. We show that this prescription is incomplete. Not all uncertainty is alike, and some forms of uncertainty should decrease exploration. Specifically, we consider environments with latent reward states that change over time (volatility) and are observed through noisy outcomes (stochasticity). Both volatility and stochasticity increase posterior uncertainty about rewards, and standard methods such as UCB and Thompson sampling treat them equivalently, prescribing more exploration in both cases. We prove that they drive optimal exploration in opposite directions: volatility enhances exploration, while stochasticity suppresses it. The intuition is that uncertainty matters for exploration only insofar as it reflects potential information gain. Volatility creates information gain: latent states are changing, so new observations reveal something new. Stochasticity destroys it: observations are noisy, so each sample is less informative. An agent that simply explores more whenever it is more uncertain conflates these two cases, exploring vigorously precisely when observations are least useful. We establish this result formally by extending the Gittins index framework from independent and identically distributed (iid) Gaussian bandits to Gaussian state-space bandits, where each arm has a latent reward state that evolves with volatility and is observed with stochasticity. In this richer setting, we prove that the optimal exploration bonus, defined as the value of information that justifies pulling an arm beyond its expected reward, is monotonically increasing in volatility and decreasing in stochasticity. This result follows from the structure of the Gittins retirement problem and requires no approximations. The Gittins index is provably optimal only for rested bandits. In restless settings, where unselected arms also evolve, Gittins loses its optimality guarantee and the truly optimal policy is intractable in general. To obtain a practical exploration strategy, we turn to the control-as-inference framework [ 19 , 35 ] , which casts action selection as posterior inference in a probabilistic graphical model. Within this framework, we derive Cause-Aware Uncertainty-Sensitive Exploration (CAUSE), a closed-form index policy for Gaussian state-space bandits. CAUSE decomposes cleanly into exploitation and exploration components, and its exploration component inherits the same qualitative dependence on volatility and stochasticity established by the Gittins analysis. To our knowledge, this is the first application of control-as-inference to derive an exploration index for restless bandits. These opposing effects parallel a known result in the learning literature. Optimal learning rates increase with volatility and decrease with stochasticity, because the same information-theoretic logic applies: volatility makes new observations more relevant, stochasticity makes them less reliable [ 26 ] . The parallel is not coincidence: both are normative properties of the same generative model. This unified view yields a testable prediction about pathological behavior. Volatility and stochasticity must be jointly inferred from observations, since both increase experienced noise and can only be disentangled by explaining away. A priori insensitivity to one source leads to systematic misattribution of the other [ 26 ] : an agent with hyposensitive stochasticity priors sees the world as more volatile than it is, and vice versa. In the learning setting, such miscalibration produces reversed learning-rate modulation, a pattern observed in human behavior [ 27 ] . We show that the analogous failure in exploration produces reversed exploratory behavior: stochasticity-blind agents over-explore precisely when observations are least informative, while volatility-blind agents under-explore precisely when the environment is changing fastest. Miscalibrated noise inference has been proposed as a candidate mechanism in several psychiatric conditions, making these reversals a concrete behavioral target for clinical investigation. Our contributions are: (i) we extend the Gittins framework from iid Gaussian bandits to Gaussian state-space bandits with latent dynamics, proving that the optimal exploration bonus increases monotonically with volatility and decreases monotonically with stochasticity; (ii) within the control-as-inference framework, we derive CAUSE, a closed-form index policy for Gaussian state-space bandits, which to our knowledge is the first closed-form index for restless bandits with continuous-state Gaussian dynamics and the first application of control-as-inference to derive such an index; (iii) we show that the opposing effects of volatility and stochasticity on exploration parallel those on learning, both following from the normative properties of the same generative model; (iv) empirically, CAUSE outperforms Thompson sampling, UCB, predictive sampling, and a numerical Gittins-per-arm baseline across volatility and stochasticity regimes, and matches the optimal Gittins reference in the rested limit; (v) lesioned inference produces reversed, not merely impaired, patterns of exploration, with the direction of reversal diagnostic of which latent inference has failed. Code will be released upon publication. 2 Related work Exploration and uncertainty in neuroscience. Behavioral and neural studies have linked exploratory behavior to uncertainty signals, including relative and total uncertainty [ 38 ] , expected and unexpected uncertainty [ 7 ] , and information value [ 8 , 11 ] . These accounts generally predict that exploration increases with uncertainty, without distinguishing between the sources of that uncertainty. Our framework predicts that volatility and stochasticity should have opposing effects on exploratory behavior, and that selective failures of inference about these quantities should produce characteristic reversals rather than uniform deficits. To our knowledge, these qualitative predictions have not been tested experimentally. Volatility, stochasticity, and adaptive learning. Adaptive learning under uncertainty has been modeled using hierarchical Bayesian frameworks that track volatility [ 4 , 22 ] or detect change-points [ 24 , 23 ] . These approaches typically estimate one source of uncertainty while treating the other as known, overlooking the computational challenge of their simultaneous estimation. 26 introduced a framework for jointly estimating volatility and stochasticity, showing that although both increase outcome variance, they can be distinguished through temporal structure and exert

Read the complete text