When to ASK: Uncertainty-Gated Language Assistance for Reinforcement Learning

Juarez Monteiro 1 , Nathan Gavenski 2 , Gianlucca Zuin 1 and Adriano Veloso 1 · 2026

Read When to ASK: Uncertainty-Gated Language Assistance for Reinforcement Learning by Juarez Monteiro 1 , Nathan Gavenski 2 , Gianlucca Zuin 1 and Adriano Veloso 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02226.txt.

Introduction

Monteiro and colleagues gate language model help for reinforcement learning agents by uncertainty. Their ASK system pairs small language models with trained policies and calls the language model only when Monte Carlo dropout signals high uncertainty. That keeps routine control cheap while bringing world knowledge to unfamiliar transfer tasks. On FrozenLake it shows no gain in domain but navigates transfer tasks robustly, reaching a reward of 0.95. The lesson: neuro symbolic gains need careful orchestration, not mere combination.

Catalog record

CREATORJuarez Monteiro 1 , Nathan Gavenski 2 , Gianlucca Zuin 1 and Adriano Veloso 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55342
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02226

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When to ASK: Uncertainty-Gated Language Assistance for Reinforcement Learning Juarez Monteiro 1 , Nathan Gavenski 2 , Gianlucca Zuin 1 and Adriano Veloso 1 Abstract Reinforcement learning (RL) agents often struggle with out-of-distribution (OOD) scenarios, leading to high uncertainty and random behavior. While language models (LMs) contain valuable world knowledge, larger ones incur high computational costs, hindering real-time use, and exhibit limitations in autonomous planning. We introduce Adaptive Safety through Knowledge (ASK), which combines smaller LMs with trained RL policies to enhance OOD generalization without retraining. ASK employs Monte Carlo Dropout to assess uncertainty and queries the LM for action suggestions only when uncertainty exceeds a set threshold. This selective use preserves the efficiency of existing policies while leveraging the language model’s reasoning in uncertain situations. In experiments on the FrozenLake environment, ASK shows no improvement in-domain, but demonstrates robust navigation in transfer tasks, achieving a reward of 0.95 \mathbf{0.95} . Our findings indicate that effective neuro-symbolic integration requires careful orchestration rather than simple combination, highlighting the need for sufficient model scale and effective hybridization mechanisms for successful OOD generalization. I Introduction Generalization is a vital requirement for learning systems, as it enables them to function beyond their training data. In learning, these systems assume that the observed data are drawn from an independent and identically distributed distribution, and that this distribution holds approximately during evaluation and testing. Yet, agentic applications do not exhibit this behavior. During testing, even if the initial state is included in the training process, the sequential nature of episodes may push the agent out of distribution (OOD) [ 13 ] . Generalizing to OOD states is challenging for agents because they lack prior information about how to act in these states. When predicting in OOD situations, agents exhibit high uncertainty and may behave randomly or be biased. Thus, ensuring that agents behave appropriately in OOD states is also a vital component to develop safe and trusted systems. Some works [ 10 , 6 , 2 ] aim to ensure generalization by employing expert systems or incorporating prior knowledge to guide agent behavior in OOD states. Recent advances in language models (LM) offer a promising avenue: pre-trained on vast corpora, these models encode rich world knowledge and common-sense reasoning that can potentially guide agents when encountering OOD states. However, integrating large LM (LLMs) into real-time decision-making systems poses computational challenges. While planners offer a safe alternative [ 16 , 9 ] , they require careful manual encoding of domain knowledge, limiting their adaptability to new tasks. Smaller LMs, small to medium-sized models, offer an elegant trade-off: they retain much of the world knowledge and reasoning of larger models while being computationally efficient enough for real-time decision-making [ 4 ] . Unlike planners, LMs provide flexible guidance across diverse scenarios without requiring task-specific adjustments, leveraging existing trained policies to improve practical utility across different agents and domains. Approaches to improving generalization can be broadly categorized into intrinsic and extrinsic methods. Intrinsic methods, such as domain randomization, architectural innovations, or curriculum learning, modify the training process to encourage broader generalization, but require retraining agents with potentially larger networks or additional data [ 13 ] . Extrinsic methods, conversely, augment existing policies at inference time without requiring retraining or additional training data. In this work, we explore an extrinsic approach that integrates LMs with already-trained PPO agents to improve generalization in OOD settings, which we call Adaptive Safety through Knowledge (ASK) 1 1 1 Data and Code Availability. Supplementary material is available at https://kclpure.kcl.ac.uk/ws/portalfiles/portal/368422362/appendix.pdf . The code is available at https://github.com/jrzmnt/ask-rl . . Our method leverages the semantic understanding and world knowledge encoded in LMs to guide reinforcement learning (RL) agents when they encounter unfamiliar states, combining the adaptability of RL with the implicit knowledge of language models without requiring retraining or architectural modifications. II Problem Formulation Reinforcement learning is a problem where a policy {\pi} learns parameters {\theta} via interacting with the environment and its reward signal r [ 21 ] . In RL, the reward signal r:S\times A\mapsto\mathbb{R} , where S is the state space and A is the action space, allows the agent {\pi_{\theta}} to learn the probability of each action a\in A given the current state s\in S by maximizing the immediate reward. In this setting, agents typically do not observe the entire state space, resulting in a situation in which they lack state information and may perform highly uncertain actions. To better formulate this change in unobserved states, we depart from the initial RL formulation and adopt one in which different contexts may not only alter how states are represented but also modify the transition functions that govern the environment’s dynamics. II-A Contextual Markov Decision Process We formulate our problem as a Contextual Markov Decision Process (CMDP), a tuple \mathcal{M}=\langle S,A,O,r,T,C,\phi\rangle , where O is the observation space, r is the immediate reward function r:O\times A\mapsto\mathbb{R} , T is the transition function T:O\times A\mapsto O , C is the set of contexts, and \phi is a projection function \phi:S\times C\mapsto O [ 13 ] . Note that in CMDPs, the policy only observes the state produced by \phi , that the current context c\in C remains the same in an episode, and the reward function r now operates under observations and not states. In other words, the policy perceives only one context c at a time in which all states s\in S are projected into it \left\{\phi(s,c)\mid s\in S\right\} . In this formulation, the context serves as the seed, ID, or parameter vector that determines the current environment. Hence, it should not change within an episode; it should change only between episodes. More importantly, the context distribution p(c) defines the training, evaluation, and testing collections of each task or environment instance. Most often, this distribution remains uniform through all collections; however, this is not true for all environments. Finally, we do not assume that the policy observes the context. That is, the policy does not perceive its context. Since two different contexts (c,c^{\prime}) might map two different states (s,s^{\prime}) into the same observation (\phi(s,c)=\phi(s^{\prime},c^{\prime})) , it is not always the case that the policy can acknowledge in which context it is in. II-B Uncertainty in Neural Networks Measuring uncertainty helps researchers assess the extent to which model predictions carry weight given their prior experience. The two most common types of uncertainty in machine learning are: (i) epistemic , which is inherent to models, caused by a lack of training data, hence reducible with more data; and (ii) aleatoric , caused by inherent noise or ambiguity in data, thence irreducible with more data [ 7 ] . Epistemic uncertainty at an observation o\in O is high for a previously unseen o , and decreases as o becomes part of the training set [ 12 ] . 14 [ 14 ] show that this behavior conforms with using mutual information in Bayesian models and deep ensembles, and 18 [ 18 ] as feature-space density in deterministic models as surrogates for epistemic uncertainty. Aleatoric uncertainty at an observation o\in O is high for ambiguous or noisy samples [ 7 ] . For example, if multiple labels are observed at o , the aleatoric uncertainty will be high,

Read the complete text