Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning
Minjae KwonUniversity of Virginia [email protected], Amir MoeiniUniversity of Virginia [email protected] Shangtong ZhangUniversity of Virginia [email protected] Lu FengUniversity of Virginia [email protected] · 2026
Read Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning by Minjae KwonUniversity of Virginia [email protected], Amir MoeiniUniversity of Virginia [email protected] Shangtong ZhangUniversity of Virginia [email protected] Lu FengUniversity of Virginia [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25267.txt.
Introduction
Minjae Kwon and colleagues add a safety shield to systems that adapt from past interactions without test-time weight updates. Pretrained safe systems trade reward against safety poorly under new conditions, since the leftover budget acts only through frozen policy settings. Their latent Q-Barrier shield first learns a compact context picture, hidden dynamics, and a cost critic. At deployment it infers context from history and screens candidate actions against predicted cost and remaining budget. Across five benchmarks, it lifts returns in four while holding or cutting cost throughout.
Catalog record
| CREATOR | Minjae KwonUniversity of Virginia [email protected], Amir MoeiniUniversity of Virginia [email protected] Shangtong ZhangUniversity of Virginia [email protected] Lu FengUniversity of Virginia [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85698 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning Minjae KwonUniversity of Virginia [email protected], Amir MoeiniUniversity of Virginia [email protected] Shangtong ZhangUniversity of Virginia [email protected] Lu FengUniversity of Virginia [email protected] Abstract Safe in-context reinforcement learning (ICRL) adapts online from interaction history without test-time parameter updates while controlling episode cost under a safety budget. Under out-of-distribution (OOD) deployment shifts, pretraining-only safe ICRL can give poor reward-safety tradeoffs because the remaining budget affects behavior only through frozen policy conditioning, not an explicit action-level check against predicted future cost. We propose a latent Q-Barrier shield that learns a context representation, latent dynamics, and an ensemble cost critic before deployment. Without parameter updates, the shield infers context from history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. We prove a conditional, error-decomposed barrier-margin result: a Q-Barrier-satisfying action leaves the next latent-budget state with an approximately budget-safe continuation under the learned critic, up to Bellman and latent-prediction errors. Across five safe ICRL benchmarks, the shield improves deployment-time reward-safety tradeoffs over a strong safe-ICRL baseline: after a short context window, it achieves higher return in four of five benchmarks while matching or lowering average episode cost in all five. 1 1 footnotetext: ∗ Equal contribution. 2 2 footnotetext: † Corresponding authors. 1 Introduction Reinforcement learning (RL) [ 55 ] agents deployed in the real world must manage safety constraints such as resource limits, physical damage, or constraint violations during deployment. Constrained Markov decision processes (CMDPs) [ 4 ] formalize this requirement by assigning each transition both a reward and a cost and constraining expected cumulative cost under a user-specified budget. Most safe RL methods enforce this constraint during training, for example through Lagrangian penalties or constrained policy optimization [ 1 , 46 ] . In-context reinforcement learning (ICRL) provides a different adaptation mechanism: a pretrained agent adapts to new tasks at test time without parameter updates by conditioning its policy on an expanding interaction history [ 14 , 63 , 33 , 53 , 42 , 36 ] . Recent work extends this idea to safe ICRL. In particular, 39 formulate safe ICRL as a constrained problem across test episodes and propose SCARED, a Lagrangian pretraining method whose fixed-point analysis characterizes budget-respecting policies under its assumptions. However, existing safe ICRL methods, such as SCARED, mainly encode safety through pretraining objectives or cost-conditioning, rather than through an explicit action-level check against the remaining budget. A frozen ICRL policy may still assign probability to actions whose predicted future cost exceeds the remaining budget, especially in generalization settings where the task and safety structure must be inferred from context. We add a budget-aware shielding layer that evaluates candidate actions against the remaining budget using a latent cost barrier and a pessimistic cost-to-go critic. The shield then filters or softly reweights the policy distribution without deployment-time parameter updates, turning learned safety preferences into an explicit action-level check. This motivates the central question of the paper: can a learned barrier improve a frozen safe-ICRL policy’s reward-safety tradeoff by turning the remaining budget into an explicit action-selection constraint? We answer this question through a Q-Barrier shielding method, a margin-based analysis, and empirical evaluation on deployment generalization tasks. In summary, our contributions are: • Barrier-based runtime shielding for safe ICRL deployment. We introduce a deployment-time shield that reweights a frozen policy using the remaining safety budget and a learned cost-to-go estimate. The shield uses latent representations, latent dynamics, and an ensemble cost critic learned during training, with all parameters fixed at deployment. • Budget-safe continuation analysis. We prove a Q-Barrier margin bound showing that a shielded action preserves next-step latent-budget margin up to Bellman upper-bound and latent-prediction errors. A Q-Barrier-satisfying action thus admits an approximately budget-safe continuation under the learned critic. • Improved in-context adaptation reward-safety tradeoffs. Q-Barrier improves early return and lowers cost after a short context window in four of five benchmarks, with one environment showing a conservative lower-cost/lower-return tradeoff. In budget sweeps, it achieves the highest cumulative return in four of five environments and satisfies the budget across most levels, with violations mostly occurring at near-zero budgets, where even small costs can exceed the budget. 2 Background Markov Decision Processes. We model the interaction with an environment as a finite-horizon Markov decision process (MDP [ 44 ] ). An MDP is defined by a state space \mathcal{S} , an action space \mathcal{A} , a reward function r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} , a transition kernel p(\cdot\mid s,a) , where for each state-action pair (s,a) , p(\cdot\mid s,a) denotes a probability distribution over next states in \mathcal{S} , an initial state distribution p_{0} over \mathcal{S} , and a horizon length T . We denote by T_{k} the terminal timestep of episode k . An agent follows a stochastic policy \pi , modeled as a Markov kernel from states to actions: for each s\in\mathcal{S} , \pi(\cdot\mid s) is a probability distribution over \mathcal{A} . When \mathcal{A} is discrete, \pi(a\mid s) denotes the probability mass of action a . When \mathcal{A} is continuous, \pi(a\mid s) denotes the action density, when it exists, which integrates to one. After starting from S_{0}\sim p_{0} , at time step t , the agent samples A_{t}\sim\pi(\cdot\mid S_{t}) , observes reward R_{t+1}\doteq r(S_{t},A_{t}) , and transitions to S_{t+1}\sim p(\cdot\mid S_{t},A_{t}) . We use k to index episodes and t to index time steps within an episode. For any episode \tau , we define its return as G(\tau)\doteq\sum_{t=1}^{T}R_{t} . The performance of a policy \pi is measured by the expected total rewards J(\pi)\doteq\mathbb{E}_{\tau\sim\pi}[G(\tau)] . Constrained Markov Decision Processes. In addition to the reward function r , a constrained MDP (CMDP [ 4 ] ) involves a cost function c:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{+} with an associated user-given budget \delta . At each time step t , after taking an action A_{t} at a state S_{t} , the agent receives a cost C_{t+1}\doteq c(S_{t},A_{t}) . An episode in a CMDP therefore becomes \tau=(S_{0},A_{0},R_{1},C_{1},S_{1},\dots,S_{T-1},A_{T-1},R_{T},C_{T}). We denote the total cost of an episode \tau by G_{c}(\tau)\doteq\sum_{t=1}^{T}C_{t} , which should, in expectation, remain below the budget \delta . The return-to-go (RTG) is G_{t}(\tau)\doteq\sum_{i=t+1}^{T}R_{i} and the cost-to-go (CTG) is G_{c,t}(\tau)\doteq\sum_{i=t+1}^{T}C_{i} . Given \delta , we define the remaining cost budget at time t as B_{t}\doteq\delta-\sum_{i=1}^{t}C_{i},~B_{0}=\delta. In-Context Reinforcement Learning. ICRL [ 40 ] trains an agent across a distribution of tasks so that, at test time, it can adapt without parameter updates by conditioning on an expanding interaction history. Let \theta_{*} denote the pretrained parameters. During deployment, \theta_{*} is fixed and the policy is written as \pi_{\theta_{*}}(\cdot\mid S_{t}^{k},H_{t}^{k}), where H_{t}^{k} is the context available at time t in episode k . A context consists of previous episodes \{\tau_{j}\}_{j=1,\dots k-1} and the current episode prefix \tau_{k}^{t} , i.e., H_{t}^{k}\doteq(\tau_{1},\ldots,\tau_{k-1},\tau_{k}^{t}). In safe ICRL [ 39 ] , each task is a CMDP, so the context includes both rewards and costs: \tau_{k}^{t}\doteq(S_{0}^{k},A_{0}^{k},R_{1}^{k},C_{1}^{k},S_{1}^{k},\ldots,S_{t-1}^{k},A_{t-1}^{k},R_{t}^{k},C_{t}^{k}). We write the remaining episode budget as B_{t}^{k}\doteq\delta-\sum_{i=1}^{t}C_{i}^{k},~B_{0}^{k}=\delta. Adaptation occurs as \pi_{\theta_{*}}(\cdot\mid S_{t}^{k},H_{t}^{k}) changes with the growing context, while the policy parameters remain fixed. 2.1 Problem Formulation We formulate safe ICRL deployment over K test episodes. All parameters are learned before deployment and fixed at test time. The deployed shielded policy \pi uses the current state S_{t}^{k} , context H_{t}^{k} , and remaining budget B_{t}^{k} for action selection. Our objective