Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought

Zixuan Xie, Xinyu Liu, Rohan Chandra, Shangtong Zhang · 2026

Read Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought by Zixuan Xie, Xinyu Liu, Rohan Chandra, Shangtong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07123.txt.

Introduction

Zixuan Xie and three co-authors give the first theory of how chain-of-thought reasoning boosts in-context reinforcement learning. Agents that adapt to new tasks from context alone, without retraining, do better when they generate reasoning steps. Why that helps had not been explained mathematically.

Working with a linear transformer in a policy evaluation setting, the authors prove chain-of-thought steps act like repeated temporal-difference updates. Evaluation error falls geometrically as reasoning lengthens, then levels at a floor set by context length. They also prove the needed transformer settings are a global best fit for pretraining, explaining how such behavior can emerge.

Catalog record

CREATORZixuan Xie, Xinyu Liu, Rohan Chandra, Shangtong Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07123

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought Zixuan Xie, Xinyu Liu, Rohan Chandra, Shangtong Zhang Abstract In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context. Recent empirical studies further demonstrate that Chain-of-Thought (CoT) generation can amplify this ICRL capability. This paper is the first to provide a theoretical understanding on how CoT interacts with ICRL. We conduct our analysis in a policy evaluation setup with linear Transformer. We prove that with specific Transformer parameters, the CoT generation process is equivalent to repeatedly executing temporal difference learning updates. Additionally, we provide finite sample convergence analysis showing that the policy evaluation error decreases geometrically with CoT length and eventually saturates at a statistical floor determined by the context length. We also prove that the desired Transformer parameters are a global minimizer of the pretraining loss, providing a theoretical understanding on the empirical emergence of those parameters. 1 Introduction Reinforcement learning (RL, 56 ) provides a foundational framework where agents learn sequential decision-making through interaction. A standard paradigm in deep RL consists of training an agent with an RL algorithm on a target problem and deploying the fixed policy within that same domain ( 39 ; 50 ; 51 ) . However, this workflow lacks flexibility against task variations. If the underlying goals or environment dynamics change, the specialized policy often fails, requiring the agent to be retrained from scratch to adapt, which can be computationally expensive. In-Context Reinforcement Learning (ICRL, 27 ; 41 ) mitigates this limitation by enabling adaptation at inference time without parameter updates. In this framework, an agent is pretrained on a diverse distribution of tasks, typically formulated as either policy evaluation or control problems ( 41 ) . Policy evaluation involves estimating value functions, whereas control focuses on searching for optimal policies. At inference time, the network parameters are frozen and the agent conditions on additional context generated within the new task. A standard form of such context is the trajectory of recent interactions, comprising past observations, actions, and rewards ( 27 ) . Empirical evidence ( 27 ; 2 ; 16 ; 46 ; 18 ; 19 ; 34 ; 3 ) shows that performance often improves as the context grows, even on test tasks that deviate significantly from the pretraining distribution. Thus, this improvement is difficult to explain by memorization of the pretraining tasks. It instead suggests an implicit implementation of an internal RL process within the network’s forward pass that leverages the context for adaptation ( 41 ) . We refer to this capability of reinforcement learning at inference time as ICRL. This ICRL capability has been shown to arise under a diverse array of pretraining algorithms, which can be broadly divided into supervised pretraining and reinforcement pretraining. In supervised pretraining ( 27 ; 52 ; 26 ; 29 ; 76 ; 13 ; 32 ) , the network is trained by imitation to match the behavior of a known RL algorithm on curated learning trajectories, so it is less surprising if the forward pass resembles that algorithm. A more intriguing phenomenon arises from reinforcement pretraining, where the agent network can be a sequence model such as a Transformer ( 63 ) and the pretraining algorithm is simply a variant of standard deep RL algorithm ( 15 ; 65 ; 18 ) . Here, the network is trained to output good actions for control tasks or to make accurate value predictions for policy evaluation tasks, yet the resulting forward pass can still behave like an RL algorithm. This ICRL capability is also observed in large language models (LLMs). For example, 53 report that multi-round prompting with scalar rewards can yield consistent inference-time improvement for LLMs in control tasks. Particularly, 53 rely on Chain-of-Thought (CoT, 69 ) generation to amplify LLM’s ICRL capability. Many works have been done to theoretically understand ICRL with most focusing on supervised pretraining. See Section 7 for a more detailed exposition. The volume of works to theoretically understand ICRL with reinforcement pretraining is much thinner. Notably works include 66 ; 67 that study ICRL under a policy evaluation setup. Specifically, 66 prove that with certain parameters, the layer-by-layer forward pass of linear Transformers is equivalent to step-by-step update of temporal difference (TD) learning ( 59 ) , arguably the most powerful algorithm for policy evaluation. 67 further prove that those parameters are a global minimizer of some reinforcement pretraining loss. However, 66 ; 67 do not explain the role of CoT in ICRL, a gap that we shall close in this paper. Specifically, we make three contributions toward closing this gap. First, we constructively prove that under specific parameters, stepwise CoT generation in a linear Transformer is equivalent to iterating TD updates on the context. Second, we establish inference-time convergence guarantees with context constructed from both known and unknown dynamics, showing that the policy evaluation error decreases geometrically up to a statistical floor controlled by the context length. Third, we prove that the desired parameters above is a global minimizer for a TD-based pretraining algorithm and we empirically validate the emergence of those parameters in controlled environments. 2 Background Notation. All vectors are treated as column vectors. We denote the identity matrix in \mathbb{R}^{n\times n} by I_{n} . We use 0_{m\times n} and 1_{m\times n} to denote the zero matrix and the all-ones matrix. Besides, \mathbf{0} and \mathbf{1} are used to denote the zero vector and the all-ones vector respectively, where the dimension is clear from context. Let e_{i} be the standard basis vector for the i -th coordinate. The transpose of a matrix Z is denoted by Z^{\top} . The inner product of two vectors x and y is denoted by \langle x,y\rangle\doteq x^{\top}y . For a symmetric positive semidefinite matrix M , define \|x\|_{M}^{2}\doteq x^{\top}Mx . We use \|\cdot\| to denote the Euclidean norm and \|A\|_{2} the spectral norm. For symmetric M , \lambda_{\min}(M) and \lambda_{\max}(M) denote its extreme eigenvalues. We use \tilde{O}(\cdot) to suppress universal constants and polylogarithmic factors in the problem parameters. For N\in\mathbb{N} , we define [N]\doteq\{0,1,\ldots,N-1\} . 2.1 Reinforcement Learning and Policy Evaluation We consider an infinite-horizon Markov Decision Process (MDP, 5 ) defined by a tuple (\mathcal{S},\mathcal{A},p_{\text{MDP}},r_{\text{MDP}},\gamma,p_{0}) . Here \mathcal{S} is a finite state space, \mathcal{A} is a finite action space, p_{\text{MDP}}(\cdot|s,a) is the transition kernel, r_{\text{MDP}}(s,a) is the reward function, \gamma\in[0,1) is the discount factor, and p_{0} is the initial distribution. In this paper, we focus on the policy evaluation problem, where the goal is to estimate the value function of an arbitrary policy \pi:\mathcal{S}\times\mathcal{A}\to[0,1] . An initial state S_{0} is sampled from p_{0} . At timestep t , given the current state S_{t} , the agent samples an action A_{t} from its policy \pi . Consequently, it obtains a reward R_{t+1} determined by r_{\text{MDP}}(S_{t},A_{t}) and moves to the next state S_{t+1} according to the transition dynamics p_{\text{MDP}}(\cdot|S_{t},A_{t}) . A fixed policy \pi reduces the MDP to a Markov Reward Process (MRP) with transition kernel p(s^{\prime}|s)\doteq\sum_{a\in\mathcal{A}}\pi(a|s)p_{\text{MDP}}(s^{\prime}|s,a) and reward function r(s)\doteq\sum_{a\in\mathcal{A}}\pi(a|s)r_{\text{MDP}}(s,a). In this case, transitions and rewards are determined solely by the current state: S_{t+1}\sim p(\cdot|S_{t}) and R_{t+1}\doteq r(S_{t}) . We focus

Read the complete text