Context Bootstrapped Reinforcement Learning

Saaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang · 2026

Read Context Bootstrapped Reinforcement Learning by Saaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18953.txt.

Introduction

Agashe and co-authors add shown examples to reinforcement learning for tasks with checkable answers. Models often fail to find any good trial runs, so they get little signal to learn from. Their method places a few worked examples before training prompts at random. The chance of help starts high, then fades to zero. The model must learn the pattern, not lean on help at test time. Tests on two model families and five tasks show better success and faster search.

Catalog record

CREATORSaaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42569
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context Bootstrapped Reinforcement Learning Saaket Agashe, Jayanth Srinivasa, Gaowen Liu, Ramana Kompella, Xin Eric Wang Abstract Reinforcement Learning from Verifiable Rewards (RLVR) suffers from exploration inefficiency, where models struggle to generate successful rollouts, resulting in minimal learning signal. This challenge is particularly severe for tasks that require the acquisition of novel reasoning patterns or domain-specific knowledge. To address this, we propose Context Bootstrapped Reinforcement Learning (CBRL) , which augments RLVR training by stochastically prepending few-shot demonstrations to training prompts. The injection probability follows a curriculum that starts high to bootstrap early exploration, then anneals to zero so the model must ultimately succeed without assistance. This forces the policy to internalize reasoning patterns from the demonstrations rather than relying on them at test time. We validate CBRL across two model families and five Reasoning Gym tasks. Our results demonstrate that CBRL consistently improves success rate, provides better exploration efficiency, and is algorithm-agnostic. We further demonstrate CBRL’s practical applicability on Q, a domain-specific programming language that diverges significantly from mainstream language conventions. Correspondence: [email protected], [email protected] Project Website: https://context-bootstrapped-rl.github.io 1 Introduction Reinforcement Learning from Verifiable Rewards (RLVR) ( 9 ) has emerged as a powerful paradigm for post-training Large Language Models (LLMs) on challenging reasoning problems ( 5 ) . In RLVR, the model receives a binary reward purely based on its final output. This learning framework has driven remarkable advances across diverse domains, including mathematical reasoning ( 5 , 22 ) , code generation ( 8 ) , and tool use ( 6 ) . However, RLVR in Large Language Models suffers from a fundamental problem: Exploration Inefficiency ( 10 , 24 , 11 ) . When a model cannot reliably produce correct rollouts during training, it receives minimal learning signal, leading to slow or failed convergence. This challenge is particularly acute in settings where the domain is underrepresented in pretraining data (e.g., domain-specific programming languages) or because the task requires novel reasoning patterns. Therefore, we propose Context Bootstrapped Reinforcement Learning (CBRL) , a novel approach that leverages the In-Context Learning ( 2 ) capabilities of LLMs to address exploration inefficiency in RLVR. The method maintains a small bank of few-shot examples that are dynamically injected into training prompts with decreasing probability as training progresses. Early in training, In-Context examples guide the model toward successful rollouts, providing the learning signal needed to bootstrap capability acquisition. As training advances, the injection probability anneals, encouraging the model to perform independently. We validate CBRL through extensive experiments on two model families (Qwen ( 21 ) and Llama ( 18 ) ) across five Reasoning Gym tasks ( 16 ) : ARC-1D, Word Sorting, Spell Backward, Matrix Manipulation, and Puzzle-24. These tasks require the model to learn specific reasoning patterns. We further evaluate on Q Programming ( 7 ) , a domain-specific language for time-series databases. Unlike mainstream languages, Q features right-to-left evaluation, implicit typing, and a terse array-oriented syntax that diverges from conventions dominant in pretraining corpora. Our results demonstrate that CBRL consistently outperforms the GRPO-only baseline, improving accuracy on all 10 model–environment pairs (Table 1 ), with gains ranging from +1.3% to +22.3%. On Q programming, CBRL improves both test-pass rate ( 27.3\%\rightarrow 43.0\% ) and Pass@1 ( 5.0\%\rightarrow 26.3\% ) over the regular GRPO (Table 2 ). We find that CBRL is algorithm-agnostic , yielding consistent gains when paired with both GRPO ( 15 ) and RLOO ( 1 ) . Under RLOO, Word Sorting improves from 20% to 67% and Puzzle-24 from 23% to 66% (Table 3 ). Training curve analysis reveals that CBRL addresses the exploration inefficiency bottleneck, attaining substantially higher mean reward early in training, consistent with improved exploration from injected few-shot demonstrations (Figure 4 ). Crucially, this advantage persists as the injection probability p_{i} anneals toward zero. Performance does not collapse once demonstrations are removed, indicating that CBRL bootstraps durable behaviors rather than inducing dependence on in-context examples. We also ablate the initial injection probability and find that moderate values around p_{i}=0.5 perform best. Setting p_{i} too high overwhelms the policy’s own exploration, while setting it too low provides insufficient scaffolding to escape reward plateaus. Our main contributions are: • We introduce Context Bootstrapped Reinforcement Learning (CBRL), an algorithm-agnostic approach that leverages in-context learning to address exploration inefficiency in RLVR. • We validate CBRL across five Reasoning Gym tasks and the domain-specific Q programming language, demonstrating consistent gains across diverse tasks and domains. • We provide an in-depth analysis of CBRL, including learning dynamics (training curves), ablations over the injection-probability schedule, and qualitative examples that characterize its usefulness. 2 Context Bootstrapped Reinforcement Learning (a) (b) Figure 1: Overview of the Context Bootstrapped Reinforcement Learning (CBRL) framework. (a) The injection probability annealing schedule. At each timestep t , training batches consist of samples without few shots and samples with few shots , with the proportion governed by p_{i} , which decreases linearly from p_{\mathrm{start}} to p_{\mathrm{end}} . (b) Construction of the two sample types. Samples without exemplars (top) present only the task query followed by an empty assistant turn. Samples with exemplars (bottom) prepend solved few-shot demonstrations as prior user–assistant exchanges before the target query. Context Bootstrapped Reinforcement Learning (CBRL) augments the standard RLVR training process by incorporating in-context examples as temporary scaffolding. The method operates through three core components: a bank of solved examples, a stochastic injection mechanism, and a curriculum schedule that phases out assistance as training progresses. At inference time, the trained policy generates completions without any injected examples, incurring no additional computational cost. Figure 1 provides an overview of the framework. 2.1 Few-Shot Example Bank For each task domain, we construct a bank \mathcal{B} of solved examples, where each e\in\mathcal{B} contains a problem statement q , an optional reasoning trace r , and an answer a . The bank can be constructed from expert demonstrations, stronger model solutions, or hand-crafted instances. 2.2 Stochastic Context Injection At each training step, with probability p_{i} , we sample k examples \{e_{1},\ldots,e_{k}\} from the bank \mathcal{B} and prepend them to the query as conversation pairs. The policy generates completions on this augmented prompt, with rewards computed solely on the generated response. As illustrated in Figure 1 b, samples without exemplars present only the task query followed by an empty assistant turn, while samples with exemplars prepend solved few-shot demonstrations as prior user-assistant exchanges before the target query. This stochastic injection ensures the policy experiences both standalone and ICL-augmented prompts. The mixture serves a critical purpose. If examples appeared on every prompt, the policy might learn to depend on their presence rather than internalize the demonstrated reasoning patterns. By randomly withholding examples, we force the model to attempt problems independently, even during early training. 2.3 Curriculum Schedule The injection probability p_{i} follows a linear annealing schedule: p_{i}=p_{\text{start}}+\frac{t-1}{T-1}(p_{\text{end}}-p_{\text{start}}) (1) where p_{\text{start}} is the initial injection probability (typically 0.5 to 1.0), p_{\text{end}} is the final injection probability (typically 0.0), T is the total training budget, and t is the current training step. This schedule addresses a fundamental tension in learning from demonstrations. Early in training, when the policy is weakest and would otherwise fail to discover correct solutions, examples appear frequently ( p\approx p_{\text{start}} ), providing strong guidance toward effective solution strategies. However,

Read the complete text