Reflective Context Learning: Studying the Optimization Primitives of Context Space
Nikita Vassilyev, William Berrios, Ruowang Zhang, Bo Han, Douwe Kiela, Shikib Mehri · 2026
Read Reflective Context Learning: Studying the Optimization Primitives of Context Space by Nikita Vassilyev, William Berrios, Ruowang Zhang, Bo Han, Douwe Kiela, Shikib Mehri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03189.txt.
Introduction
Vassilyev and colleagues treat learning from context as an optimization problem, not a bag of tricks, and unify it as Reflective Context Learning. Reflection turns past trajectories and current context into a steering signal, like a gradient, while mutation applies that update to future behavior. Adding classical tools such as batching, failure replay, and grouped rollouts improves strong baselines across three benchmarks.
Catalog record
| CREATOR | Nikita Vassilyev, William Berrios, Ruowang Zhang, Bo Han, Douwe Kiela, Shikib Mehri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77634 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reflective Context Learning: Studying the Optimization Primitives of Context Space Nikita Vassilyev, William Berrios, Ruowang Zhang, Bo Han, Douwe Kiela, Shikib Mehri Abstract Generally capable agents must learn from experience in ways that generalize across tasks and environments. The fundamental problems of learning, including credit assignment, overfitting, forgetting, local optima, and high-variance learning signals, persist whether the learned object lies in parameter space or context space. While these challenges are well understood in classical machine learning optimization, they remain underexplored in context space, leading current methods to be fragmented and ad hoc. We present Reflective Context Learning (RCL), a unified framework for agents that learn through repeated interaction, reflection on behavior and failure modes, and iterative updates to context. In RCL, reflection converts trajectories and current context into a directional update signal analogous to gradients, while mutation applies that signal to improve future behavior in context space. We recast recent context-optimization approaches as instances of this shared learning problem and systematically extend them with classical optimization primitives, including batching, improved credit-assignment signal, auxiliary losses, failure replay, and grouped rollouts for variance reduction. On AppWorld, BrowseComp+, and RewardBench2, these primitives improve over strong baselines, with their relative importance shifting across task regimes. We further analyze robustness to initialization, the effects of batch size, and the impact of allocating stronger or weaker models to different optimization components. Our results suggest that learning through context updates should be treated not as a set of isolated algorithms, but as an optimization problem whose mechanisms can be studied systematically and improved through transferable principles. † † footnotetext: Code available at https://github.com/nvassilyev/RCL 1 Introduction Generally capable agents must learn from experience in ways that generalize across tasks and environments. No fixed model or policy, however capable at training time, can anticipate every environment, user preference, or edge case it will encounter in deployment, and adaptation from online experience is a prerequisite for robust, general-purpose agents ( 33 ; 8 ) . The standard approach to adaptation is to update model weights through gradient-based optimization, but this requires expensive retraining, risks catastrophic forgetting, and is difficult to apply continuously during agent operation. An increasingly viable alternative is to optimize the agent’s context instead: the interpretable artifacts that shape behavior at inference time, such as prompts, structured playbooks, persistent memory, learned tools, and behavioral guidelines ( 23 ) . Context can be optimized iteratively — through reflection on execution experience rather than backpropagation over a loss surface — without modifying the underlying model. These updates require no gradient computation, can be applied continuously as the agent accumulates experience, and produce human-readable artifacts that can be inspected and debugged. As models become stronger reasoners and more faithful instruction followers, context-space optimization becomes both more effective and more significant: an optimizer that reasons about agent behavior can produce increasingly meaningful, targeted revisions to the agent’s operational policy ( 36 ; 17 ) . The fundamental difficulties of learning do not disappear when the learned object moves from weights to context. Local optima and overfitting under greedy improvement ( 24 ) , high-variance updates from limited or noisy samples ( 37 ; 10 ) , catastrophic forgetting from sequential updates ( 22 ; 15 ) , and inefficient use of sparse, delayed feedback ( 33 ) are properties of iterative optimization under partial information, not of any particular optimization medium. As we demonstrate, these failure modes arise equally in discrete context spaces: a playbook rewritten from one anomalous failure oscillates rather than converges, a fix for a new edge case overwrites a previously mastered strategy, and a reflector that reasons over entire trajectories rather than critical decisions wastes its signal on noise. The classical remedies developed for these problems — batching for variance reduction, replay for forgetting, momentum for stability, and structural regularization for generalization ( 14 ; 30 ; 3 ) — are motivated by properties of learning itself, not by properties of any particular optimization substrate. In parameter-space optimization, the gradient is the mechanism that converts data and a loss signal into a directed update. In context-space learning, this role is played by reflection : reasoning over execution trajectories and the current context to diagnose what should change and why. This reflective operation — producing a directional signal from (context, trajectory, outcome) — is what distinguishes learning from search in context space, just as computing gradients distinguishes gradient descent from random search in weight space. When we refer to optimization primitives in this paper, we mean mechanisms that improve the quality, stability, or efficiency of this reflective update process. A growing number of methods have begun developing variants of a shared loop — attempt tasks, reflect on outcomes, update context — introducing increasingly sophisticated mechanisms along the way. These include batched textual gradients ( 28 ) , modular credit assignment over computation graphs ( 41 ) , persistent structured memory with curation rather than compression ( 42 ; 34 ) , verbal self-critique as a cross-episode learning signal ( 31 ) , grouped rollouts with contrastive advantage estimation ( 5 ) , Pareto-aware evolutionary search ( 1 ) , historical feedback retention as optimizer state ( 38 ) , and sampling-based momentum for textual gradient descent ( 7 ) . Yet these methods were introduced over a period of rapid change in the field’s underlying infrastructure. The models available to early methods were fundamentally different from those available today, as were prevailing prompting practices, context representations, and evaluation benchmarks. This makes it difficult to isolate the contribution of a learning primitive from the effects of model capability, prompt engineering conventions, and task difficulty. Good ideas may appear to fail because the models of their era couldn’t execute them, and implementation choices that were necessary workarounds for weaker models may persist in newer methods without re-examination. The problem is compounded by a regime shift. Early prompt optimization was largely concerned with conjuring the right sequence of tokens to elicit latent capabilities from models that were imperfect instruction followers ( 28 ; 45 ) . In that setting, the optimization problem was closer to black-box function optimization than to learning: the model already possessed the relevant knowledge, and the challenge was finding wording that reliably surfaced it. The potential for context-space learning is now fundamentally different. Current models are generally capable, sophisticated instruction followers with strong reasoning abilities, which means context updates can do more than optimize phrasing — they can encode genuine strategies, learned from ground-truth labels and reward signals, that transfer across task instances. The challenge shifts from eliciting latent knowledge to learning new knowledge: abstracting multi-step strategies from execution experience on tasks like AppWorld ( 35 ) and BrowseComp+ ( 6 ) . In this regime, the optimization bottleneck moves from search breadth to reflection quality, and the question of which optimization primitives matter, and how they compose, becomes urgent. These observations motivate treating context-space adaptation not as a collection of isolated algorithms but as a single optimization problem that can be studied systematically. We formalize this view as Reflective Context Learning ( RCL ), a framework centered on reflection as the core learning mechanism. RCL constrains the design space to the setting we believe is most practically