Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi · 2026

Read Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs by Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21198.txt.

Introduction

Embodied language models repeat mistakes because they cannot reflect on failures. Yining Hong and six co-authors give them reflection in this 2026 arXiv paper with Reflective Test-Time Planning. Reflection-in-action generates and scores candidate actions with internal reflection before acting, while reflection-on-action updates the reflection model and policy from external feedback after acting.

Retrospective reflection re-evaluates earlier decisions with hindsight for proper long-horizon credit assignment. Tests on a new long-horizon household benchmark and a cupboard-fitting benchmark show significant gains, with zero-shot generalization to photorealistic environments and real-robot trials on a Franka Panda arm. Ablations show the two reflection modes depend on each other, and retrospective reflection beats step-wise feedback at lower cost.

Catalog record

CREATORYining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21198

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Jiajun Wu, Yejin Choi Abstract Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: reflection-in-action , where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and reflection-on-action , which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with ablative studies validating the complementary roles of reflection-in-action and reflection-on-action. Qualitative analyses, including real-robot trials, highlight behavioral correction through reflection. Website: https://reflective-test-time-planning.github.io Code: https://github.com/Reflective-Test-Time-Planning/Reflective-Test-Time-Planning Figure 1. Conceptual overview of Reflective Test-Time Planning. The agent (a) receives a long-horizon task, (b) performs reflection-in-action by internally simulating and scoring candidate actions, (c) performs reflection-on-action by updating its beliefs and decision systems based on execution outcomes, and (d) conducts retrospective reflection to revise earlier decisions with hindsight. “Error isn’t simple darkness, it sheds a light of its own.” — Kathryn Schulz, Being Wrong 1 Introduction Embodied LLMs ( 41 ; 15 ; 10 ) equip agents with task planning abilities, but they remain brittle static oracles that cannot learn from failures, turning deployment into independent trials of repeated mistakes rather than accumulated experience. Humans, in contrast, are natural reflective practitioners. Drawing on Schön’s framework about reflective planning ( 28 ) , humans fluidly alternate between two modes of reflection: through reflection-in-action , we engage in internal simulation, questioning whether our planned approach will actually work given what we currently understand; through reflection-on-action , we use the actual outcomes to reshape both our beliefs about the environment and our strategies for acting within it. An illustrative example of these reflection modes is shown in Figure Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs . This bidirectional flow allows us to learn not only from outcomes, but also from the very process of engaging with an uncertain world. Current approaches, however, capture at best a superficial version of one mode while neglecting the other. One line of work ( 30 ; 21 ) uses LLM-based verbal reflection, generating natural-language critiques of past behavior to condition future actions. While this enables reflection-on-action at the level of reasoning traces, reflections are stored only as contextual text and do not update the underlying decision process, making their effects transient and prone to repetition under distributional shift. A second line of work ( 38 ; 5 ) relies on internal world models to guide action selection in embodied environments. These approaches support reflection-in-action through anticipated outcomes, but typically assume fixed, pretrained dynamics models that may be wrong in ways only revealed during execution. To operationalize both reflection modes in embodied settings, we introduce Reflective Test-Time Planning, a framework that seamlessly unifies reflection-in-action and reflection-on-action for embodied agents during test-time deployment. Concretely, the framework employs three embodied LLMs during deployment: an action generation model \pi_{\theta} , an internal evaluator V_{\phi_{i}} , and an external evaluator V_{\phi_{e}} . During reflection-in-action, the agent samples N candidate actions via high-temperature sampling, uses V_{\phi_{i}} to generate internal reflections scoring each candidate, then executes the highest-scoring action. After execution, V_{\phi_{e}} generates an external reflection, providing an immediate, language-based evaluation of what happened and why. This immediate external reflection grounds the agent’s beliefs in reality, but remains inherently local—it only evaluates consequences visible at the next timestep. Yet many embodied failures are non-local: an action that appears successful may later block progress, and a seemingly suboptimal action may enable future success. To address this temporal credit assignment problem, we introduce retrospective reflection, where V_{\phi_{e}} periodically re-evaluates earlier decisions with hindsight (e.g., at room transitions or after repeated failures). These hindsight assessments provide self-supervised signals at deployment time, enabling two forms of test-time training: (1) policy gradient for \pi_{\theta} to favor actions that score well under hindsight, and (2) supervised learning for V_{\phi_{i}} to anticipate what hindsight will reveal. Because these updates revise not only the action policy but also the predictive assumptions behind it, the process constitutes a form of double-loop learning ( 1 ) , in which agents learn not merely from outcomes but from diagnosing and correcting the underlying causes of their errors. We evaluate our approach on two embodied benchmarks that we design to stress error-driven adaptation: (1) a Long-Horizon Household benchmark that requires failure recovery during multi-step planning across rooms, and (2) a controlled MuJoCo Cupboard Fitting benchmark that isolates geometric placement failures. Our framework achieves large gains over reflective language, RL and world-model baselines. Ablations indicate that improvement emerges only when both reflection-in-action and reflection-on-action take place, and when both action policy and internal reflection model are updated during deployment. Qualitative analyses, including preliminary real-robot trials, show that reflection reduces repetitive failure modes in practice. 2 Related Works Test-Time Adaptation (TTA) and Learning enables models to adjust to distribution shifts during inference without source data ( 32 ; 33 ; 20 ) . Early methods minimize prediction entropy, with Tent ( 33 ) updating batch-norm parameters online and later work adding calibrated objectives ( 26 ; 34 ) . Parameter-efficient tuning further boosts TTA: LoRA ( 11 ) enables low-rank weight updates with little memory ( 16 ) , while bias-only tuning ( 3 ) offers alternative efficiency–accuracy trade-offs. Recent extensions operate on hidden state representations ( 31 ) , supporting long-context memory. For embodied settings, continual learning frameworks ( 17 ; 23 ) demonstrate viability in manipulation and navigation ( 8 ; 22 ) . We adapt models at test time via self-supervised reflective signals extracted from the agent’s own verbal assessments. Multimodal Embodied Large Language Models couple visual perception with language understanding to enable embodied planning. Recent foundation models leverage large-scale robotic datasets for zero-shot generalization ( 41 ; 15 ; 2 ) , with RT-2 transferring web knowledge and OpenVLA offering open-source support across heterogeneous embodiments. 3D spatial understanding has become central, spanning point clouds ( 9 ) , 3D patches ( 40 ) , and lightweight point cloud injection ( 18 ) . Further extensions incorporate multisensory interaction ( 10 ) , generative world models for manipulation ( 38 ) , and long-term spatial–temporal embodied memory ( 12 ) . Architectural advances explore interleaved multimodal instructions ( 4 ) and chain-of-thought reasoning ( 36 ; 24 ) for improved task decomposition. In contrast, we view deployment as a learning phase: instead of acting as a fixed policy, our embodied multimodal LLM reflects on its own actions and updates itself via test-time training. Figure 2: Method overview. (a) Reflection-in-action : multiple candidate actions are generated and scored by an internal reflection LLM prior to execution. (b) Reflection-on-action : iteratively invoked when working memory hits K

Read the complete text