Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models

Meng Zheng, Samhita Marri*, Anwesa Choudhuri, Benjamin Planche, Zhongpai Gao, Van Nguyen Nguyen, Terrence Chen, Girish Chowdhary, Ziyan Wu, [0.3em] United Imaging Intelligence, Boston, MA, USA · 2026

Read Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models by Meng Zheng, Samhita Marri*, Anwesa Choudhuri, Benjamin Planche, Zhongpai Gao, Van Nguyen Nguyen, Terrence Chen, Girish Chowdhary, Ziyan Wu, [0.3em] United Imaging Intelligence, Boston, MA, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08434.txt.

Introduction

Meng Zheng, Samhita Marri, Anwesa Choudhuri, Benjamin Planche, and coauthors teach robot policies from their own failures. Vision-language-action models are usually cloned from successful demonstrations only. Small execution errors then compound during a rollout. The robot drifts somewhere it cannot recover from.

Their AFIL framework has a pretrained policy generate failure rollouts online. That spares hand-designed failure modes and human rescue demonstrations. Two action generators train together on a shared vision-language backbone. One handles success, and one handles failure. At sampling time, the failure generator steers actions away from bad modes. The method targets diffusion and flow based policies.

Catalog record

CREATORMeng Zheng, Samhita Marri*, Anwesa Choudhuri, Benjamin Planche, Zhongpai Gao, Van Nguyen Nguyen, Terrence Chen, Girish Chowdhary, Ziyan Wu, [0.3em] United Imaging Intelligence, Boston, MA, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models Meng Zheng, Samhita Marri*, Anwesa Choudhuri, Benjamin Planche, Zhongpai Gao, Van Nguyen Nguyen, Terrence Chen, Girish Chowdhary, Ziyan Wu, [0.3em] United Imaging Intelligence, Boston, MA, USA Abstract Vision-language-action (VLA) models provide a promising paradigm for scalable robotic manipulation, yet their reliance on success-only behavioral cloning leaves them brittle; lacking corrective training signals, minor execution errors rapidly compound into unrecoverable, out-of-distribution failures. To address this limitation, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies. AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design or human-in-the-loop recovery. It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone, enabling efficient failure-aware policy learning with limited parameter overhead. During sampling, the failure generator adaptively steers action generation away from failure-prone regions and toward more reliable success modes, with guidance strength determined by the per-diffusion-step distance between success and failure distributions. Experiments across in-domain and out-of-domain robotic manipulation tasks, covering both short- and long-horizon settings, show that AFIL consistently improves task success rates and robustness over existing VLA baselines, demonstrating its effectiveness, efficiency, and generality. 1 1 footnotetext: This work was carried out during the internship of Samhita Marri at United Imaging Intelligence, Boston, MA 2 2 footnotetext: Corresponding author. 1 Introduction Robotic manipulation has long been a central challenge in robotics and artificial intelligence, requiring agents to perceive complex environments, reason over long horizons, and execute precise control under uncertainty. Conventional learning-based manipulation policies, including behavior cloning and reinforcement learning (RL), have achieved notable success in structured settings, but often require task-specific reward design, extensive interaction data, and careful environment engineering, limiting their scalability and generalization to diverse real-world tasks. Recently, vision-language-action (VLA) models have attracted growing attention in robotics ( 34 ; 3 ; 20 ; 28 ; 4 ; 50 ; 48 ) , particularly following the release of OpenVLA ( 22 ) , which demonstrated the feasibility of leveraging large-scale pretrained vision-language representations for embodied decision-making. Figure 1 : Overview of the proposed Adaptive Failure-Informed Learning (AFIL) pipeline. AFIL uses online-generated failure trajectories as adaptive negative guidance, steering action generation away from failure-prone regions and toward more reliable success modes. Despite their promise, VLA models often struggle to generalize during deployment, as training data typically consists primarily of teleoperated, success-only trajectories collected in confined environments ( 22 ; 32 ; 21 ; 30 ) . Just as a smooth sea never made a skilled sailor, a smooth demonstration never made a skilled robot . Such data can cause policies to overfit to narrow behavioral manifolds and lack corrective signals for recovering from suboptimal or unseen states, resulting in brittle execution at inference time. Recent work has therefore highlighted the value of learning from failure data ( 29 ; 25 ; 19 ; 9 ) , showing that unsuccessful trajectories can improve robustness and task success rates. However, existing approaches often rely on handcrafted or predefined failure modes; for example, 9 manually specifies categories such as incomplete grasp, slip, and translation failures for data generation. These designs require domain expertise and may fail to capture the diversity of realistic deployment-time failures. Moreover, recovery often depends on task-specific heuristics or real-time human intervention ( 25 ; 9 ; 42 ; 46 ) , limiting scalability and generalization to unseen tasks or embodiments. Most VLA policies adopt diffusion-based ( 17 ; 4 ) or flow-based ( 26 ) generative architectures ( 22 ; 3 ; 20 ; 4 ) , owing to their ability to model multimodal action distributions and capture complex dependencies between perception and control. However, since most existing diffusion-/flow-based VLA models are trained only on successful demonstrations, sampling from the learned success-conditioned distribution may be insufficient for robust deployment, as the policy lacks explicit signals to avoid nearby failure-prone actions. In image generation, guided diffusion methods such as classifier guidance (CG) ( 7 ) and classifier-free guidance (CFG) ( 18 ; 36 ; 5 ; 12 ) improve conditional generation by steering samples toward desired semantic attributes. Negative prompting further extends this idea by repelling samples away from undesirable regions of the generation space ( 13 ; 39 ; 23 ) . This naturally motivates an analogous strategy for VLA learning: treating failure trajectories as negative guidance to encourage the policy to avoid action distributions associated with unsuccessful outcomes. Directly applying negative guidance to VLAs, however, is non-trivial ( 33 ; 35 ) . Unlike image generation, robotic action generation is sequential and must produce temporally consistent actions conditioned on evolving observations. In addition, conventional negative guidance methods often rely on separate models for desired and undesired distributions, which is computationally expensive for VLA systems built on large vision-language model (VLM) backbones. These challenges motivate an efficient, unified failure-informed guidance mechanism tailored to diffusion- and flow-based VLA policies. To address these challenges, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework for diffusion- and flow-matching VLA models that dynamically guides action generation using failure examples ( c.f . Figure 1 ). AFIL learns from mixed success and failure data by jointly training Dual Action Generator (DAG)-VLA for successful and failed behaviors while sharing a VLM backbone for visual perception and semantic grounding. During sampling, AFIL iteratively queries both success and failure action generators and uses the predicted failure actions as adaptive negative guidance, steering the generative process away from undesirable regions of the action distribution. Extensive experiments on both in-domain and out-of-domain robotic manipulation tasks show that AFIL consistently improves task success rates and robustness over strong VLA baselines, while introducing only moderate additional parameters. Our contributions are summarized as follows: • We introduce Adaptive Failure-Informed Learning (AFIL) for diffusion- and flow-based VLAs, using online-generated failure rollouts as a principled form of adaptive negative guidance for sequential action generation. • We propose a Dual Action Generator (DAG)-VLA architecture that jointly learns from success and failure data while sharing a common VLM backbone, enabling efficient failure-aware policy learning with limited parameter overhead. • We demonstrate that AFIL consistently improves success rates across diverse in-domain and out-of-domain manipulation tasks, spanning both short- and long-horizon settings, highlighting its effectiveness, efficiency, and generality. 2 Related Work 2.1 Robotic Manipulation via Imitation Learning Recent advances in robotic manipulation have widely adopted imitation learning, particularly behavior cloning, to address tasks ranging from grasping and pick-and-place to dexterous in-hand manipulation and assembly. Many prior works train deep visuomotor policies, typically based on convolutional neural networks, using simulation data or human demonstrations to enable efficient policy learning from visual observations. For example, 47 ; 11 ; 37 ; 1 employ explicit policy learning by directly regressing from the observed state to the action space. In contrast, another line of work formulates policy learning implicitly by modeling action distributions with Energy-Based Models (EBMs), allowing more expressive representations of multimodal behaviors ( 10 ; 24 ; 14 ; 6 ) . We refer readers to 43 for a comprehensive survey of imitation- and reinforcement-learning-based methods for robotic policy learning. 2.2 Vision-Language-Action (VLA) Models VLA models have emerged recently and demonstrated impressive

Read the complete text