CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models

Catherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, Sergey Levine · 2025

Read CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models by Catherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, Sergey Levine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13446.txt.

Introduction

Catherine Glossop and co-authors teach robots to heed fine words by forging counterfactual labels for old data. Robot datasets tend to pair each scene with one bare task line, so models learn scenes, not words. Change the words and the model still does the old act. Their 2025 trick asks a vision model to write what else the robot could have been told to do in the same scene.

Those forged lines give the same images many new, sharp meanings. A model trained on the enriched set was then sent to roam three places, indoors and outdoors, on word-led tasks. Its win rate on the tasks hit twice that of a model trained on the plain set. Hand-task tests with decoy objects showed the same gain. No new robot runs were needed to earn it.

Catalog record

CREATORCatherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, Sergey Levine
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13446

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models Catherine Glossop, William Chen, Arjun Bhorkar, Dhruv Shah, Sergey Levine Abstract Generalist robots should be able to understand and follow user instructions, but current vision-language-action (VLA) models struggle with following fine-grained commands despite providing a powerful architecture for mapping open-vocabulary natural language instructions to robot actions. One cause for this is a lack of semantic diversity and language grounding in existing robot datasets and, specifically, a lack of fine-grained task diversity for similar observations. To address this, we present a novel method to augment existing robot datasets by leveraging vision language models to create counterfactual labels. Our method improves the language-following capabilities of VLAs by increasing the diversity and granularity of language grounding for robot datasets by generating counterfactual language and actions. We evaluate the resulting model’s ability to follow language instructions, ranging from simple object-centric commands to complex referential tasks, by conducting visual language navigation experiments in 3 different indoor and outdoor environments. Our experiments demonstrate that counterfactual relabeling, without any additional data collection, significantly improves instruction-following in VLA policies, making them competitive with state-of-the-art methods and increasing success rate by 27% on navigation tasks. I Introduction Large vision-language models (VLMs) are powerful not only because of their diverse capabilities but also because they can be steered with fine-grained instructions to produce specific outputs. Ideally, powerful generalist robot policies should exhibit the same level of controllability on embodied tasks. Vision-language-action (VLA) policies take advantage of strong VLM priors to train robot policies end-to-end that map open-vocabulary language instructions to robot actions. However, in practice, these policies often struggle with following language instructions. Even with high-quality and diverse data, the future action distribution typically collapses given any single observation (e.g., given an observation of a chest of drawers, the only probable task for a robot is “open the drawer”). Thus, even powerful models have little incentive to pay attention to the language command, suffering from posterior collapse [ 1 , 2 ] . While this problem can be partially mitigated with curated data collection, this seems contradictory to the promise of training generalist models on all available data, including data collected without the intention of promoting language following. In this paper, we address the problem of training steerable language-conditioned robot policies. To this end, we present an approach to augment existing datasets with language-labeled counterfactual trajectories in hindsight by leveraging VLM priors, without requiring existing language labels. We leverage a VLM to generate hypothetical instructions that the robot could perform given a situation encountered in the dataset, exploring variations in behavior and language. Given a hypothetical instruction, we then synthesize plausible actions that would have been taken in these observed states. This effectively expands the training label distribution beyond what is present in the original dataset. For example, given a trajectory of a robot navigating straight down a hallway, our method can generate counterfactual labels such as “navigate along the left wall" or “drive to the right of the pedestrian," even if these specific situations, and therefore labels, were not present in the original data. To enable this sort of counterfactual data generation, our key observation is that it is much easier to train a policy to produce good actions for simple commands like “turn left” or “turn right.” This policy can then generate action labels for more complex commands, like “follow the left wall,” by breaking them up into these simple commands using large VLMs. This counterfactual data can then be used to train VLA policies that understand and respond to a broad range of user instructions at test time. The main contribution of our work is C ounterfactual A ugmentation with S ynthetic T rajectories (CAST): a novel data augmentation scheme for improving instruction following in VLA models. We instantiate this scheme for the task of last-mile vision-language navigation (VLN), focusing on following nuanced prompts rather than navigating long distances, in unseen environments. We fine-tune an open-source VLM [ 3 ] with our augmented dataset and evaluate performance in 3 environments on 27 language prompts. We also compare CAST to alternate schemes proposed in prior work, as well as state-of-the-art vision-language navigation baselines, and find that our policy achieves a 53% success rate on challenging, diverse language prompts, outperforming a VLA trained without CAST by 27%. In the interest of reproducibility, we have open-sourced the CAST augmentation code, CounterfactualVLA policy training code, the CAST dataset, as well as trained checkpoints. II Related Work Task-conditioned policies. The problem of visual robot navigation has been tackled with image goals [ 4 , 5 , 6 , 7 , 8 ] to perform short- and long-horizon navigation tasks. However, deploying image goal conditioned policies requires prior access to the environment or more complex approaches such as image generation models [ 9 ] . Language-conditioned methods provide a more user-friendly way to specify navigation tasks [ 10 , 11 , 12 , 13 ] and leverage the image-language association already present in pre-trained VLMs. However, truly general instruction following requires the policy to understand complex language instructions beyond object referencing, such as long-form text, referential queries, etc. The first hurdle to successfully training these models then lies in ensuring that the policy can interface with language. While some methods only use pre-trained language encoders to get language embeddings [ 14 , 15 , 16 , 17 ] , more recent works leverage VLMs as their base models, providing strong language and vision priors, and then finetune on robot data [ 18 , 19 , 20 , 21 , 22 , 23 ] to obtain VLA policies. Standard language-conditioned behavior cloning (LCBC) methods use datasets annotated with ground truth language labels, though more recent works use captioning methods to generate language labels post-hoc. Our focus in this paper is not to propose a new model or algorithm for training language-conditioned policies, but rather to develop a new way of synthetically labeling existing data to improve the degree to which the policy follows the language commands. Our contribution is complementary to prior advances in language-conditioning and can be combined with a variety of model classes. Zero-shot robot control with VLMs. While most IL methods assume access to language labels, some methods aim to avoid the need for ground truth labels. Several works have developed systems to use the power of semantic understanding present in VLMs to produce robot actions directly without any training or fine-tuning [ 24 , 25 , 26 , 27 , 11 , 28 , 29 ] . However, these methods are limited by the strong assumption that the training distribution of these models provides a sufficient prior to performing real-world robot behaviors. Most pretraining tasks for vision foundation models are centered on static image understanding, visual question answering (VQA), object detection, and captioning, and thus such models tend to have limited temporal, spatial, and physical understanding of the real world [ 30 , 31 , 3 ] . Our method instead leverages VLM priors by casting counterfactual instruction generation as a captioning task. We ensure that these labels are grounded in real robot behaviors by simultaneously querying for a corresponding atomic command and, therefore, action label. Hindsight data augmentation. CAST is

Read the complete text