RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models

Zhaolong Shen Changti Wu Ruina Hu Bailing Wang Cong Huang Kai Chen · 2026

Read RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models by Zhaolong Shen Changti Wu Ruina Hu Bailing Wang Cong Huang Kai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02277.txt.

Introduction

Yu and co-authors build a benchmark that tests semantic grounding in robot action models. Vision-language-action models should let semantics guide robot actions. Pretrained language and vision backbones supply that understanding. Robot fine-tuning is optimized as imitation over task actions though. Many evaluations can be solved through shortcuts instead.

Visual or instruction-action shortcuts can fake grounding in tests. Their RoboSemanticBench diagnoses whether grounding is really used. In each episode, the robot gets a multiple choice question. Math and general knowledge questions are both used. Candidate answer blocks sit in front of the robot. It must grasp the block with the correct answer. Complex instruction semantics decide the right target. Controlled arithmetic tasks are part of the suite.

Catalog record

CREATORZhaolong Shen Changti Wu Ruina Hu Bailing Wang Cong Huang Kai Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models Zhaolong Shen Changti Wu Ruina Hu Bailing Wang Cong Huang Kai Chen Abstract Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB) , an embodied benchmark for diagnosing semantic grounding in action prediction : whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction. https://github.com/ZGC-EmbodyAI/RoboSemanticBench † † footnotetext: Work done at Zhongguancun Academy (Beijing). 1 Introduction Vision-language-action (VLA) models are motivated by a compelling promise: the semantic competence of pretrained language or vision-language models should become part of robot action prediction. Representative systems, including \pi_{0} , are often described as dual-system architectures with a low-frequency System-2 Semantic Expert and a high-frequency System-1 Action Expert [ 16 , 1 , 39 ] . As illustrated in Figure 1 , the Semantic Expert processes observations and instructions, while the Action Expert turns its outputs and proprioception into continuous actions. Under this view, a VLM backbone should not merely condition a controller on text; its general knowledge and semantic understanding should participate in the action-generation pathway. Figure 1: Dual-system VLA architecture and the semantically grounded action prediction question. RoboSemanticBench tests whether the System-2 Semantic Expert’s competence is preserved and used by the System-1 Action Expert after VLA post-training. The central concern is that this promise may not survive current VLA post-training pipelines. Robot demonstrations are much smaller and more task-specific than language pretraining corpora, and imitation losses can reward fitting conditional action distributions even when the Semantic Expert is weakened or decoupled from the Action Expert. In a typical demonstration dataset, an instruction is paired with a successful trajectory, but the loss rarely forces the model to expose the semantic decision that made the trajectory correct. The policy may therefore learn instruction-action or visual shortcuts rather than use language semantics to determine which action is correct, creating a direct tension with the motivation for VLA models [ 12 , 35 , 34 ] . This tension matters because realistic robot instructions are often complex, underspecified, or require commonsense interpretation, whereas many simulation benchmarks use short explicit commands such as picking a named object or moving to a visible location. In such settings, VLA policies can ignore language and still score well by exploiting visual shortcuts or dataset regularities [ 18 , 9 , 11 ] . A high task success rate can therefore be ambiguous: it may indicate genuine instruction understanding, or merely that the benchmark admits non-semantic shortcuts. Recent studies further suggest that preserving the general semantic competence of pretrained VLM backbones is important for action-generation generalization [ 32 , 36 , 11 ] . Yet existing benchmarks rarely measure whether such competence is actually grounded in action prediction. We call the missing capability semantic grounding in action prediction : when controlling a robot, a VLA model should understand the semantics of a human instruction, ground them in the current observation, and act according to the user’s intent. We propose RoboSemanticBench ( RSB ), a controlled diagnostic for semantic grounding in action prediction. As shown in Figure 2 , each episode asks a VLA to understand a multiple-choice math or general-knowledge instruction, bind the correct answer to a visible target, and execute the corresponding grasp. Because the manipulation primitive is fixed while the semantic content and option set vary, RSB tests whether instruction semantics guide target selection rather than merely fitting instruction-action correlations. This design is deliberately simple in the motor domain but demanding in the semantic domain. It exposes two questions: whether the Semantic Expert can still solve the instruction-level problem, and whether the Action Expert can follow the resulting implicit target during action prediction. A model that can reliably grasp candidate targets should still fail if it cannot ground the instruction’s semantic answer into the target-selection action. We therefore compare grasping any candidate block with grasping the correct one: a large GSR–TSR gap reveals successful manipulation but failed semantic grounding. This makes RSB a diagnostic of the VLA interface between semantic understanding and action prediction, rather than a generic test of grasping skill. Our contributions are: • We introduce RoboSemanticBench , a benchmark that turns math, hard-math, and general-semantic understanding into embodied answer selection across six evaluation suites. • We define GSR, TSR, and nSG to separate low-level grasping from semantic target selection and diagnose whether instruction semantics participate in action prediction. • We evaluate representative VLA models and show that many perform near or below random semantic target selection after controlling for grasp success. • We report negative interventions and error analyses, showing that CoT-style ReasoningVLA and VLA cotraining do not reliably close the semantic grounding gap. 2 Related Work 2.1 Vision-Language-Action Models and Benchmarks Recent VLA models connect pretrained vision-language representations with robot control. Early embodied foundation models show that web-scale visual-language pretraining can provide reusable semantic knowledge for robot tasks [ 7 , 2 ] ; later systems scale this idea through large robot datasets and open generalist policies, including Open X-Embodiment/RT-X [ 23 ] , Octo [ 27 ] , OpenVLA [ 16 ] , \pi_{0} [ 1 ] , \pi_{0.5} [ 15 ] , and GR00T N1 [ 22 ] . Recent work further studies how to adapt VLM backbones into VLA policies without losing pretrained capability [ 12 , 32 , 36 , 11 ] . Yet standard task success does not reveal whether semantic competence is actually preserved and used for action prediction. Benchmarks for language-conditioned manipulation measure complementary abilities. CALVIN and LIBERO study multi-task and lifelong manipulation [ 20 , 19 ] , SimplerEnv evaluates policies through simulation-based reproduction [ 17 ] , and other suites scale household tasks, robot data generation, world-knowledge manipulation, or VLA comparison [ 21 , 38 , 3 , 33 , 4 ] . These benchmarks are valuable for robustness and generalization, but their task success metrics often entangle motor execution, object recognition, and language grounding. RoboSemanticBench instead fixes the manipulation primitive and varies the instruction’s semantic content, making it a targeted diagnostic for whether semantic understanding is grounded in robot action prediction. 2.2 Diagnosing Language Grounding and Shortcut Behavior in VLAs Several studies question whether VLA policies use language as intended. Policies may exploit visual regularities or action priors, ignore instructions under conflicting visual evidence, or suffer degraded VLM representations after robot fine-tuning [ 32 , 9 , 11 , 12 , 35 , 34 ] . Related diagnostics probe instruction perturbations, counterfactual commands, linguistic diversity,

Read the complete text