RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation

Songhao Han Boxiang Qiu Yue Liao Siyuan Huang, Chen Gao Shuicheng Yan Si Liu · 2025

Read RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation by Songhao Han Boxiang Qiu Yue Liao Siyuan Huang, Chen Gao Shuicheng Yan Si Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06677.txt.

Introduction

Songhao Han and co-authors introduce RoboCerebra, a benchmark for testing robot planning over long tasks, in a 2025 paper. New vision-language models can guide robots, but most tests stress quick reactions rather than slow, goal-directed thought. Long household tasks need memory, planning, and reflection, which short tests miss.

The benchmark pairs a large simulated data set of household tasks with a two-level system: a high-level planner and a low-level controller. Task instructions were drafted by a model, split into subtasks, and carried out by human operators in simulation. The result is longer action sequences with denser labels than past benchmarks. The team tested top models as planners and scored their planning, reflection, and memory.

Catalog record

CREATORSonghao Han Boxiang Qiu Yue Liao Siyuan Huang, Chen Gao Shuicheng Yan Si Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation Songhao Han Boxiang Qiu Yue Liao Siyuan Huang, Chen Gao Shuicheng Yan Si Liu Abstract Recent advances in vision-language models (VLMs) have enabled instruction-conditioned robotic systems with improved generalization. However, most existing work focuses on reactive System 1 policies, underutilizing VLMs’ strengths in semantic reasoning and long-horizon planning. These System 2 capabilities—characterized by deliberative, goal-directed thinking—remain underexplored due to the limited temporal scale and structural complexity of current benchmarks. To address this gap, we introduce RoboCerebra, a benchmark for evaluating high-level reasoning in long-horizon robotic manipulation. RoboCerebra includes: (1) a large-scale simulation dataset with extended task horizons and diverse subtask sequences in household environments; (2) a hierarchical framework combining a high-level VLM planner with a low-level vision-language-action (VLA) controller; and (3) an evaluation protocol targeting planning, reflection, and memory through structured System 1–System 2 interaction. The dataset is constructed via a top-down pipeline, where GPT generates task instructions and decomposes them into subtask sequences. Human operators execute the subtasks in simulation, yielding high-quality trajectories with dynamic object variations. Compared to prior benchmarks, RoboCerebra features significantly longer action sequences and denser annotations. We further benchmark state-of-the-art VLMs as System 2 modules and analyze their performance across key cognitive dimensions, advancing the development of more capable and generalizable robotic planners. robocerebra.github.io Figure 1: We shift the focus of robotic imitation learning from fast, reactive System 1 behavior to slow, deliberative System 2 reasoning. To support this, we introduce RoboCerebra , a benchmark centered on long-horizon tasks composed of extended subtask sequences. (a) A top-down data generation pipeline uses an LLM to produce high-level task instructions and decompose them into subtasks. Human operators execute these in simulation to collect trajectories, with multi-stage verification ensuring quality and semantic consistency. (b) A dataset example showing a long, fine-grained subtask sequence under dynamically changing visual conditions. (c) RoboCerebra features significantly longer trajectories, approximately 6× those in existing robotic manipulation benchmarks. Table 1: Comparison of Benchmarks. Our RoboCerebra benchmark is designed to evaluate System 2 capabilities in robotic manipulation. It is designed to generate long-horizon tasks with large language models (LLMs), enriched with human-collected trajectories.Our benchmark also includes fine-grained decomposed substeps, dynamic scene variations, and time-segment annotations, all of which are critical factors in long-horizon tasks yet currently missing in existing benchmarks. Benchmarks Train (var) Test (var) Long-horizon LLM-Gen Tasks Human Traj. Dynamic Time-Anno. FG. Decomp. RLBench james2020rlbench 100 (323) 100 (323) ✗ ✗ ✗ ✗ ✗ ✗ VLMBench zheng2022vlmbench 8 (233) 8 (374) ✗ ✗ ✗ ✗ ✗ ✗ ARNOLD gong2023arnold 8 (3,571) 8 (800) ✗ ✗ ✗ ✗ ✗ ✗ ALFRED shridhar2020alfred 7 (21,023) 7 (1,529) ✓ ✗ ✓ ✗ ✗ ✗ Calvin mees2022calvin 34 34 (1,000) ✓ ✗ ✓ ✗ ✗ ✗ RoboCasa robocasa2024 100 100 ✓ ✓ ✗ ✗ ✗ ✗ Libero-Long liu2024libero 10 (500) 10 (500) ✓ ✗ ✓ ✗ ✗ ✗ VLABench zhang2024vlabench 100 100 ✓ ✗ ✗ ✗ ✗ ✗ RoboCerebra 1,000 (100,000) 60 (600) ✓ ✓ ✓ ✓ ✓ ✓ 1 Introduction Recent advances in vision-language models (VLMs) Qwen-VL ; liu2023llava ; openai2024gpt4o ; reid2024gemini have introduced new capabilities for robotic manipulation kim2024openvla ; driess2023palme . Departing from conventional control paradigms, recent research huang2024copa ; huang2023voxposer has increasingly explored the use of foundation models to enable more generalizable, instruction-conditioned robotic behavior. By integrating VLMs, robotic systems gain enhanced competence in grounding natural language commands within complex visual contexts, thereby improving adaptability across diverse and unstructured environments. A prevalent paradigm instantiates VLMs as fast-reactive System 1 modules—vision-language-action (VLA) models driess2023palme ; rt22023arxiv ; kim2024openvla that function as reactive policies, mapping multimodal inputs directly to low-level control signals. While effective for real-time execution, this usage fails to fully exploit the models’ strengths in semantic abstraction, relational understanding, and contextual reasoning. These capabilities are fundamentally aligned with slow-thinking System 2 processes shi2025hi ; shah2024bumble , such as long-horizon planning and subgoal decomposition. To unlock the full potential of VLMs in robotics, it is imperative to move beyond reactive policy deployment and utilize these models as deliberative planners within hierarchical frameworks. In line with this vision, recent benchmarks liu2024libero ; robocasa2024 ; mees2022calvin ; shridhar2020alfred have extended robotic tasks from single-step instructions to multi-step procedures. However, they remain limited in temporal scale and structural complexity, typically involving only 2 to 5 sub-tasks and fewer than 500 action steps zhang2024vlabench ; robocasa2024 . While these benchmarks move beyond early reactive settings, they fall short of capturing real-world demands such as hierarchical goal decomposition, temporal abstraction, and adaptive planning. As a result, the System 2 capabilities of VLMs—particularly high-level reasoning and long-horizon planning—remain underexplored. Addressing this gap requires benchmarks with extended horizons, diverse subgoals, and complex reasoning in dynamic, partially observable environments. To address these limitations and enable a comprehensive evaluation of System 2 capabilities, we present RoboCerebra , a novel benchmark designed to assess long-horizon planning and high-level reasoning in robotic manipulation. RoboCerebra includes: (1) a large-scale manipulation dataset featuring extended task horizons and dynamically evolving environments that better reflect real-world complexity; (2) a baseline framework that integrates System 2–System 1 coordination for hierarchical policy execution; and (3) an evaluation protocol tailored to isolate and measure System 2 performance. The dataset is constructed in simulation to support long-horizon task evaluation. Since the focus is on high-level reasoning rather than low-level control, the sim-to-real gap is less critical. Simulation further offers scalability and reproducibility, enabling systematic benchmarking. We design an efficient and high-quality top-down data generation pipeline: GPT chatgpt is prompted to generate high-level task instructions conditioned on environment context and to decompose them into coherent subtask sequences. To encourage deeper temporal dependencies and complex subgoal structures, we craft prompt strategies that promote long-horizon compositionality. Human operators execute these subtasks in simulation to collect demonstration trajectories, with dynamic object variations introduced to increase scene complexity and semantic diversity. Compared to prior datasets, RoboCerebra provides significantly longer action sequences (6×) and denser subtask annotations, forming a more rigorous testbed for evaluating System 2 reasoning. To demonstrate the utility of RoboCerebra, we develop a Hierarchical Planning and Execution (HPE) Framework composed of a high-level VLM planner and a low-level VLA controller. The VLM generates structured multi-step plans from low-frequency observations and stores them in a memory bank, while the VLA executes fine-grained actions using high-frequency visual inputs. During training, the VLM learns temporal grounding from video demonstrations, and the VLA acquires visuomotor skills from paired visual inputs and primitive actions. At inference, the VLM updates the plan and memory, and the VLA executes actions accordingly. This hierarchical design combines semantic reasoning with precise control, enabling robust execution in dynamic, long-horizon tasks. To systematically assess System 2 capabilities, we design an evaluation pipeline that targets key cognitive functions in robotic manipulation. We first train a VLA model at the subtask level as a fixed System 1 controller. Given this shared System 1, we evaluate three critical dimensions of System 2 reasoning through structured System 1–System 2 interaction: (1) planning, the ability to decompose high-level goals into subtask sequences; (2) reflection, the ability to assess task completion status; and (3) memory, the ability to retain and utilize long-term context. We conduct a comprehensive evaluation of state-of-the-art VLMs, including GPT-4o openai2024gpt4o

Read the complete text