VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models

Alex S. Huang, Jiahui Zhang, Shiqing Tang, Yu Xiang · 2026

Read VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models by Alex S. Huang, Jiahui Zhang, Shiqing Tang, Yu Xiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20774.txt.

Introduction

Alex Huang and three colleagues present VLA-REPLICA, a low-cost, reproducible benchmark for real-world testing of vision-language-action models. Simulation tests miss real-world complexity, while existing real-world tests often demand costly hardware or centralized testing. Built from off-the-shelf parts, their system can be assembled and copied across labs for consistent policy testing in any lab. It includes diverse manipulation tasks and a small demo set for target-domain adaptation, with rules for in-distribution and out-of-distribution settings.

Catalog record

CREATORAlex S. Huang, Jiahui Zhang, Shiqing Tang, Yu Xiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71633
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20774

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models Alex S. Huang, Jiahui Zhang, Shiqing Tang, Yu Xiang Abstract Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, but their real-world evaluation remains limited by a lack of accessible, reproducible, and consistent benchmarks. Simulation benchmarks fail to capture real-world complexity, while existing real-world benchmarks often require expensive hardware, centralized evaluation, or are limited in task diversity. We introduce VLA-REPLICA , a low-cost, easily reproducible real-world benchmark for evaluating VLA models. Built from off-the-shelf components, our system can be quickly assembled and replicated across laboratories, providing a consistent environment for policy evaluation anywhere in the world. VLA-REPLICA includes a diverse suite of manipulation tasks and a small-scale demonstration dataset for target-domain adaptation, with real-world evaluation protocols for both in-distribution and out-of-distribution settings. Experiments with imitation learning and state-of-the-art VLA models reveal model strengths and limitations, while consistent results across independently constructed setups demonstrate the reproducibility of our benchmark. 1 1 1 Data, code, and videos for the project are available at https://irvlutd.github.io/VLAReplica/ . 1 Introduction Vision-Language-Action (VLA) models [ 16 , 2 , 12 ] have recently emerged as a promising paradigm for building general-purpose robotic systems that can interpret natural language instructions and execute corresponding manipulation behaviors. By leveraging large-scale multimodal data [ 25 , 15 ] , these models demonstrate impressive generalization across objects, scenes, and tasks, bringing robotics closer to the goal of flexible, human-level interaction. However, despite their success in controlled settings, a fundamental challenge remains: how to reliably evaluate VLA models in the real world . Existing evaluation protocols fall short in addressing this challenge. Simulation-based benchmarks such as Meta-World [ 20 ] and LIBERO [ 18 ] provide scalable and standardized environments, but they suffer from the well-known sim-to-real gap, often leading to overly optimistic estimates of real-world performance. In contrast, real-world benchmarks offer more faithful evaluation but introduce new limitations. Some benchmarks, such as REPLAB [ 30 ] and SceneReplica [ 14 ] , focus primarily on object grasping, limiting task diversity. Others, such as FurnitureBench [ 10 ] and ManipulationNet [ 6 ] , require expensive hardware setups, carefully engineered environments, and substantial human effort to reproduce. Additionally, systems like RoboArena [ 1 ] rely on centralized or remote evaluation, where users must submit policies for external execution, restricting accessibility, slowing iteration, and reducing transparency. As a result, there is currently no widely adopted benchmark that is both real-world grounded and easily reproducible across independent laboratories . To address these limitations, we introduce VLA-REPLICA , a low-cost, reproducible real-world benchmark for evaluating VLA models. Our key design principle is to make real-world evaluation accessible, standardized, and locally executable . Specifically, VLA-REPLICA is built on an affordable robotic platform using off-the-shelf components, including a low-cost SO-101 manipulator, RGB-D cameras, and a controlled light-box environment. The hardware setup is illustrated in Fig. 1 (a). The full setup can be assembled in under an hour by non-expert users, and detailed instructions ensure consistent replication across different sites. Beyond hardware accessibility, VLA-REPLICA provides a structured evaluation protocol tailored to modern VLA models. The benchmark includes a suite of real-world manipulation tasks with diverse interaction patterns (e.g., pick-and-place, tool use, and object manipulation), along with a curated dataset of human demonstrations for target-domain adaptation. Fig. 1 (b) shows the 10 manipulation tasks in our benchmark. Crucially, the evaluation is designed to measure both in-distribution performance , i.e. how well models learn from limited real-world data, and out-of-distribution generalization , including variations in object properties and task requirements. This setup reflects practical deployment scenarios, where pretrained VLA models must adapt to new environments with minimal data. A central contribution of our work is reproducibility at scale . We introduce a standardized environment construction pipeline, including precise workspace configuration, camera placement, background lighting, and object placement protocols, enabling consistent evaluation across independently built setups. This design allows different users to recreate nearly identical physical environments, making it possible to compare results across laboratories without centralized infrastructure. Figure 1 : Overview of the VLA-Replica benchmark. (a)1. Hardware components. (a)2. Our assembled platform with the leader arm, the follower arm, the light box, the cameras, and the manipulation workspace. (b) 10 manipulation tasks in the benchmark. We validate VLA-REPLICA by benchmarking both imitation learning methods [ 33 , 7 ] and state-of-the-art VLA models [ 2 , 12 , 26 ] under a unified training and evaluation protocol. Our experiments reveal key insights into the performance of current VLA systems, particularly their ability to adapt to new environments and generalize beyond training distributions. We further demonstrate that evaluation results are consistent across independently constructed setups, highlighting the robustness and reproducibility of our benchmark. In summary, this paper makes the following contributions: • We introduce VLA-REPLICA , a low-cost ( \sim $1050 USD), fully reproducible real-world benchmark for evaluating VLA models. • We design a standardized hardware and environment setup that can be replicated across laboratories with minimal effort and cost. • We propose a comprehensive evaluation protocol covering both in-distribution learning and out-of-distribution generalization. • We provide extensive empirical evaluation of state-of-the-art VLA models, offering insights into their real-world performance and limitations. 2 Related Work Table 1 : Comparison of robot manipulation benchmarks across environment, reproducibility, hardware cost, task diversity, dataset availability, and evaluation protocol. Benchmark Year Environment Repro- ducibility Hardware Cost Task Diversity Expert Dataset Distributed Evaluation Meta-World [ 32 ] 2020 Simulation High – High (50 tasks) ✓ ✓ LIBERO [ 18 ] 2023 Simulation High – High (100 tasks) ✓ ✓ RoboCasa [ 22 ] 2024 Simulation High – High (100 tasks) ✓ ✓ RoboLab [ 31 ] 2026 Simulation High – High (120 tasks) ✗ ✓ REPLAB [ 30 ] 2019 Real Low Low Low (Grasping) ✓ ✓ RAMP [ 8 ] 2023 Real High (April Tag) High Low (Assembly) ✗ ✓ SceneReplica [ 14 ] 2024 Real High (Overlay) High Low (Grasping) ✗ ✓ FurnitureBench [ 10 ] 2025 Real High (April Tag) High Low (Assembly) ✓ ✓ FMB [ 19 ] 2025 Real Low High Low (Assembly) ✓ ✓ AutoEval [ 35 ] 2025 Real Low None (Service) Low (4 tasks) ✓ ✗ RoboArena [ 1 ] 2025 Real Low High High (User-defined) ✗ ✓ RoboChallenge [ 29 ] 2025 Real High (Overay) None (Service) High (30 tasks) ✓ ✗ VLA-Replica (Ours) 2026 Real High (Overlay+Lighting) Low ( \sim $1050 USD) High (10 Tasks) ✓ ✓ Vision-Language-Action Models Recent years have seen the emergence of VLA models [ 4 , 3 , 16 , 2 , 12 , 24 , 11 ] , which are general-purpose robotic manipulation policies trained on diverse robot datasets for language-conditioned tasks. While these models show promising generalization across objects, scenes, and tasks, they often struggle when deployed in new environments or on new robot embodiments. Prior work therefore commonly fine-tunes pretrained VLA policies using a small amount of target-domain data, enabling adaptation to the target environment, embodiment, and tasks. Simulation Benchmarks for Robot Manipulation Robotic evaluation benchmarks have been widely used to measure the performance and generalization of robot policies. Table 1 provides a comparison of representative robot manipulation benchmarks. A large body of benchmarks are

Read the complete text