RobotArena ∞ \infty : Scalable Robot Benchmarking via Real-to-Sim Translation
Yash Jangir, Yidi Zhang, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki · 2025
Read RobotArena ∞ \infty : Scalable Robot Benchmarking via Real-to-Sim Translation by Yash Jangir, Yidi Zhang, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23571.txt.
Introduction
Yash Jangir and nine co-authors built RobotArena Infinity, a framework for grading robot control policies without endless real-world trials. Testing robots in the physical world is slow, costly, hard to repeat, and risky at scale. Judging success is also tricky, since it often rests on subtle human views of how well a motion was carried out.
The framework turns video demonstrations from popular robot datasets into matching simulated scenes, using vision-language models, 2D-to-3D generation, and differentiable rendering. Inside these digital twins, policies are scored by automated model-guided ratings plus lightweight preference votes from crowdworkers. The team also perturbs scenes — changing textures, moving objects — to stress-test how well policies generalize. The result is a benchmark that keeps evolving while staying repeatable, aimed at real-world-trained manipulation policies.
Catalog record
| CREATOR | Yash Jangir, Yidi Zhang, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23571
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RobotArena ∞ \infty : Scalable Robot Benchmarking via Real-to-Sim Translation Yash Jangir, Yidi Zhang, Kashu Yamazaki, Chenyu Zhang, Kuan-Hsun Tu, Tsung-Wei Ke, Lei Ke, Yonatan Bisk, Katerina Fragkiadaki Abstract The pursuit of robot generalists—instructable agents capable of performing diverse tasks across diverse environments—demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. Existing simulation benchmarks are similarly limited, as they train and test policies within the same synthetic domains and cannot assess models trained from real-world demonstrations or alternative simulation environments. As policies expand in scope and complexity, these barriers only intensify, since defining “success” in robotics often hinges on nuanced human judgments of execution quality. In this paper, we introduce a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision-language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated VLM-guided scoring and scalable human preference judgments collected from crowdworkers—transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons. To measure robustness, we systematically perturb simulated environments along multiple axes— textures and object placements — stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for real-world–trained robot manipulation policies, addressing a critical missing capability in today’s robotics landscape. More details and demos are available at robotarenainf.github.io . . 1 Introduction Figure 1: RobotArena \infty provides a scalable and extensible robot benchmarking framework by automating environment construction and evaluation. It automatically generates simulated environment seeded from real videos, deploys robot policies, and evaluates them using VLMs and crowdsourced workers that cast preferences between pairs of execution videos. The simulated environments are derived from both in-distribution and out-of-distribution videos, enabling rigorous tests of generalization in contemporary VLAs. While recent years have witnessed substantial progress in developing more capable and general robot policies, their evaluation remains a persistent challenge and lacks standardization. This problem becomes especially acute as policies grow more generalist, requiring broader and more diverse evaluation scenarios. Real-world evaluation is inherently unscalable : it is limited by logistics, safety concerns, and reproducibility issues, and requires significant human involvement for setup, execution, and scoring. Human operators must supervise trials and manually reset scenes, which restricts the scale and frequency of evaluations Vincent et al. 2024 ; Abou-Chakra et al. 2025 ; Li et al. 2024 . Such manual oversight also raises concerns about consistency and fairness, particularly when baselines and new models are compared under slightly different conditions. Centralized physical evaluation provides a gold standard, where policies are tested under identical conditions, typically by submitting containers or shipping robots to a shared testing site. Notable examples include the Amazon Picking Challenge Correll et al. 2016 , the Open-Vocabulary Mobile Manipulation Challenge Yenamandra et al. 2023 , and RoboCup@Home Matamoros et al. 2019 . However, the high cost for both organizers and participants means such events occur infrequently, often no more than once a year. In contrast, fields such as computer vision and natural language processing have advanced rapidly thanks to standardized benchmarks that provide consistent metrics, clear performance targets, and a foundation for fair comparison across methods Deng et al. 2009 ; Schuhmann et al. 2022 . Our work is particularly inspired by LMarena Chiang et al. 2024 , a large-scale, crowdsourced evaluation framework that benchmarks LLMs and VLMs through direct pairwise comparisons of responses to the same prompt by human annotators. By aggregating thousands of such head-to-head matchups across diverse prompts, LMarena produces an Elo-style ranking that reflects collective judgments of model quality. Motivated by this success, we ask: What would be the analogue of LMArena for Robotics? RobotArena \infty : We introduce RobotArena \infty , a new benchmarking framework that scales robot evaluation by deploying policies in automatically constructed simulated environments and assessing them through automatic VLM score and online human preference feedback. Our framework first automatically translates real videos into corresponding simulation environments. This reality-to-simulation translation method builds on recent advances in vision-language models for scene understanding, 2D-to-3D generative models for 3D asset creation from image crops and videos, and our differentiable rendering techniques for object pose estimation and robot–camera calibration. Once constructed, we deploy VLAs in these environments and evaluate their execution trajectories using two complementary strategies: (1) absolute evaluation, in which prompted VLMs or crowdsourced human workers estimate task progress scores for each video frame, and (2) relative evaluation, in which human annotators express pairwise preferences between execution videos from different VLAs performing the same task. A key strength of RobotArena \infty is its ability to measure both in-distribution performance by testing on simulation environments seeded from training videos in established datasets such as Bridge Walke et al. 2023b , and out-of-distribution performance, by testing on environments generated from videos outside the training set. To assess robustness and generalization, we also introduce systematic perturbations, such as variations in lighting, object placement, and background appearance. Our initial benchmark aggregates more than 7000 preference pairs across one hundred nominal environments and hundreds of perturbations, comparing four VLAs from independent labs worldwide. To the best of our knowledge, this constitutes the largest-scale robot evaluation effort to date. The diversity of our reproducible evaluations yields several insights. First, VLAs exhibit notable sensitivity to dataset differences: their performance declines when evaluated on environments beyond those seen during training, suggesting that current models are not true generalists yet, and instead specialize to the environments underlying their training data, as also found in Xing et al. 2025 . Second, within the same environment, performance degrades under perturbations, indicating that robustness to distribution shifts remains an open area for improvement. Yet, consistent performance rankings emerge across models, demonstrating that architectural and data choices do produce measurable differences. RobotArena \infty is inspired by prior efforts to design scalable robot benchmarks, particularly the seminal contributions of BEHAVIOR Li et al. 2024 and SIMPLER Li et al. 2024 . BEHAVIOR boasts an impressive manual effort of asset and environment creation, while SIMPLER reconstructs four real-world Bridge scenes and includes hand-designed reward functions. Compared to these efforts, RobotArena \infty offers a far more scalable and extensible framework by automating environment construction and evaluation. In summary, our contributions are as follows: 1. We present a scalable and extensible benchmarking protocol for robotics , by coupling physics engines, real-to-sim translation and human preference feedback. 2. We introduce a fully automated reality-to-simulation translation pipeline built upon VLMs, 2D-to-3D generative models and differentiable rendering. 3. We evaluate VLAs from labs worldwide across hundreds of environments with thousands of human preferences , the most extensive robot evaluation to date. 4. We present key evaluation results that reveal how current robot policies generalize—or fail to—under distribution shifts. Our benchmark is not without limitations. We outline these and discuss future directions and extensions. Importantly, RobotArena \infty will continue to benefit from advances in physics engines and real-to-sim research. Both our benchmark environments and evaluation code will be publicly released and centrally maintained for continual support. 2 Related Work Generalist Robot Manipulation