RoboGate : Adaptive Failure Discovery for Safe Robot Policy Deployment via Two-Stage Boundary-Focused Sampling
Azuki Kim · 2026
Read RoboGate : Adaptive Failure Discovery for Safe Robot Policy Deployment via Two-Stage Boundary-Focused Sampling by Azuki Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22126.txt.
Introduction
Kim presents ROBOGATE, a framework for finding failure cases before a robot policy ships. Testing every setting in a high-dimensional space is not feasible. ROBOGATE runs physics simulation with two-stage adaptive sampling. The first stage spreads samples by Latin hypercube across eight parameters. The second stage concentrates on the 30 to 70 percent success zone. Across four robot arms and over 50,000 runs, a logistic risk model reaches AUC 0.780. One vision-language-action policy scores 97.65% on LIBERO yet fails all 68 industrial scenarios here.
Catalog record
| CREATOR | Azuki Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49645 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22126
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoboGate : Adaptive Failure Discovery for Safe Robot Policy Deployment via Two-Stage Boundary-Focused Sampling Azuki Kim Abstract Deploying learned robot manipulation policies in industrial settings requires rigorous pre-deployment validation, yet exhaustive testing across high-dimensional parameter spaces is intractable. We present RoboGate , a deployment risk management framework that combines physics-based simulation with a two-stage adaptive sampling strategy to efficiently discover failure boundaries in the operational parameter space. Stage 1 employs Latin Hypercube Sampling (LHS) across an 8-dimensional parameter space to establish a coarse failure landscape from 20,000 uniformly distributed experiments. Stage 2 applies boundary-focused sampling that concentrates 10,000 additional experiments in the 30–70% success rate transition zone, enabling precise failure boundary mapping. Using NVIDIA Isaac Sim with Newton physics, we evaluate a scripted pick-and-place controller on two robot embodiments—Franka Panda (7-DOF) and UR5e (6-DOF)—across 30,000 total experiments. Our logistic regression risk model achieves an AUC of 0.780 on the combined dataset (vs. 0.754 for Stage 1 alone), identifies a closed-form failure boundary equation μ ∗ ( m ) = ( 1.469 + 0.419 m ) / ( 3.691 − 1.400 m ) \mu^{*}(m)=(1.469+0.419m)/(3.691-1.400m) , and reveals four universal danger zones affecting both robot platforms. We further demonstrate the framework on VLA (Vision-Language-Action) model evaluation, where Octo-Small achieves 0.0% success rate on 68 adversarial scenarios versus 100% for the scripted baseline—a 100-point gap that underscores the challenge of deploying foundation models in industrial settings. RoboGate is open-source and runs on a single GPU workstation. Keywords: robot safety, deployment validation, failure analysis, adaptive sampling, sim-to-real, VLA evaluation 1 Introduction The proliferation of learned manipulation policies—from imitation learning [ 5 ] to Vision-Language-Action (VLA) models [ 1 , 2 ] —has created an urgent need for systematic pre-deployment validation in industrial robotics. While these policies demonstrate impressive capabilities in controlled benchmarks, their behavior under adversarial or edge-case conditions remains poorly characterized. A single undetected failure mode can lead to costly equipment damage, production downtime, or safety incidents. Current validation approaches suffer from two fundamental limitations. First, uniform random testing wastes computational budget on regions of the parameter space that are either trivially easy or trivially hard, providing little information about the critical transition zones where success gives way to failure. Second, most evaluation frameworks test a single robot embodiment, making it impossible to distinguish between policy failures and embodiment-specific limitations. We introduce RoboGate , a deployment risk management framework that addresses both limitations through three contributions: 1. Two-stage adaptive sampling : A principled strategy that first maps the parameter space uniformly (Stage 1, 20K experiments), then concentrates additional experiments in the 30–70% success rate boundary zone (Stage 2, 10K experiments), improving failure boundary resolution by 31.1% coverage of transition regions. 2. Cross-embodiment validation : Parallel evaluation on Franka Panda (7-DOF, parallel-jaw gripper) and UR5e (6-DOF, suction gripper) across shared parameter dimensions, revealing four universal danger zones where both platforms exhibit SR < 40%. 3. Interpretable risk model : A logistic regression model with interaction terms that produces a closed-form failure boundary equation, critical parameter thresholds with bootstrap confidence intervals, and per-experiment risk scores (AUC = 0.780). The framework is validated on 30,000 physics-based simulation experiments using NVIDIA Isaac Sim with Newton physics engine. We also demonstrate its application to VLA model evaluation, where RoboGate reveals that Octo-Small [ 1 ] achieves 0.0% success rate on 68 pick-and-place scenarios—a complete failure across all four difficulty categories. 2 Related Work 2.1 Robot Policy Evaluation Traditional robot policy evaluation relies on fixed benchmark suites with predetermined test configurations [ 6 , 7 ] . RLBench [ 6 ] provides 100 manipulation tasks but evaluates under nominal conditions only. Meta-World [ 7 ] offers parametric task variation but does not systematically explore failure boundaries. LIBERO [ 8 ] benchmarks lifelong learning but lacks adversarial scenario coverage. In contrast, RoboGate explicitly targets the success-failure transition zone through adaptive sampling. 2.2 Sim-to-Real Transfer and Domain Randomization Domain randomization [ 9 , 10 ] has become standard practice for bridging the sim-to-real gap. While DR improves policy robustness during training , it does not address systematic evaluation of the resulting policies across the randomized parameter space. Our work is complementary: we use the same parameter dimensions (friction, mass, visual properties) but focus on mapping the failure landscape rather than improving robustness. Recent work on sim-to-real transfer validation [ 11 ] proposes Bayesian optimization for finding failure-inducing parameters. However, their approach optimizes for worst-case performance, whereas RoboGate maps the entire boundary surface to produce interpretable risk models. 2.3 Adaptive Testing and Falsification Adaptive stress testing (AST) [ 12 ] formulates failure discovery as a Markov decision process, using reinforcement learning to find maximally likely failure trajectories. While effective for autonomous driving, AST focuses on temporal sequences rather than static parameter configurations. The falsification community [ 13 ] has developed coverage-guided techniques for cyber-physical systems, but these typically operate on lower-dimensional specification spaces. Our two-stage approach is most similar to sequential experimental design [ 14 ] , but we replace Bayesian acquisition functions with a simpler binning strategy that scales to 8+ dimensions and produces directly interpretable results. 2.4 Safety Validation for Robot Deployment ISO 10218 [ 15 ] and ISO/TS 15066 [ 16 ] establish safety requirements for industrial robots but provide limited guidance on learned policy validation. Recent frameworks for safe deployment [ 17 ] emphasize the need for quantitative safety metrics, but most focus on reinforcement learning reward shaping rather than pre-deployment testing. The notion of a “deployment gate”—a hard pass/fail check before production release—is standard in software engineering (CI/CD pipelines) but has no established equivalent in robotics. Several recent efforts address this gap. SafeBench [ 18 ] provides a safety evaluation framework for autonomous driving but does not extend to manipulation. RoboCasa [ 19 ] offers large-scale simulation environments for household tasks but focuses on training rather than pre-deployment validation. The NIST Agile Robotics for Industrial Automation Competition (ARIAC) defines standardized evaluation criteria for industrial tasks, but its scoring does not produce interpretable risk models. RoboGate differs from these approaches in three ways: (1) it provides a metric-based validation gate with hard thresholds derived from industrial safety standards, (2) it uses adaptive sampling to efficiently discover failure boundaries rather than exhaustively testing fixed scenarios, and (3) it produces interpretable logistic regression risk models with confidence intervals that can be directly translated into operational constraints. 2.5 Vision-Language-Action Models VLA models represent a paradigm shift in robot learning, combining vision encoders, language understanding, and action prediction in a single architecture [ 1 , 2 , 3 ] . RT-2 [ 3 ] demonstrated that web-scale vision-language pretraining transfers to robotic manipulation, while Octo [ 1 ] provided an open-source generalist model. \pi_{0} [ 4 ] and OpenVLA [ 2 ] pushed the boundaries further with flow matching and 7B-parameter architectures. However, systematic evaluation of VLA models under adversarial conditions—low lighting, cluttered scenes, transparent objects—remains largely absent from the literature. Our VLA evaluation pipeline addresses this gap by testing models against RoboGate ’s 68-scenario suite spanning nominal, edge-case, adversarial, and domain-randomized conditions. 3 Problem Formulation 3.1 Operational Parameter Space We define the operational parameter space \mathcal{P}\subset\mathbb{R}^{d} as the set of environmental and object conditions under which a manipulation