MiniVLA-Nav v1: A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation
Ali Al-Bustami, Jaerock Kwon · 2026
Read MiniVLA-Nav v1: A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation by Ali Al-Bustami, Jaerock Kwon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00397.txt.
Introduction
Ali Al-Bustami and Jaerock Kwon release a simulation dataset for teaching robots to follow spoken directions to an object. In their task, a wheeled robot in photorealistic virtual rooms must drive to the named object and stop within a meter of it. The four settings are an office, a hospital, and two warehouses built in a leading simulator.
Each of the 1,174 recorded episodes pairs a short instruction with camera images, depth maps, and object masks, plus the expert's driving actions recorded sixty times per second. Starting distances fall into three tiers, from near to far across the room. The set covers twelve object categories, with fresh wordings and unseen categories held out for testing. Five evaluation splits probe accuracy, robustness to rephrasing, and performance on new object types.
Catalog record
| CREATOR | Ali Al-Bustami, Jaerock Kwon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00397
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MiniVLA-Nav v1: A Multi-Scene Simulation Dataset for Language-Conditioned Robot Navigation Ali Al-Bustami, Jaerock Kwon Abstract We present MiniVLA-Nav v1 , a simulation dataset for Language-Conditioned Object Approach (LCOA) navigation: given a short natural-language instruction, an NVIDIA Nova Carter differential-drive robot must navigate to the named object and stop within 1 m across four photorealistic Isaac Sim environments (Office, Hospital, Full Warehouse, and Warehouse with Multiple Shelves). Each of the 1 174 episodes pairs an instruction with synchronized 640×640 RGB images, metric depth maps (float32, metres), and instance segmentation masks, together with continuous ( v , ω ) (v,\omega) and 7×7 tokenized expert action labels recorded at 60 Hz from a vision-based proportional controller. Trajectory diversity is ensured through three spawn-distance tiers (near: 1.5–3.5 m, mid: 3.5–7.0 m, far: global curated points; Pearson r = 0.94 r=0.94 between spawn distance and trajectory length), 12 object categories, 18 training templates, and 12 paraphrase-OOD templates. Five evaluation splits support in-distribution accuracy, template-paraphrase robustness, and OOD object-category benchmarking. The dataset is publicly available at https://huggingface.co/datasets/alibustami/miniVLA-Nav . I Introduction Teaching robots to follow natural-language navigation instructions is a foundational challenge in embodied AI. Recent vision-language-action (VLA) models [ 3 , 7 , 9 ] demonstrate that language-conditioned policies can be trained end-to-end from large demonstration datasets, yet acquiring diverse, high-quality robot demonstrations at scale remains expensive and logistically difficult in the real world. Simulation offers a practical remedy: physics-faithful environments generate demonstrations at zero marginal cost with perfect ground-truth annotation. However, existing datasets have key gaps: continuous-action navigation datasets with dense per-step expert labels remain rare [ 8 ] ; large-scale object-goal navigation benchmarks use discrete panoramic actions rather than differential-drive control [ 4 ] ; and language-annotated demonstration datasets target tabletop manipulation rather than wheeled navigation [ 12 ] . Critically, no publicly available dataset combines multiple indoor scene types , continuous differential-drive expert actions , language instructions , and multi-modal per-step observations (RGB + depth + segmentation) in a single collection. This paper introduces MiniVLA-Nav v1 , a dataset designed for the Language-Conditioned Object Approach (LCOA) task: given a short natural-language instruction and front RGB-D observations, a differential-drive robot must navigate to the named object and stop within 1 m. Our primary contributions are: 1. A scalable, resumable data-generation pipeline built on NVIDIA Isaac Sim 5.1, automating scene loading, tiered spawn sampling, expert rollout, and structured episode archiving with full reproducibility from a single random seed. 2. Four indoor scenes spanning office, hospital, and warehouse domains, with 12 object categories, three spawn-distance tiers, and per-scene seen/held-out category configurations. 3. Multi-modal per-timestep observations : synchronized 640×640 RGB images, metric depth maps (float32, metres), and instance segmentation masks, paired with continuous (v,\omega) and 7×7 discretized action tokens. 4. Five evaluation splits supporting in-distribution accuracy, template-paraphrase OOD robustness, and OOD object-category generalization, with curated heldout categories per scene. 5. Comprehensive dataset statistics characterizing trajectory length, spawn distance, tier difficulty, instruction template coverage, and object frequency across 1,174 validated episodes. This paper releases the dataset, generation pipeline, and evaluation tooling; training baselines will appear in a companion publication. II Related Work II-A Vision-and-Language Navigation Anderson et al. [ 1 ] introduced Room-to-Room (R2R), requiring agents to follow multi-step instructions in panoramic indoor environments; follow-up work extended this to continuous action spaces [ 8 ] and outdoor scenes [ 5 ] . Gu et al. [ 6 ] survey the full VLN landscape and identify single-step approach tasks with dense action labels as an under-explored regime — the gap MiniVLA-Nav directly targets. MiniVLA-Nav is complementary to multi-step VLN: it focuses on the approach sub-problem with continuous differential-drive control and dense per-step imitation labels. II-B Simulated Robot Datasets The Habitat platform [ 13 ] provides large-scale rearrangement and navigation benchmarks with discrete panoramic actions; continuous-action extensions [ 8 ] exist but lack language conditioning and multi-scene diversity. The ObjectNav benchmark [ 2 ] targets object-goal finding using discrete panoramic actions without language instructions; Chaplot et al. [ 4 ] extend it with semantic exploration but still use discrete actions. ALFRED [ 12 ] provides language-annotated demonstrations but targets tabletop manipulation in static scenes, not mobile navigation. RoboVQA [ 11 ] and OpenX [ 10 ] aggregate large real-robot datasets but lack systematic OOD evaluation splits and are focused on manipulation rather than wheeled navigation. MiniVLA-Nav v1 addresses the specific gap of multi-scene, continuous-action, language-conditioned wheeled-navigation demonstrations with structured OOD evaluation. II-C Language-Conditioned Control RT-2 [ 3 ] and OpenVLA [ 7 ] show that VLA models fine-tuned on imitation data can follow novel instructions zero-shot, but both target tabletop manipulation. To our knowledge, MiniVLA-Nav v1 is the first dataset combining multiple indoor scene types, continuous differential-drive expert trajectories, multi-modal per-step observations, and language conditioning with systematic OOD evaluation splits. Table I positions MiniVLA-Nav v1 against representative prior datasets on key dimensions. TABLE I: Comparison with Related Datasets Dataset Sim? Action Lang. Scenes OOD splits R2R [ 1 ] ✗ Discrete ✓ 90 ✗ VLN-CE [ 8 ] ✓ Cont. ✓ 1 ✗ ObjectNav [ 2 ] ✓ Discrete ✗ 1 ✗ ALFRED [ 12 ] ✓ Discrete ✓ 1 ✗ MiniVLA-Nav v1 (ours) ✓ Cont. ✓ 4 ✓ III Task Definition III-A Language-Conditioned Object Approach (LCOA) Given a natural-language instruction \ell and a stream of front-facing observations o_{t}=(\mathbf{I}_{t}^{\text{RGB}},\mathbf{D}_{t}) , the robot must output a sequence of actions a_{t}=(v_{t},\omega_{t}) such that \|p_{T}-p_{g}\|\leq r_{\text{success}}=1.0\,\text{m}, (1) where p_{T} is the robot position at termination and p_{g} is the 3-D centroid of the target object bounding box. Episode termination occurs on one of three conditions: • Success : robot within r_{\text{success}} and stationary for \geq 5 consecutive steps (stopped-hold criterion). • Collision : robot commanded forward but makes no progress for \geq 16 consecutive steps with a near obstacle (stall detection). • Timeout : maximum T_{\max}=1000 steps reached without success. Only successful episodes are retained in the released dataset. III-B Action Space The continuous action is (v,\omega)\in[0,1]\,\text{m/s}\times[-1.5,1.5]\,\text{rad/s} . For VLA-style token prediction, each dimension is quantized to 7 uniform bins, yielding a 7×7 = 49-token joint action vocabulary (Fig. 1 ). Fig. 1: Discrete 7×7 action space. Each grid point is one valid ( v , \omega ) token. The origin (stop) and maximum-forward tokens are annotated. IV Data Collection Pipeline IV-A Simulation Environment All data were collected in NVIDIA Isaac Sim 5.1 (release 5.1.0-rc.19 ), which provides GPU-accelerated rigid-body physics (PhysX), photorealistic rendering (RTX), and a Python API for programmatic scene control. The physics time step is \Delta t=1/60\,\text{s} , so the expert controller and sensor observations operate at 60 Hz . IV-B Robot Platform We use the NVIDIA Nova Carter , a differential-drive platform with a 0.52 m wheelbase and 0.14 m wheel radius. The onboard front-facing stereo camera ( front_hawk/right ) is configured at 640×640 pixels and provides synchronized RGB, distance-to-image-plane depth (float32, metres), and instance segmentation frames at every simulation step. IV-C Scene Catalog and Target Discovery Four scene USDs from the Isaac Assets library are used (Table II ). At the first episode of each scene, the USD stage is traversed to discover navigable targets by matching prim names against 12 category rules ( chair , sofa , table , monitor , plant , trash_can , fire_extinguisher ,