WoMAP: World Models For Embodied Open-Vocabulary Object Localization
Tenny Yin, Zhiting Mei, Tao Sun, Lihan Zha, Jeremy Bao, Miyu Yamane, Emily Zhou, Ola Shorinwa, Anirudha Majumdar · 2025
Read WoMAP: World Models For Embodied Open-Vocabulary Object Localization by Tenny Yin, Zhiting Mei, Tao Sun, Lihan Zha, Jeremy Bao, Miyu Yamane, Emily Zhou, Ola Shorinwa, Anirudha Majumdar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01600.txt.
Introduction
Tenny Yin and colleagues present WoMAP, world models for embodied open-vocabulary object localization. Their 2025 arXiv paper teaches robots to find named objects. A robot must explore unknown spaces efficiently to succeed. Imitation methods generalize poorly beyond their demonstrations. Vision-language models often propose physically ungrounded actions.
WoMAP trains policies inside a Gaussian splatting simulation pipeline. Real scenes convert to simulation and back to reality. Open-vocabulary detectors supply dense reward signals. A latent world model predicts dynamics and rewards. High-level action proposals get grounded through that model. Trials beat vision-language and diffusion baselines by wide margins. A real TidyBot robot confirmed the transfer from simulation.
Catalog record
| CREATOR | Tenny Yin, Zhiting Mei, Tao Sun, Lihan Zha, Jeremy Bao, Miyu Yamane, Emily Zhou, Ola Shorinwa, Anirudha Majumdar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01600
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WoMAP: World Models For Embodied Open-Vocabulary Object Localization Tenny Yin, Zhiting Mei, Tao Sun, Lihan Zha, Jeremy Bao, Miyu Yamane, Emily Zhou, Ola Shorinwa, Anirudha Majumdar Abstract Language-instructed active object localization is a critical challenge for robots, requiring efficient exploration of partially observable environments. However, state-of-the-art approaches either struggle to generalize beyond demonstration datasets (e.g., imitation learning methods) or fail to generate physically grounded actions (e.g., VLMs). To address these limitations, we introduce WoMAP ( Wo rld M odels for A ctive P erception): a recipe for training open-vocabulary object localization policies that: (i) uses a Gaussian Splatting-based real-to-sim-to-real pipeline for scalable data generation without the need for expert demonstrations, (ii) distills dense rewards signals from open-vocabulary object detectors, and (iii) leverages a latent world model for dynamics and rewards prediction to ground high-level action proposals at inference time. Rigorous simulation and hardware experiments demonstrate WoMAP’s superior performance in a broad range of zero-shot object localization tasks, with more than 9 × 9\times and 2 × 2\times higher success rates compared to VLM and diffusion policy baselines, respectively. Further, we show that WoMAP achieves strong generalization and sim-to-real transfer on a TidyBot. Keywords: Active Perception, World Models, Object Localization Figure 1: WoMAP uses a world model to ground high-level action proposals and maximize predicted rewards. In this example, given three high-level VLM proposals, WoMAP selects “look behind the bowl” as the optimal choice after evaluating outcomes of each action roll-out in latent space. 1 Introduction Perceptual activity in biological agents is inherently active and exploratory [ 1 , 2 ] . As an example, consider the task of open-vocabulary object localization , where an agent needs to approach a target object specified by natural language in a previously unseen environment. In such settings, humans will actively seek information guided by prior expectations to search efficiently. For example, when looking for keys, we preferentially inspect locations where they are most likely to be found, e.g., near the door or on the couch. However, reproducing intelligent search behavior for robots remains challenging, as it requires semantic understanding of open-vocabulary object descriptions and commonsense reasoning from partial observations in unfamiliar environments. While vision-language models (VLMs) provide useful heuristics for exploration [ 3 , 4 , 5 , 6 ] , effectively grounding these high-level action proposals to physical execution is a non-trivial problem. This grounding can be achieved via imitation learning methods [ 7 , 8 ] , which require large-scale expert demonstrations and can struggle to generalize beyond demonstration datasets. Alternatively, reinforcement learning (RL) [ 9 , 10 , 11 ] offers another route to grounding, but is challenging to employ without an accurate simulation environment. To address these challenges, we present WoMAP ( Wo rld M odels for A ctive P erception): a novel recipe for efficient active object localization that can be trained without expert demonstrations or online interactions with the environment ( Figure 1 ). In order to achieve this, we propose an approach that learns a latent world model [ 12 ] using three key ingredients ( Figure 2 , left). First, we introduce a scalable data generation pipeline based on Gaussian Splatting [ 13 ] that allows us to generate photorealistic data with broad coverage using real-world videos. Second, we propose a training framework that is reconstruction-free ; instead of using image reconstruction as a supervisory training signal (which can lead to poor generalization, training stability, and sample efficiency [ 14 ] ), we construct dense rewards from the confidence outputs of open-vocabulary object detectors and distill these into the latent space of the world model. Finally, we present an inference-time planning scheme that optimizes high-level action proposals from VLMs using the trained world model. Taken together, we contribute a novel approach to open-vocabulary object localization that can be trained in a data-efficient manner, generalize to novel scenes and object descriptions, and exploit commonsense reasoning abilities of VLMs. We demonstrate our approach on a suite of simulated and real-world object localization tasks with significant improvements ( 2\times – 9\times higher success rates) over baselines that only utilize imitation learning or VLMs. 2 Related Work Active Object Localization. Broadly, active object localization has been explored with both end-to-end approaches [ 15 , 16 ] , such as imitation learning (IL) [ 7 , 8 ] and reinforcement learning (RL) [ 9 , 10 , 11 ] , and modular approaches incorporating foundation models [ 3 , 4 , 5 ] . End-to-end methods map visual observations directly to actions but typically require large amounts of expert demonstrations or interactions to learn effective exploration behaviors [ 15 , 7 ] and generalize poorly to new environments or tasks [ 17 ] . In contrast, WoMAP does not require on-policy demonstrations or interactions with the environment, and leverages a latent world model to plan sequences of actions at inference time in order to achieve strong generalization across tasks and sim-to-real gaps. Modular approaches incorporate foundation models such as pre-trained object detectors or VLMs [ 4 , 5 , 18 ] to reason over observations and plan exploration. However, they rely heavily on the accuracy of each system component and require engineered spatial scene representations to ground executions in the real world. Instead, WoMAP directly optimizes VLM outputs within a learned environment model, offering a light-weight solution for grounded actions with minimal reliance on external representations. Finally, in terms of task setup, most prior work focuses on simulated indoor navigation, utilizing rich contextual cues (e.g., sofas are more likely to be in the living room) and restricting action spaces for tractability. In contrast, our framework makes no such assumptions and tackles more general settings requiring full 6-DoF camera control to locate objects in cluttered scenes. World Models for Robotics. World models have become increasingly prominent in robotics, providing predictive foresight in learning action-conditioned dynamics for long-horizon planning [ 19 , 20 , 14 , 21 ] . To capture environment dynamics that generalize to test-time distributions, existing works typically rely on large quantities of uncurated data [ 20 , 14 ] , expert demonstrations [ 21 , 22 ] , or on-policy interactions [ 23 ] , which are labor-intensive and costly to collect in real-world settings. In contrast, WoMAP learns a policy-agnostic environment model from synthetically generated offline data via Gaussian Splatting [ 13 ] . Many existing works also employ an image reconstruction loss to provide dense learning signals [ 20 , 24 ] , which often results in greater model complexity and unstable training. Without image reconstruction, WoMAP instead leverages dense reward signals distilled from pretrained models to encode rich visual, spatial, and semantic information in a pretrained latent representation, improving both scalability and robustness for downstream tasks. 3 Method 3.1 Problem Formulation We consider an open-vocabulary object localization task with a robot equipped with an onboard RGB camera with a six degree-of-freedom (6-DoF) action space, operating in an environment E\in\mathcal{E} . We model the problem as a partially observable Markov decision process (POMDP), where given the current state of the environment and the robot, the camera returns a partial observation o_{t} under sensing uncertainty, occlusions, and limited field of view. At each time step, the robot selects