PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, Anqi Li · 2025
Read PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies by Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, Anqi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18282.txt.
Introduction
Jesse Zhang and 8 co-authors let a vision-language model point a robot hand where it counts. Their 2025 method, PEEK, deals with a snag in robot policy learning: one policy must learn where to look, what to do, and how to move, all at once. The first two parts can be offloaded, they argue.
A tuned vision model marks two things on the robot's view: the path the hand should take, and the zones that matter for the task. Those marks work with any policy form. An auto-tag pipe built training data across many robot sets. Real tests gave large zero-shot gains for both big and small policies.
Catalog record
| CREATOR | Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, Anqi Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-18282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, Anqi Li Abstract Robotic manipulation policies often fail to generalize because they must simultaneously learn where to attend, what actions to take, and how to execute them. We argue that high-level reasoning about where and what can be offloaded to vision-language models (VLMs), leaving policies to specialize in how to act. We present PEEK (Policy-agnostic Extraction of Essential Keypoints) , which fine-tunes VLMs to predict a unified point-based intermediate representation: (1) end-effector paths specifying what actions to take, and (2) task-relevant masks indicating where to focus. These annotations are directly overlaid onto robot observations, making the representation policy-agnostic and transferable across architectures. To enable scalable training, we introduce an automatic annotation pipeline, generating labeled data across 20+ robot datasets spanning 9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot generalization, including a 41.4 × \times real-world improvement for a 3D policy trained only in simulation, and 2–3.5 × \times gains for both large VLAs and small manipulation policies. By letting VLMs absorb semantic and visual complexity, PEEK equips manipulation policies with the minimal cues they need— where , what , and how . Website at https://peek-robot.github.io . I Introduction Imagine walking through a crowded store when your child suddenly cries out, “I want the Labubu!” Though you’ve never heard the word before, context clues guide your eyes to the fuzzy toy on the shelf, and you effortlessly weave through the crowd to grab it. What makes this possible is not raw perception ability, but the ability to interpret ambiguous instructions and distill them into just the right cues— where to focus, what actions to take, and how to perform these actions at the low level. Similarly, if given where to focus and what motions to take, a robot manipulation policy should be able to achieve the visual robustness and semantic generalization necessary for open-world deployment by focusing only on how to perform actions. A common tactic for training manipulation policies is through imitation learning of human-collected robotics data [ 1 , 2 , 3 , 4 ] , which attempts to learn the where, what, and how all at the same time. Yet their performance degrades on novel objects, clutter, or semantic variations [ 5 , 6 ] , since the policy alone bears the burden of handling task, semantic, and visual complexity. Such failures often entangle the axes of where , what , and how —for example, grasping a distractor simultaneously reflects misplaced attention, an incorrect object choice, and a wrong motion. Fig. 1 : PEEK enables policy generalization by modulating minimal representations of where to focus and what to do for robust policy learning. Our key idea is to offload high-level reasoning to vision-language models (VLMs), which can excel at semantic and visual generalization [ 7 , 8 ] , leaving the policy to determine how low-level behavior should be executed. Instead of forcing the policy to directly parse raw images and instructions, a high-level VLM modulates the input representation to the low-level policy by providing: (1) a path that encodes what the policy should do, and (2) masks showing where to attend. By “absorbing” semantic and visual variation, the VLM provides the policy a simplified, annotated “peek” of the scene that gives the what and the where, while the policy only needs to learn how to perform the low-level actions. This intermediate representation helps policy execution inherit many of the VLM’s semantic and visual generalization capabilities. Our VLM-modulated representation is naturally policy-agnostic, allowing it to be applied to arbitrary image-input robot manipulation policies, including state-of-the-art RGB and 3D manipulation policies [ 9 , 1 , 3 ] . To concretely instantiate this insight into a practical algorithm, we introduce PEEK ( P olicy-agnostic E xtraction of E ssential K eypoints), which proposes a unified, point-based intermediate representation that trains VLMs to predict what policies should do and where to focus on. Specifically, we propose to finetune pretrained VLMs [ 10 ] to predict a sequence of points corresponding to (1) a path that guides the robot end-effector in what actions to take and (2) a set of task-relevant masking points that show the policy where to focus on (see Figure 1 ). During low-level visuomotor policy training and inference, we modulate the policy’s image observations by directly drawing these VLM-predicted paths and masks onto the image, allowing the policy to simply focus on how to act, rather than learning all three simultaneously. Doing so significantly bolsters policy generalization, combining the generality of high-level VLM predictions with the precision of low-level policy learning. In this paper, we instantiate a full-stack implementation of PEEK, from devising a scalable data annotation scheme that enables large-scale VLM finetuning on robotic datasets to representation-modulated training of low-level robot policies from simulation and real world data. In 535 real-world evaluations across 17 task variations, we demonstrate that PEEK consistently boosts zero-shot policy generalization: a 3D policy (3DDA [ 9 ] ) trained only in simulation achieves 41.4 \times higher success in the real world when guided by PEEK, and both large-scale vision-language-action models ( \pi_{0} [ 3 ] ) and small transformer-based policies [ 1 ] see 2–3.5 \times success rate improvements. These results demonstrate the power of using high-level VLMs to absorb task complexity, providing low-level policies with exactly the minimal cues they need for generalizable manipulation. II Related Works Object-Centric Representations. One approach to improving the visual generalization of imitation learning (IL) policies is to build object-centric representations [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] . Earlier works relied on human-selected abstractions or manual annotation [ 11 ] , while more recent methods leverage pre-trained, open-vocabulary segmentation models to visually isolate task-relevant objects [ 12 , 13 , 14 , 15 , 17 ] . Among these, ARRO [ 13 ] is closest to our work, proposing a policy-agnostic masking scheme using GroundingDINO [ 18 ] to filter images for task-relevant objects. However, we found in Section IV that such object detectors often fail in cluttered, realistic scenes. By contrast, PEEK queries a fine-tuned VLM to predict task-relevant masking points directly, resulting in more robust masks than using object-detection models due to the VLM’s extensive pre-training. Another approach, OTTER [ 16 ] , implements implicit masking by filtering CLIP image patches, but this approach is architecture specifc. PEEK’s policy-agnostic explicit masking allows us to integrate it with far more powerful policy backbones than OTTER, i.e., vision-language-action models like \pi_{0} [ 3 ] . Finally, while masking alone helps mitigate visual distractors, it alone cannot handle semantic variation; PEEK also provides explicit action guidance via predicted paths. Another line of object-centric methods relies on learning to decompose scenes into object-level representations in a self-supervised manner, e.g., via slot-attention [ 19 , 20 , 21 , 22 ] , which learns to map visual features into a set of discrete, object-centric “slots” through competitive attention mechanisms. However, these methods have not been applied to real-world robot manipulation settings and generally do not work zero-shot. PEEK’s use