Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
Kuancheng Wang 1 Vaibhav Saxena 1 Shuo Cheng 1 Yotto Koga 2 Danfei Xu 1,3 1 Georgia Institute of Technology 2 Autodesk Research 3 NVIDIA Website: see2act.github.io · 2026
Read Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation by Kuancheng Wang 1 Vaibhav Saxena 1 Shuo Cheng 1 Yotto Koga 2 Danfei Xu 1,3 1 Georgia Institute of Technology 2 Autodesk Research 3 NVIDIA Website: see2act.github.io (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23625.txt.
Introduction
Wang and colleagues teach a robot to move its camera to find hidden objects while it learns to act, in a 2026 paper on robot imitation learning. Most imitation methods assume everything relevant is visible from a fixed viewpoint. Real tabletops break that assumption: objects hide behind others, so the robot must search as well as grasp. Demonstrations mix both skills, which makes the viewpoint part hard to learn.
Their approach, See2Act, couples action prediction with viewpoint choice inside one diffusion process. Each step refines the planned action and the next camera pose together. Training anchors camera poses to keyframe actions in recorded demonstrations. In simulated Ravens and RLBench tasks the policy recovers useful viewpoints under severe occlusion, improving results by up to thirty four percent. Real world trials use demonstrations from a digital twin.
Catalog record
| CREATOR | Kuancheng Wang 1 Vaibhav Saxena 1 Shuo Cheng 1 Yotto Koga 2 Danfei Xu 1,3 1 Georgia Institute of Technology 2 Autodesk Research 3 NVIDIA Website: see2act.github.io |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23625
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation Kuancheng Wang 1 Vaibhav Saxena 1 Shuo Cheng 1 Yotto Koga 2 Danfei Xu 1,3 1 Georgia Institute of Technology 2 Autodesk Research 3 NVIDIA Website: see2act.github.io Abstract Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act , and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see , while learning how to act . We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability. Figure 1: See2Act couples seeing and acting in a single denoising loop. At each step the camera view and the predicted action are refined together: the action sets the next viewpoint (transition shown in white dotted trajectory), and the new viewpoint denoises the next action. This results in a policy that repositions the camera to reveal placement positions initially hidden from overhead views, achieving 95% zero-shot sim-to-real transfer on tasks with occlusion. Keywords: Robot Manipulation, Diffusion Models, Active Perception 1 Introduction Imitation learning (IL) for visuomotor control [ 15 , 18 ] typically assumes fixed viewpoints, conflating the abilities to see and act . In realistic settings, however, a robot must decide where to look , especially under occlusion when task-relevant objects are not initially visible. This introduces a fundamental challenge: demonstrations entangle two distinct behaviors – viewpoint selection and manipulation – which require different forms of supervision. While manipulation can often be learned from aligned state-action pairs, selecting informative viewpoints requires spatial reasoning over unseen regions of the scene. Simply augmenting demonstrations with additional viewpoints increases data requirements combinatorially, without teaching the policy which views are useful. Consequently, policies trained with limited data often fail in occluded environments, revealing a key limitation of current IL approaches. Recent methods have significantly improved data-efficient visuomotor control. Diffusion policies [ 3 ] model actions through conditional denoising, with further gains from 3D representations [ 25 ] , symmetry-aware architectures [ 22 , 17 ] , and camera-conditioned refinement [ 19 ] . Keyframe-based approaches [ 20 , 6 , 5 ] improve efficiency by supervising only task-critical waypoints. However, these methods remain passive in perception, assuming that the required visual information is already visible at inference time. As a result, policies with fixed or pre-defined viewpoints often fail under occlusion, where informative observations must be actively acquired. The missing capability is therefore not only data-efficient imitation learning, but policies that jointly learn where to see and how to act . The central insight of this work is that manipulation should be treated as a coupled process of action refinement and visual evidence acquisition, rather than action prediction from a fixed viewpoint. We introduce See2Act , a diffusion-based imitation learning framework that jointly refines robot actions and camera viewpoints. At test time, a wrist-mounted camera iteratively updates its pose alongside the denoised 6-DoF action, actively acquiring observations that reduce task ambiguity. This behavior emerges from training the diffusion model on target actions paired with viewpoint trajectories anchored to the same demonstration keyframe, enabling the policy to learn which viewpoints are most informative for precise manipulation under occlusion. Training is performed entirely in simulation using scalable viewpoint-agnostic data generation, and the resulting policy transfers zero-shot to real robots using depth observations to reduce the sim-to-real gap. In summary, our key contributions are: 1. We present See2Act, a viewpoint-agnostic imitation learning policy that learns viewpoint-action coupling by leveraging digital twins and a novel training strategy for keyframe-action prediction from offline manipulation datasets. 2. See2Act enables a test-time mechanism ( Active Viewpoint Inference ), that iteratively refines the robot’s camera pose and predicted action to autonomously resolve occlusions. 3. We create four new tasks based on the Ravens Benchmark [ 27 ] , on which See2Act achieves 91% average success rates while addressing the search-vs-manipulation learning disparity. We further evaluate on nine RLBench tasks [ 11 ] , achieving an average success rate of 81.1%. 4. Finally, we demonstrate zero-shot sim-to-real transfer on two real-world tasks with explicit occlusions, achieving average success rate of 95% without any real-world fine-tuning. 2 Related Works Diffusion models for visuomotor policies. Diffusion Policy [ 3 ] formulates visuomotor control as conditional denoising over action trajectories, enabling multimodal action generation. Subsequent works improve spatial generalization using point clouds [ 25 ] , SO(2) symmetry [ 22 ] , or SE(3) bi-equivariance [ 17 ] , often relying on privileged geometric information. C3DM [ 19 ] additionally couples denoising with fixation-based zooming into task-relevant regions, but limits camera motion to translational zoom along a fixed orientation. See2Act extends this to full 6-DoF camera trajectories, allowing active reorientation and repositioning to uncover occluded regions. Keyframe actions in robotics. Keyframe-based action representations decompose trajectories into critical waypoints, reducing supervision while focusing on decisive task moments [ 24 ] . Prior methods include voxelized point-cloud approaches such as PerAct [ 20 ] , view-based methods RVT/RVT-2 [ 6 , 5 ] , 3D feature-field prediction in Act3D [ 4 ] , and enhanced spatial-temporal encoding in PolarNet and HiveFormer [ 2 , 7 ] . However, all rely on fixed camera configurations and cannot actively acquire new views when task-relevant regions are occluded. Active perception. Active perception methods move sensors instead of relying on fixed views, differing mainly in how viewpoint control is learned. ViA [ 23 ] imitates human active perception strategies using a 6-DoF robotic neck, EyeRobot [ 13 ] learns gaze policies via reinforcement learning, and SPARTN [ 28 ] injects viewpoint perturbations offline using NeRFs. These methods treat viewpoint control separately from action prediction. In contrast, See2Act jointly denoises actions and camera poses within a single generative process, enabling active perception to emerge directly from the denoising loop without separate viewpoint policies, rewards, or planning stages. 3 Method Figure 2: Training and Inference Pipeline in See2Act. (Left: Training) Given demonstrations (extracted object states s and keyframe actions a_{0} ), we sample diffusion timesteps t , compute camera poses {C_{t}} , render observations O_{t} in a digital twin, and generate noisy actions \hat{a}_{t} by transforming a_{0} into each camera frame and adding Gaussian noise. The visual encoder and score network are trained jointly to predict noise \tilde{\epsilon} using an MSE loss. (Right: Inference) Starting from an overview view ( t=T ), See2Act iteratively refines both the action and camera pose. At each denoising step, it captures an observation, predicts noise, updates the action estimate, and computes the next camera pose. The final action is executed by the robot. Visuomotor policies with fixed viewpoints