Strategic Vantage Selection for Learning Viewpoint-Agnostic Manipulation Policies

David S. Hippocampus · 2025

Read Strategic Vantage Selection for Learning Viewpoint-Agnostic Manipulation Policies by David S. Hippocampus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12261.txt.

Introduction

A robot taught from one camera seat fails when the seat moves. The head of this 2025 work, by Sreevishakh Vasudevan, Som Sagar, and Ransalu Senanayake, offers a cure that picks its seats with care. The catalog, oddly, credits a name that matches no one on the byline and gives a flipped title; the head is followed here. Rather than film from a crowd of random seats, their frame treats seat choice as a hunt for the most telling views.

The hunt weighs fresh ground against known good spots in one smooth search, with math bounds on where it lands. A few chosen seats, used to fine-tune a trained arm, lifted success by a quarter for one policy kind. Gains held when the camera moved mid-task. Tests ran in both fake and real rooms.

Catalog record

CREATORDavid S. Hippocampus
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Strategic Vantage Selection for Learning Viewpoint-Agnostic Manipulation Policies David S. Hippocampus Abstract Vision-based manipulation has shown remarkable success, achieving promising performance across a range of tasks. However, these manipulation policies often fail to generalize beyond their training viewpoints, which is a persistent challenge in achieving perspective-agnostic manipulation, especially in settings where the camera is expected to move at runtime. Although collecting data from many angles seems a natural solution, such a naive approach is both resource-intensive and degrades manipulation policy performance due to excessive and unstructured visual diversity. This paper proposes Vantage , a framework that systematically identifies and integrates data from optimal perspectives to train robust, viewpoint-agnostic policies. By formulating viewpoint selection as a continuous optimization problem, we iteratively fine-tune policies on a few vantage points. Since we leverage Bayesian optimization to efficiently navigate the infinite space of potential camera configurations, we are able to balance exploration of novel views and exploitation of high-performing ones, thereby ensuring data collection from a minimal number of effective viewpoints. We empirically evaluate this framework on diverse standard manipulation tasks using multiple policy learning methods, demonstrating that fine-tuning with data from strategic camera placements yields substantial performance gains, achieving average improvements of up to 46.19% when compared to fixed, random, or heuristic-based strategies. 1 Introduction Vision-based robot learning critically depends on the quality, consistency, and comprehensiveness of visual input, making camera placement a decisive yet frequently overlooked factor in training robust manipulation policies [ 1 , 2 ] . The choice of camera viewpoint directly influences feature extraction, state estimation, and ultimately, policy performance. For instance, consider a robot tasked with picking objects from a cluttered table: a top-down camera initially offers a clear overhead view, but as the robot’s arm moves to grasp an object, it may obstruct the camera, complicating precise manipulation. Conversely, a side-view camera can continuously track the robot’s motion without obstruction but may fail to clearly represent essential object attributes such as length or orientation, crucial for successful grasping [ 3 ] . Figure 1: Starting from an initial camera viewpoint, we first define the model and search space, then sequentially fine-tune the policy at three additional selected viewpoints. Each fine-tuning step yields a measurable gain in task success rate rising from 37% at the first refine to 48 %, 64 %, and finally 83 % accuracy, illustrating how strategic viewpoint selection progressively enhances Despite the importance of viewpoints, most manipulation policies become highly specialized to their training viewpoints, performing reliably only when the camera is the same as in training. While suitable for static research settings, these policies fail in humanoid robots with constantly moving heads, mobile manipulators with the camera mounted on pan-tilt head or mobile bases, robots on moving assembly lines [ 4 , 5 ] , etc. One might assume that simply training on data from many random viewpoints enhances robustness, but in practice, excessive variation or over-augmentation increases the sample complexity and can lead to poor convergence. From the perspective of bias–variance tradeoff [ 6 ] , high-capacity models exposed to overly diverse data may overfit noise across viewpoints and struggle to learn consistent features. This phenomenon parallels findings in domain generalization, where large shifts between training distributions hinder model generalization [ 7 ] . Therefore, it is important to learn the manipulation policy from a few meaningful viewpoints. Instead of training a viewpoint-agnostic policy from scratch, we find it more stable to fine-tune a pretrained policy with some additional strategic views. This process also aligns with the prevailing trend of fine-tuning, as models continue to grow in size and complexity [ 8 ] . For example, a warehouse robot trained to grasp items from a fixed viewpoint pre-deployment may later need to adapt to dynamic environments such as variable shelf heights and mobile platforms at deployment [ 9 ] . This shift necessitates structured methods capable of systematically identifying a few informative camera perspectives, thereby balancing diversity in viewpoints with the stability required for effective learning. Since our goal is to find the optimal viewpoint from the space of all viewpoints, which increases the model’s performance across diverse camera perspectives, using the minimum number of samples as possible, we propose a Bayesian Optimization (BO)-driven framework for systematically identifying and combining training viewpoints to enhance policy performance. Instead of exposing the robot to all possible viewpoints at once, risking instability and poor convergence, we adopt a structured, iterative approach. This method systematically finds the most informative camera angles over multiple iterations, balancing exploration of new viewpoints with the stability required for effective learning. By doing so, our approach ensures that the robot learns from perspectives that enhance task performance while avoiding the pitfalls of excessive viewpoint variation. Our main contribution is a framework that iteratively finds additional camera viewpoints to progressively finetune an arbitrary manipulation policy to ensure the manipulation policy is agnostic to viewpoints. 2 Related Work Active vision in robotics: The paradigm of active vision, wherein an agent dynamically controls its viewpoint to enhance perceptual efficiency, has long been a foundational pursuit in robotics and computer vision [ 10 , 11 ] . Recent advances have leveraged deep learning to address the next best view (NBV) problem, where the agent selects optimal views to maximize information gain. Learning based methods have emerged, using reinforcement learning or uncertainty modeling [ 12 , 13 ] to guide viewpoint selection for tasks such as object recognition and 3D reconstruction [ 14 , 15 , 16 ] . Notably, recent works have substantially raised the bar. For instance, GenNBV [ 2 ] introduced a generalizable NBV policy that learns in a 5D action space, while SUGARL [ 4 ] learns intrinsic sensorimotor rewards to guide view selection under partial observability. Active Neural Mapping [ 17 ] and VIN-NBV [ 18 ] further refined the NBV task using receding horizon planning and view quality introspection networks. In multi-view tasks, Hou et al. [ 19 ] developed methods to select informative views for efficient 3D understanding [ 20 ] . Additionally, Bayesian methods for NBV planning [ 16 ] are gaining traction in robotics, such as in safe contact-based exploration [ 21 ] and dynamic view planning for robotic manipulation [ 5 ] . Recent studies have also explored attention-driven NBV strategies for targeted perception in complex environments [ 22 ] and affordance-driven NBV planning for robotic grasping in cluttered scenes [ 23 ] . While active vision aims to solve the problem of “where to look” at inference time, complementary to them, we address the problem of how to collect optimal data at the training stage to develop a perspective agnostic policy. Viewpoint selection for generalization: Beyond selecting views for immediate perceptual gain, recent research has explored how actively varying viewpoints can improve the robustness and generalization of learned visual representations. Jayaraman et al. [ 24 ] demonstrated that agents can learn viewpoint-invariant features by controlling the camera and predicting changes across views. Follow up work from Wu et al. [ 25 ] used neural predictors to estimate utility in active 3D reconstruction, while Lin et al. [ 26 ] leveraged neural implicit representations for efficient scene understanding. In parallel, multi-agent or collaborative view selection strategies

Read the complete text