Persistent Object Gaussian Splat (POGS) for Tracking Human and Robot Manipulation of Irregularly Shaped Objects

Justin Yu, Kush Hari, Karim El-Refai, Arnav Dalal, Justin Kerr, Chung Min Kim, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg, https://berkeleyautomation.github.io/POGS · 2025

Read Persistent Object Gaussian Splat (POGS) for Tracking Human and Robot Manipulation of Irregularly Shaped Objects by Justin Yu, Kush Hari, Karim El-Refai, Arnav Dalal, Justin Kerr, Chung Min Kim, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg, https://berkeleyautomation.github.io/POGS (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05189.txt.

Introduction

Justin Yu and co-authors present a system that tracks odd shaped objects as people and robots move them, in a paper from 2025. Knowing where an object is, and how it is turned, is key for grasping and assembly. Their system builds a rich three dimensional model from a first set of views.

After that first scan, a single stereo camera updates the model pose in real time, with no need for a prior design file. The model carries meaning and visual traits that help it follow the object. Tests showed up to twelve resets of an object in a row, and recovery from most slips of a tool in the grip. Language based commands for handling objects are also supported.

Catalog record

CREATORJustin Yu, Kush Hari, Karim El-Refai, Arnav Dalal, Justin Kerr, Chung Min Kim, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg, https://berkeleyautomation.github.io/POGS
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05189

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Persistent Object Gaussian Splat (POGS) for Tracking Human and Robot Manipulation of Irregularly Shaped Objects Justin Yu, Kush Hari, Karim El-Refai, Arnav Dalal, Justin Kerr, Chung Min Kim, Richard Cheng, Muhammad Zubair Irshad, Ken Goldberg, https://berkeleyautomation.github.io/POGS Abstract Tracking and manipulating irregularly-shaped, previously unseen objects in dynamic environments is important for robotic applications in manufacturing, assembly, and logistics. Recently introduced Gaussian Splats [ 1 ] efficiently model object geometry, but lack persistent state estimation for task-oriented manipulation. We present Persistent Object Gaussian Splat (POGS), a system that embeds semantics, self-supervised visual features, and object grouping features into a compact representation that can be continuously updated to estimate the pose of scanned objects. POGS updates object states without requiring expensive rescanning or prior CAD models of objects. After an initial multi-view scene capture and training phase, POGS uses a single stereo camera to integrate depth estimates along with self-supervised vision encoder features for object pose estimation. POGS supports grasping, reorientation, and natural language-driven manipulation by refining object pose estimates, facilitating sequential object reset operations with human-induced object perturbations and tool servoing, where robots recover tool pose despite tool perturbations of up to 30°. POGS achieves up to 12 consecutive successful object resets and recovers from 80% of in-grasp tool perturbations. I Introduction In environments like factories, workshops, or homes, robots must not only successfully identify and manipulate objects but also adapt to changes in object pose over time. The challenge is greater when dealing with irregularly shaped objects for which obtaining an accurate Computer-Aided Design (CAD) model is impractical. While any physical object can in principle be CAD-modeled, this process is often labor-intensive and may require reaching out to manufacturers or purchasing specialized scanning equipment. Approaches that rely on predefined CAD models struggle in scenarios where such models are unavailable, limiting adaptability to previously unseen objects [ 2 , 3 , 4 ] . Traditional and deep RGBD or point cloud object tracking methods are attractive as components of state estimators for robotic manipulation because they do not require predefined meshes or CAD models [ 5 , 6 ] . However, many of these approaches fail to effectively integrate geometric information across multiple object viewpoints or timesteps, and do not address the estimation or reconstruction of occluded object regions based on prior information. As a result, they struggle to maintain a persistent and holistic object representation over time. Fig. 1 : Autonomous Object Manipulation and Tracking with POGS Unified Representation (Top) A robot autonomously performs a pick and place primitive to move the shoe onto a shoebox given input natural language pick query ”shoe” and place query ”shoebox”. (Bottom) A POGS unified representation enables language querying, grasp sampling, and continuous tracking of irregular objects as they move. Implicit 3D representations like NeRFs [ 7 ] offer high-quality scene reconstructions but are ill-suited as a representation for dynamic scenes where objects may be moved and re-oriented. Recently, Gaussian Splatting [ 1 ] was introduced to create high quality 3D reconstructions by explicitly modeling scenes as a set of 3D gaussians that can be partitioned to allow rigid transforms on object-level clusters. Fig. 2 : POGS Pipeline After capturing multiple images of a scene using a robot wrist-mounted ZED mini, POGS segments objects using Detic, extracts DINO features, and embeds language through CLIP. Training images are used to optimize a 3DGS, and features extracted from 2D foundation models are distilled into feature fields, producing our POGS unified representation. During robot object reset and tool servoing, the POGS is updated based on depth geometry and DINO tracking features. To enable online state estimation, tracking, and manipulation of unseen objects in dynamic environments, we present Persistent Object Gaussian Splat (POGS), an editable object-centric feature field representation with embedded language features, self-supervised visual features, and object-level grouping features to support robot manipulation. POGS uses 3D Gaussian Splatting (3DGS) to model full 3D geometry of irregular objects, allowing for continuous updates as the scene evolves. By embedding features from encoders and detectors pre-trained on internet-scale datasets such as CLIP [ 8 ] , DINO [ 9 ] , and Detic [ 10 ] , POGS can respond to open-vocabulary natural language queries and also identify, track, and manipulate objects even without any predefined models. As such objects are moved by humans or robots, POGS can update their state online, allowing for flexible, multi-step tasks that require continuous interaction with dynamic objects, eliminating the need to re-scan the environment. This paper makes the following contributions: • Persistent Object Gaussian Splat (POGS), a novel feature field representation for tracking and manipulating previously unseen irregularly shaped objects. • A robot system for creating and using POGS to perform object-reset and tool servoing tasks. • Physical robot experiments on object reset with an average pose error of 2.92 cm. • Physical robot experiments for tool servoing where targets are moved up to 30 \degree and the tool can recover from human perturbations 80% of the time. II Related Work II-A Feature Fields for Robotics Recent advances in foundation vision models such as CLIP [ 8 ] and DINO [ 9 ] have enabled many methods that perform robot manipulation from visual features. Works [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] have used CLIP with point-based fusion methods to build open-vocabulary 3D representations. However, visual occlusion and multi-view pose misalignment can hinder consistent semantic fusion across the 3D scene. To address this, more recent approaches such as DFF [ 18 ] and LERF [ 19 ] have proposed distilling learned features into neural radiance fields (NeRFs) [ 7 ] by aggregating information across multiple views and scales. F3RM [ 20 ] and LERF-TOGO [ 21 ] extend these works respectively for robot manipulation. However, NeRF-based representations are limited by NeRF’s training speed and implicit spatial representation, making it impossible to update when objects move without further scene-scale optimization. Works like Dex-NeRF [ 22 ] and Evo-nerf [ 23 ] attempt to address this by partially re-scanning scenes to account for object movement; however, this process remains computationally intensive, limiting its suitability for online updates. An alternative to NeRF is Gaussian Splatting [ 1 ] , which models scenes using explicit 3D Gaussian primitives, enabling faster training and rendering while maintaining high fidelity in 3D reconstructions. Recent works [ 24 , 25 , 26 , 27 ] have shown that Gaussian Splatting can also integrate semantic and grouping features. GaussianGrasper [ 28 ] extends these approaches to robotic manipulation by updating the scene representation after objects are moved, using the robot end-effector pick-and-place transform followed by a few views to fine-tune the Gaussian Splat. In this work, we develop a method capable of updating the scene where a human can also move the objects repeatedly without any partial re-scans of the scene. II-B Object Tracking for Manipulation Object pose estimation networks [ 29 , 30 , 31 , 32 ] are able to track the 6DOF pose of an object of interest, but typically not multiple objects at once in a scene without scaling compute requirements. Using Gaussian Splatting, several works [ 33 , 34 , 35 ] collect image data from one or multiple views

Read the complete text