TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Objects in Contact-Rich Scenes

Wen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin · 2025

Read TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Objects in Contact-Rich Scenes by Wen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22882.txt.

Introduction

Wen Yang and six co-authors present in 2025 TwinTrack, a system that follows an unknown object's full position and orientation in real time while hands or robots jostle it. Cameras alone lose the object when fingers cover it or a knock blurs the frame, exactly the moments that matter in dexterous manipulation.

TwinTrack fuses vision with contact physics: it builds a quick simulation twin of the scene from real data, then lets the physics of touch correct the visual track. The loop runs both ways between reality and simulation. The aim is steady six-degree tracking of objects never seen before, mid-grasp.

Catalog record

CREATORWen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TwinTrack: Bridging Vision and Contact Physics for Real-Time Tracking of Unknown Objects in Contact-Rich Scenes Wen Yang, Zhixian Xie, Yiting Wang, Abhijit Tadepalli, Heni Ben Amor, Shan Lin, Wanxin Jin Abstract Real-time tracking of previously unseen, highly dynamic objects in contact-rich scenes, such as during dexterous in-hand manipulation, remains a major challenge. Pure vision-based approaches often fail under heavy occlusions due to frequent contact interactions and motion blur caused by abrupt impacts. We propose TwinTrack , a physics-aware perception system that enables robust, real-time 6-DoF pose tracking of unknown dynamic objects in contact-rich scenes by leveraging contact physics cues. At its core, TwinTrack integrates Real2Sim and Sim2Real. Real2Sim combines vision and contact physics to jointly estimate object geometry and physical properties: an initial reconstruction is obtained from vision, then refined by learning a geometry residual and simultaneously estimating physical parameters (e.g., mass, inertia, and friction) based on contact dynamics consistency. Sim2Real achieves robust pose estimation by adaptively fusing a visual tracker with predictions from the updated contact dynamics. TwinTrack is implemented on a GPU-accelerated, customized MJX engine to guarantee real-time performance. We evaluate our method on two contact-rich scenarios: object falling with environmental contacts and multi-fingered in-hand manipulation. Results show that, compared to baselines, TwinTrack delivers significantly more robust, accurate, and real-time tracking in these challenging settings, with tracking speeds above 20 Hz. Project page Fig. 1: TwinTrack is a physics-aware perception framework for robust, real-time tracking of unseen, dynamic objects in contact-rich scenes with motion blur (Cols. 1–4) and heavy occlusion (Cols. 5–6). The framework integrates two key components: Real2Sim , which combines the complementary strengths of vision and contact dynamics to jointly estimate an object’s geometry and physical properties; and Sim2Real , which performs robust object pose estimation through adaptive fusion of visual tracking and contact simulation. video link I INTRODUCTION Real-time 6-DoF pose tracking of previously unseen, highly dynamic objects in contact-rich scenes is fundamental to robotic manipulation [ 1 , 2 ] and scene understanding [ 3 ] . Contact-rich scenes typically involve frequent object-robot-environment interactions and abrupt motion changes, such as jumps in the object velocity profile, causing large occlusions and motion blur. Pure vision-based tracking [ 4 ] struggles with the incomplete and/or degraded observations. Integrating contact physics cues [ 5 ] can overcome these challenges. Simulated contact dynamics, when aligned with reality, anticipate object poses in highly occluded frames and compensate for degraded visual tracking. Furthermore, incorporating contact dynamics also bridges the gap between visual perception and digital physics simulation, enabling downstream applications such as model-based dexterous manipulation control [ 6 , 7 , 8 ] . Building physics-informed perception system from monocular RGB-D vision for unknown objects is challenging with real-time constraints. The system must jointly estimate object pose, geometry, and physical properties (e.g., mass, inertia, friction coefficients) from RGB-D inputs. The high-dimensional optimization space requires a modular optimization pipeline with GPU-accelerated implementation. Specifically, three major challenges need to be addressed: (i) Efficiently integrating the complementary benefits of vision and contact dynamics cues for geometry estimation. Vision provides dense geometric reconstructions, but these are often corrupted by noise and occlusion. In contrast, contact events imply highly accurate geometric information but are sparse and intermittent [ 9 ] . (ii) Contact interactions introduce non-smooth motion, which challenges gradient-based learning and optimization [ 10 ] . (iii) Real-time performance requires highly efficient implementation. To address these challenges, we propose TwinTrack (Fig. 1 ), a physics-aware perception framework that bridges vision and contact dynamics for robust, real-time 6-DoF tracking of dynamic unknown objects in contact-rich environments. TwinTrack takes monocular RGB-D images and robot proprioceptive data as input and outputs real-time 6-DoF object poses that are resilient to occlusions and degraded vision observations. The core contributions of TwinTrack include: (I) An architecture that closes the loop between Real2Sim and Sim2Real. Real2Sim reconstructs object geometry and aligns a contact dynamics simulator with temporal RGB-D observations. Sim2Real then leverages the aligned simulator to perform physics-aware 6-DoF pose tracking that remains robust under occlusions and degraded visual input. (II) Geometry reconstructed from RGB-D vision and refined by contact dynamics. In Real2Sim, a dense geometry representation of the object is first extracted from pure vision. Due to noise and occlusion, the geometry is often inaccurate or incomplete. To address this, we propose refining it by learning a geometry residual in the alignment of contact dynamics simulator, together with estimating physical properties of the contact dynamics. This two-phase process enables effective integration of visual and reliable contact-dynamics cues to derive contact-consistent geometry. (III) Real-time implementation and optimization. Real2Sim (slow backend) and Sim2Real (fast frontend) are implemented separately for real-time tracking performance. To handle the non-smooth contact dynamics in Real2Sim optimization, we use GPU-parallelized, sampling-based optimization. We developed a deeply customized MJX-based physics engine [ 11 ] for collision detection and contact dynamics alignment. We evaluate TwinTrack in two challenging contact-rich scenarios: object free-fall and in-hand manipulation, both involving highly dynamic motion, frequent occlusions, and contact impacts. Compared with the state-of-the-art visual-only tracker [ 12 ] , TwinTrack achieves physically plausible and robust tracking across diverse real-world objects while maintaining real-time performance (>20Hz). II Related Work II-A Visual Tracking and Geometry Reconstruction Vision-based estimation of object pose and geometry has been a long-standing focus in computer vision. Methods with strong priors (CAD models, template matching) [ 13 , 14 , 15 ] achieve high accuracy but lack generalization to unseen objects. More recent approaches aim to track previously unseen objects without any prior knowledge, either by integrating pre-trained image descriptors [ 16 ] or by jointly learning 3D representations during tracking via neural fields [ 4 , 12 ] . While vision-based approaches perform well in occlusion-free and moderately dynamic conditions [ 16 , 4 , 12 ] , they struggle in highly dynamic, contact-rich scenarios involving fast motion, abrupt changes, and frequent occlusion. Visual observations become degraded or incomplete, leading to unreliable tracking. Object geometry from visual data may be uncertain due to noise, causing inconsistencies in motion prediction [ 9 ] . This underscores the need to incorporate physics knowledge (contact dynamics) to enhance robustness and ensure physically plausible estimates. II-B Physics-Informed Tracking and Reconstruction Physics-informed tracking incorporates physics priors to enforce physical plausibility. Human motion capture (e.g., [ 17 ] ) uses simulation-based constraints and soft physics priors for improved realism, but often assumes known physical properties limiting generalization. More recent approaches (e.g., [ 18 ] ) focus on jointly learning geometry and physical parameters by minimizing simulation/prediction loss from differentiable simulators. For example, [ 3 ] proposes using Real2Sim to extract collision geometry from robot sensors and cameras, while [ 19 ] reconstructs interactive digital twins from object interaction videos. However, their scenarios involve relatively slow and less occluded motion. In contact-rich scenes with high occlusion and motion blur, visual geometry may be inaccurate, degrading dynamics learning. Thus, [ 20 ] proposes combining visual and contact cues for geometry inference under occlusion, but they focus on reconstruction rather than real-time tracking. Fig. 2: Overview of TwinTrack framework. Our framework includes two main components: Real2Sim for learning object geometry and simulation-ready contact physics, and Sim2Real for physics-aware real-time pose tracking. In Real2Sim , object visual geometry, represented as the

Read the complete text