Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

Jiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin · 2026

Read Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation by Jiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28812.txt.

Introduction

Jiahe Pan, Stelian Coros, Jitendra Malik, and Toru Lin introduce Center-of-Pressure, a tactile code grounded in physical rules keeping dense contact information for sim-to-real transfer. Current sim-to-real methods shrink tactile data into coarse features, losing richness complex handling needs. A sensor tuning scheme using smooth motion estimates taxel angles without force readings. Across peg-in-hole fitting and ball balance, policies tuned on Center-of-Pressure reach zero-shot sim-to-real transfer with a multi-finger hand, beating binary-contact and raw rivals.

Catalog record

CREATORJiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54686
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28812

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation Jiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin Abstract A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods often mitigate this gap by simplifying tactile data into coarse low-dimensional features – sacrificing the richness required for complex manipulation. In this work, we introduce C enter- o f- P ressure ( CoP ), an effective tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer. To support this representation, we propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two blind, challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Across both tasks, policies conditioned on CoP achieve zero-shot sim-to-real transfer on a multi-fingered hand, and outperform both coarse binary-contact and raw-taxel baselines. Analysis of learned policy states further suggests that CoP-conditioned policies encode task-relevant physical properties, such as object mass, as an emergent byproduct of control. Project site and supplementary video . Keywords: Dexterous Manipulation, Tactile Representation, Sim-to-Real 1 Introduction A central challenge in robotics is contact-rich dexterous manipulation. While recent advances in imitation learning from large-scale human demonstrations [ 2 , 9 , 48 ] have show impressive progress, the scalability of these approaches is fundamentally limited by the high cost and human effort required for real-world data collection. Sim-to-real reinforcement learning (RL) offers a compelling alternative through large-scale, autonomous learning in simulation. However, its success has been largely confined to relatively simple tasks [ 1 , 16 , 6 , 29 , 28 ] , where policies can be learned from low-dimensional features that reliably transfer from simulation to the real world. Among common input modalities, tactile inputs are often avoided or heavily simplified, as accurately simulating tactile sensing is notoriously difficult. Tactile sensor responses depend on complex, high-dimensional, and often unmodeled physical processes, and vary widely due to the lack of standardized sensor designs. As manipulation tasks become more contact-rich and demand greater precision, control policies must reason over more fine-grained physical interactions – making tactile sensing more essential for policy learning. The lack of an effective tactile representation thus becomes a key bottleneck in extending sim-to-real RL to more challenging settings. At the core of this bottleneck lies a representation trade-off: simplistic tactile features [ 31 , 47 , 35 ] enable more robust sim-to-real transfer, while representations closer to raw tactile signals [ 19 , 38 , 46 , 4 ] contain richer contact information necessary for more complex manipulation. To balance this trade-off, we introduce C enter- o f- P ressure ( CoP ), a physics-grounded representation that summarizes local tactile information as a 3D contact force vector and a 3D contact location. CoP provides a middle ground for effective tactile sim-to-real transfer: it is compact enough to align across simulation and hardware, yet expressive enough to preserve force and location information needed for contact-rich dexterous control. To support this representation, we further propose a sensor calibration method based on differentiable dynamics, enabling the estimation of CoP without requiring ground-truth force measurements. Existing evaluations can make it difficult to isolate the role of touch, either because policies rely heavily on vision [ 7 , 18 , 37 ] or because the tasks involve relatively simple periodic manipulation [ 34 , 5 , 45 , 20 ] . To address this, we introduce two challenging blind manipulation tasks that explicitly require tactile feedback with minimal visual cues: peg-in-hole insertion and ball balancing. We evaluate our approach through extensive experiments, demonstrating that policies conditioned on CoP achieve successful zero-shot sim-to-real transfer on multi-fingered hands and significantly outperform both simplified and raw tactile baselines. Importantly, we find that learned policies implicitly capture underlying physical properties – such as object mass – highlighting the potential of physically grounded representations for enabling more general and scalable manipulation. 2 Background Imitation Learning for Dexterous Manipulation . Recent advances in teleoperation [ 9 , 30 , 44 , 49 ] and imitation learning [ 10 , 26 ] have enabled remarkable progress in dexterous manipulation. However, teleoperation data remains costly to scale, and achieving high levels of success and robustness requires large real-world demonstration datasets [ 50 , 25 , 27 ] , making purely supervised methods expensive for reaching human-level dexterity on complex, dynamic tasks. Sim-to-Real Transfer of Contact-Informed Policies . Sim-to-real RL has demonstrated success for learning manipulation policies in simulation and transferring onto real-world hardware. Earlier works have used proprioceptive joint control error as an implicit representation of external contact [ 34 ] ; however, the resulting performance gains have been found to be limited [ 8 ] . The recent emergence of diverse tactile sensors has inspired many explicit contact representations, primarily simplified contact representations such as binary [ 31 , 47 ] and discretized ternary signals [ 35 ] . While these representations can transfer reliably, the coarse discretization discards high-fidelity contact information that is critical for fine-grained contact-rich control. A few recent works have attempted to directly model and simulate the raw tactile sensor behaviors [ 46 , 37 , 12 ] , often by training a tactile encoder to compress the raw simulated sensor activations to a latent observation vector for the policy. Although these approaches better capture the complexity of real-world tactile signals, the resulting representations are often sensor-specific, less interpretable, and harder to align with simulator contact quantities. A recent work [ 45 ] employed a sensor-specific contact representation consisting of force scalar magnitude and contact position in polar coordinates. However, a learned tactile encoder is still used, and the evaluation is limited to only in-hand rotation. 3 Methodology 3.1 Physics-Aware Contact Representation We propose Center-of-Pressure (CoP) 1 1 1 We use the term CoP to denote the resultant contact wrench reduced to a single force vector applied at a centroidal contact point; this approximation is intended as a compact local contact descriptor, rather than a complete representation of arbitrary multi-contact pressure distributions. as a physics-aware contact representation. On a round fingertip of a robot hand (Fig. 1(a) ), CoP consists of a 3D force vector {}^{\mathcal{S}}f_{\rm{cop}}\in\mathbb{R}^{3} representing the total contact force acting on the robot link, and a 3D Cartesian contact position {}^{\mathcal{S}}p_{\rm{cop}}\in\mathbb{R}^{3} . Both the force vector and the contact point are expressed in the sensor frame \mathcal{S} . Most rigid-body simulators such as IsaacSim [ 33 ] and MuJoCo [ 41 ] provide pairwise contact information between simulation bodies, including 3D contact force vector and the contact position expressed in the world frame. (a) Left : CoP consists of a contact force vector f_{cop} and contact point p_{cop} , both expressed in \mathcal{S} . Right : Transformation between taxel frame \mathcal{T}_{i} and sensor frame \mathcal{S} . (b) (1) A contact force f_{\rm{cop}} is decomposed into normal f_{n} and shear f_{s} components. We model the (2) normal f_{i,n} and (3) shear f_{i,s} forces on each taxel i under surface deformation. (4) Components are combined to obtain the resulting taxel force f_{i} .

Read the complete text