ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing
Aditya Kamireddypalli, João Moura, Russell Buchanan, Matías Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy · 2025
Read ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing by Aditya Kamireddypalli, João Moura, Russell Buchanan, Matías Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16592.txt.
Introduction
Aditya Kamireddypalli and co-authors present in 2025 ContactFusion, a way to map part surfaces for robot builds by touch as well as sight. Tight fit builds fail when sight noise tops the fit gap. Force sense at the wrist can tell where the grip tool touched.
A sample based step turns force reads into touch spots on the tool. Those spots join sight clouds in a chance based surface map that can grow with new data. Sim tests first prove the touch step finds the right spot. In a peg in hole task, touch fusion cut the hole pose error in a clear way.
Catalog record
| CREATOR | Aditya Kamireddypalli, João Moura, Russell Buchanan, Matías Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing Aditya Kamireddypalli, João Moura, Russell Buchanan, Matías Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy Abstract Robotic assembly tasks such as peg-in-hole insertion require precise geometric reasoning, yet sensor noise in real-world systems often exceeds the tight tolerances required for successful insertion. Vision-based pose estimation alone can therefore lead to misalignment and failure. In this work, we propose ContactFusion , a probabilistic mapping framework that fuses depth sensing and force–torque measurements to estimate the geometry of insertion targets. Our method builds a Stochastic Poisson Surface Map (SPSMap), an uncertainty-aware implicit surface representation constructed using Stochastic Poisson Surface Reconstruction (SPSR). To incorporate contact information, we introduce a contact location estimator that converts force–torque measurements into spatial hypotheses over candidate contact locations on the robot end-effector. These hypotheses are fused with depth observations within a sequential reconstruction framework, enabling the map to be refined through both visual and contact interactions. We evaluate ContactFusion in simulation and on a real robotic system in a peg-in-hole setting. Our results show that SPSMap produces more accurate and geometrically consistent reconstructions, improving reconstruction F-score by up to 30–35%, while providing uncertainty estimates that enable active reconstruction strategies. I Introduction Dexterous manipulation often requires precise geometric reasoning; however, sensor noise in real robot applications often exceeds the necessary tolerances. Tasks such as part alignment, constrained sliding, snap-fitting, and peg-in-hole insertion demand accurate estimates of the relative pose between interacting objects. As tolerances tighten, millimeter-scale geometric errors can lead to jamming, excessive contact forces, or failure to complete the task. This makes geometric state estimation a crucial component for the success of current manipulation systems. Current manipulation pipelines determine the pose of the insertion target by geometric registration of RGB-D measurements against a reference model (e.g., CAD model) [ 6 ] , or using deep learning-based prediction methods [ 18 ] . While vision-based solutions can estimate the target’s pose up to the millimeter-level, they can be affected by sensor noise, environmental and robot-induced occlusions, producing misalignment. Exploiting passive mechanical compliance and robust control strategies, such as impedance or compliant control, can reduce contact forces and compensate for this misalignment, at the cost of introducing a tradeoff between compliance and trajectory tracking [ 7 ] . One way to tackle such a requirement for precision could be to complement global visual information with fine local contact information. Motivated by recent works [ 9 ] , we explore this problem from the perspective of building multi-modal scene representations. Fig. 1 : ContactFusion builds probabilistic maps of an insertion target. By building upon the Stochastic Poisson Surface Reconstruction method [ 13 ] in a sequential formulation and a new contact location model fused with depth observations, ContactFusion produces continuous estimates of the target’s surface and uncertainty that are up to 30% more accurate than prior approaches. In this work, we introduce ContactFusion , a probabilistic mapping framework that builds a geometric representation of the insertion target by fusing depth and contact measurements. Our framework integrates these different sensor modalities using the Stochastic Poisson Surface Reconstruction (SPSR) method [ 13 ] , producing maps that explicitly model the surface of the target object and its uncertainty. We experimentally demonstrate that by combining vision and contact into this probabilistic representation, we can improve estimates of the target’s shape by 30% compared to existing methods, uncertainty estimates of the geometry, and also enable further directions such as active perception. Our specific contributions are: • We present a probabilistic mapping framework which combines vision and contact sensing using the Stochastic Poisson Surface Reconstruction method, into an uncertainty-aware representation— SPSMap . • We introduce a contact location estimator to transform force/torque measurements into spatial likelihoods over candidate surface contact locations. • We validate our framework through simulation and real-world experiments, demonstrating up to 30% improvements in 3D reconstruction accuracy of the insertion target with respect to alternative mapping baselines. • We further demonstrate how SPSMap enables uncertainty-driven active perception, accelerating the identification of the insertion target. II Related Work II-A Manipulation-enhanced Mapping We first focus on prior mapping methods which are explicitly focused on the manipulation setting. Interactive Perception [ 1 ] is a general approach whereby an agent gains information about its environment through continuous perception and interaction while solving a task. This makes it ideal for insertion tasks, where the robot continuously perceives the insertion target and interacts with it. Inspired by this approach, we focus on the problem of representing the insertion target from vision and contact sensing. The work by [ 5 ] is one of the earliest works that merged visual and tactile sensing into a common scene representation. Building upon a Gaussian Process Implicit Surfaces (GPIS) [ 19 ] representation, they were able to combine haptic and 3D sensing while explicitly modelling uncertainty in the scene reconstruction. The approach was later extended to online estimation of 3D surfaces, demonstrating it in robotic grasping tasks [ 2 ] , as well as for estimation of surface and object pose from tactile sensing [ 14 ] . Occupancy maps are a well-established map representation, that has also been used for fusing contact and visual sensing. [ 11 ] used an occupancy map to to infer geometry and pose of objects using tactile and depth sensing, achieving improved manipulation efficiency within the context of an active perception pipeline [ 11 ] . The occupancy probability stored in occupancy maps has also been used to guide non-prehensile maneuvers in kinematically-constrained spaces using information gain strategies [ 4 , 10 ] . More recently, neural implicit representations have also opened new avenues for joint object surface reconstruction and pose estimation for in-hand object manipulation [ 15 ] . However, these approaches are data intensive, requiring an offline training stage, and do not encode reconstruction uncertainty as part of the same model. In this work, we explicitly aim to build a representation from online streams, fusing contact and vision sensing into a common representation for surface’s geometry and uncertainty. To achieve this, we build upon the GPIS paradigm and extend it using the Stochastic Poisson Surface Reconstruction (SPSR) [ 13 ] . This enables us not only to fuse different sensing modalities, but also to perform statistical queries like on-surface probability and occupancy probability. II-B Contact Location Estimation While the previous approaches have fused contact sensing with vision, they assume that the contact location is known. Previous works have proposed different strategies to determine it, depending on the sensor modalities available in each specific robot setup. When force/torque (F/T) sensors are available, it has been proposed to use precomputed force-torque maps in Euclidean space [ 17 ] . These maps relate expected force-torque values to Euclidean increments in robot end-effector positions . This map is computed offline using CAD information, with force-torque sensory information being used to query the map online. [ 3 ] extended this idea by constructing a map using force-torque information in configuration space. When no F/T sensors are available, the robot kinematics model can be used as a proxy for location, up to dynamic effects. [ 20 ] followed this approach to implement a state estimation pipeline to determine the location of a known