Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction

Yuanbo Wang, Zhaoxuan Zhang, Jiajin Qiu, Dilong Sun, Zhengyu Meng, Xiaopeng Wei, Xin Yang · 2025

Read Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction by Yuanbo Wang, Zhaoxuan Zhang, Jiajin Qiu, Dilong Sun, Zhengyu Meng, Xiaopeng Wei, Xin Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13091.txt.

Introduction

Yuanbo Wang and six co-authors present Touch2Shape, a model that rebuilds 3D shapes from touch rather than sight. Their paper from twenty twenty-five notes that image-based diffusion models grasp global shape well. They miss fine local detail, and they suffer under blocked views and poor light. Touch sensors read local surface form directly, with no such limits.

A touch embedding module feeds tactile images into the diffusion model as a compact code. A fusion module then sharpens the rebuilt shape with that code. For active search, reinforcement learning trains a policy that picks where to touch next. The diffusion code guides that policy through a novel reward. Tests in look and in numbers back the rebuild quality. The touch policy lifts scores further. The work suits robots that must handle objects they cannot fully see.

Catalog record

CREATORYuanbo Wang, Zhaoxuan Zhang, Jiajin Qiu, Dilong Sun, Zhengyu Meng, Xiaopeng Wei, Xin Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction Yuanbo Wang, Zhaoxuan Zhang, Jiajin Qiu, Dilong Sun, Zhengyu Meng, Xiaopeng Wei, Xin Yang Abstract Diffusion models have made breakthroughs in 3D generation tasks. Current 3D diffusion models focus on reconstructing target shape from images or a set of partial observations. While excelling in global context understanding, they struggle to capture the local details of complex shapes and limited to the occlusion and lighting conditions. To overcome these limitations, we utilize tactile images to capture the local 3D information and propose a Touch2Shape model, which leverages a touch-conditioned diffusion model to explore and reconstruct the target shape from touch. For shape reconstruction, we have developed a touch embedding module to condition the diffusion model in creating a compact representation and a touch shape fusion module to refine the reconstructed shape. For shape exploration, we combine the diffusion model with reinforcement learning to train a policy. This involves using the generated latent vector from the diffusion model to guide the touch exploration policy training through a novel reward design. Experiments validate the reconstruction quality thorough both qualitatively and quantitative analysis, and our touch exploration policy further boosts reconstruction performance. Figure 1 : (1) Exploring the target object and capturing the tactile image to reconstruct the 3D shape. We trained a diffusion model to obtain a low-dimensional and compact latent vector, which is used for predicting next touch location and reconstructing the target shapes. The numbers \scriptsize{1}⃝ with black arrows and \scriptsize{2}⃝ with greeen arrows in the diagram represent two consecutive time steps. We only generate the full reconstruction at the final step. (2) As the touch exploration progresses, the reconstruction results gradually approach the ground truth. (3) Reconstruction results compared with ActiveVT [ 34 ] (under visual-tactile settings). 1 Introduction 3D generation and reconstruction tasks have emerged as key focal points in the fields of computer vision and graphics, offering valuable applications in areas like autonomous driving and robot interactions characterized by environmental occlusions and camera measurement errors [ 6 , 32 , 5 ] . However, acquiring 3D data presents greater challenges and costs compared to 2D image and text data. This underscores the critical importance of ongoing research into 3D reconstruction and generation. Diffusion models have garnered substantial attention in generative tasks for their innovative approaches and promising results. While their impact has been particularly notable in 2D image generation [ 10 , 27 ] , these models are also being applied to 3D tasks. Pioneering projects such as SDFusion [ 5 ] and DiffusionSDF [ 6 ] have demonstrated the ability to create the 3D shapes using visual images or multi-modal data. However, the current focus is on 3D shape reconstruction based on predetermined partial observations. This limitation presents challenges on two fronts. Firstly, while current methods can estimate the overall shape of a target from limited data, they may overlook the local details of objects, making it challenging to generate complex shapes. This motivates further research into 3D reconstruction across new modalities. Secondly, practical scenarios present challenges where environmental factors like occlusions and varying lighting conditions impede the complete extraction of crucial local information. The complexity of real-world conditions underscores the necessity of actively exploring the target to capture information and facilitate the reconstruction process. In this work, we employ a simulated robotic arm guided by a trained policy model to touch the target, enabling the acquisition of tactile images to reconstruct the target through touch interaction. Tactile images provide local 3D shape information, including spatial contact points and detailed shape information, enhancing the reconstruction of local details, while visual data assists in predicting overall shape characteristics. Especially, we propose Touch2Shape, a touch-conditioned diffusion model for shape exploration through continuous touch and shape reconstruction by integrating all captured tactile images (along with the optional visual image). The reason for utilizing the diffusion model is to leverage its powerful generative capabilities to assist the model in generating a latent shape when the initial data perception is limited. The latent shape is then used to guide the policy model to predict beneficial touch locations for reconstruction based on potential missing areas. To train the exploration policy, we integrate the diffusion model with reinforcement learning and design a reward function. In contrast to previous methods, our approach not only produces superior 3D reconstruction outputs but also eliminates the need to generate the final shape at every step. Due to the fact that the diffusion model can generate a low-dimensional and compact latent space, we only generate the full reconstruction of the target object at the final step. Extensive experiments validate the effectiveness of our method, demonstrating significant improvements in both reconstruction performance and the ability to improve reconstruction quality through touch exploration. The main contributions of this article are as follows: • We propose Touch2Shape, a touch-conditioned 3D diffusion model for shape exploration and reconstruction, utilizing the latent vector to guide the touch location planning and shape decoding. • Touch2Shape utilizes contrastive touch encoder to embed touch information and shapes in a joint space. • We propose a touch shape fusion module to optimize the reconstructed shape using touch information. • We combine the diffusion model with reinforcement learning for shape exploration policy training and design the corresponding reward function. 2 Related Work Touch-based 3D Object Reconstruction. There are a lot of works addressing 3D shape reconstruction from visual signals. The approaches in this field vary depending on the type of visual input utilized, such as single view RGB images [ 14 ] , multi-view RGB images [ 8 ] , and depth images [ 2 ] . Additionally, the types of 3D representations they predict also differ, encompassing aspects such as voxels [ 24 ] , point clouds [ 13 ] , meshes [ 41 ] , and signed distance functions [ 26 ] . However, due to the high-dimensionality of the observation space and the interference of occlusions and external lighting conditions, training computer vision algorithms for object manipulation faces significant challenges. Recently, some researchers use high-resolution tactile perception such as DIGIT [ 22 ] and Gelsight sensor [ 45 ] to accomplish 3D object reconstruction. This type of perception, compared to vision-based perception, can obtain rich contact information and remains effective in the face of occlusions, transparent or reflective object materials. TouchSDF [ 7 ] predicts the local 3D shape corresponding to the tactile image and completes the 3D expression of the object by training an implicit neural function to represent the signed distance. Smith et al. [ 33 ] propose the first approach for reconstruction using both vision and touch, highlighting their complementary nature. Combining vision and touch modalities can yield richer information and improve 3D shape reconstruction. In this work, our Touch2Shape model trains a touch-conditioned diffusion model for 3D shape reconstruction, supporting settings with both tactile only and visual-tactile inputs. Figure 2 : We pretrained (a) the shape encoder and decoder, (b) the touch CNN model that is used for touch chart prediction, and (c) the contrastive touch encoder. During the training of (d) the touch-conditioned 3D diffusion model, we kept the parameters of the pretrained modules fixed and train a touch-conditioned 3D diffusion model

Read the complete text