DeSPITE: Exploring Contrastive De ep S keleton- P ointcloud- I MU- T ext E mbeddings for Advanced Point Cloud Human Activity Understanding

Thomas Kreutz, Max Mühlhäuser, Alejandro Sanchez Guinea · 2025

Read DeSPITE: Exploring Contrastive De ep S keleton- P ointcloud- I MU- T ext E mbeddings for Advanced Point Cloud Human Activity Understanding by Thomas Kreutz, Max Mühlhäuser, Alejandro Sanchez Guinea (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13897.txt.

Introduction

Thomas Kreutz, Max Mühlhäuser, and Alejandro Sanchez Guinea built a joint learning model in 2025 linking four kinds of human activity data. Laser ranging sensors offer a privacy-friendly way to watch activity, but multi-way pre-training for them is little explored. Their model ties laser point clouds to skeletons, body sensors, and text.

By merging two existing data sets, the team synchronised all four data types for the same actions. Their model, DeSPITE, learns one shared space across them. New tasks became possible: matching and retrieval between skeletons, point clouds, and sensors, plus finding moments in time. The model also worked well as pre-training for activity recognition on two standard benchmarks.

Catalog record

CREATORThomas Kreutz, Max Mühlhäuser, Alejandro Sanchez Guinea
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeSPITE: Exploring Contrastive De ep S keleton- P ointcloud- I MU- T ext E mbeddings for Advanced Point Cloud Human Activity Understanding Thomas Kreutz, Max Mühlhäuser, Alejandro Sanchez Guinea Abstract Despite LiDAR (Light Detection and Ranging) being an effective privacy-preserving alternative to RGB cameras to perceive human activities, it remains largely underexplored in the context of multi-modal contrastive pre-training for human activity understanding (e.g., human activity recognition (HAR), retrieval, or person re-identification (RE-ID)). To close this gap, our work explores learning the correspondence between LiDAR point clouds, human skeleton poses, IMU data, and text in a joint embedding space. More specifically, we present DeSPITE, a D e ep S keleton- P ointcloud- I MU- T ext E mbedding model, which effectively learns a joint embedding space across these four modalities. At the heart of our empirical exploration, we have combined the existing LIPD and Babel datasets, which enabled us to synchronize data of all four modalities, allowing us to explore the learning of a new joint embedding space. Our experiments demonstrate novel human activity understanding tasks for point cloud sequences enabled through DeSPITE, including Skeleton ↔ \leftrightarrow Pointcloud ↔ \leftrightarrow IMU matching, retrieval, and temporal moment retrieval. Furthermore, we show that DeSPITE is an effective pre-training strategy for point cloud HAR through experiments in MSR-Action3D and HMPEAR. 1 Introduction A key challenge in multi-modal human activity understanding tasks, such as human activity recognition (HAR), human pose estimation (HPE), retrieval, or person re-identification (RE-ID) “in the wild” is obtaining paired sensor data for each individual in a multi-person scene (e.g., IMU with human poses, point clouds, or RGB videos). Prior work has studied RGB-IMU matching for identity-aware tracking/RE-ID [ 20 , 4 ] , RGB-IMU matching for video retrieval [ 39 ] , and IMU-Skeleton Pose matching [ 57 ] to correct IMU drift in multi-modal HPE. However, existing methods primarily focus on RGB-centric modalities, limiting applicability to privacy-sensitive scenarios like healthcare and surveillance, where RGB cameras may not be able to be deployed. To address privacy concerns, silhouette masks [ 37 , 38 ] or skeletons [ 1 ] have been proposed to anonymize detected individuals from RGB video. While effective, these anonymization techniques still come with the limitation that they require post-processing and short-term storage of the raw RGB data. In contrast, LiDAR is a privacy-preserving alternative, with proven capabilities for multi-modal HAR (e.g., [ 61 , 29 ] ) and HPE (e.g., [ 24 , 48 ] ). However, matching skeleton or IMU signals to LiDAR-based point cloud sequences is underexplored. Beyond matching, recent advances in multi-modal contrastive learning, such as ImageBind [ 15 ] , IMU2CLIP [ 39 ] , BabelTower [ 8 ] , MotionCLIP [ 54 ] , or LAVIMO [ 63 ] have demonstrated the power and benefits of cross-modal alignment for human activity understanding. These models learn a shared embedding space, enabling cross-modality matching, retrieval, and effective neural network pre-training for downstream tasks. Despite their success, they all use RGB data as a main visual modality to bind the learned representations. Extending this line of research, our paper asks the important research question What happens if we only depend on LiDAR in multi-modal contrastive learning as the main visual modality? , which has not been studied before. Figure 1 : DeSPITE links different data modalities that represent human activities and, therefore, have a natural correspondence into a joint embedding space. As a result, DeSPITE enables tasks that depend on this correspondence that were not possible before. We present DeSPITE, a D e ep S keleton- P ointcloud- I MU- T ext E mbedding model, which is illustrated in Figure 1 . Inspired by CLIP [ 46 ] , ImageBind [ 15 ] , and IMU2CLIP [ 39 ] , DeSPITE learns a shared embedding space with a contrastive loss based on InfoNCE [ 41 ] between paired sequences of point cloud \leftrightarrow skeleton \leftrightarrow IMU \leftrightarrow text data. Unlike prior works leveraging frozen text or RGB data embeddings as a binding modality (e.g., IMU2CLIP [ 39 ] , MotionCLIP [ 54 ] , LAVIMO [ 63 ] ), our primary goal is not to demonstrate modality alignment to text. Instead, we present novel applications for point cloud-based human activity sequences that were not possible before, enabled by unifying these modalities into a joint embedding space. Furthermore, while our primary focus is on enabling novel cross-modal retrieval and matching tasks, we find that DeSPITE also serves as an effective pre-training method for HAR, demonstrating improvements over uni-modal baselines. Finally, to understand the contribution of each modality to the joint embedding space, we train several DeSPITE variants (DeSPIE, DeSPE, etc.), evaluating their individual impact on cross-modal matching, retrieval, and HAR performance. As described in Figure 1 , after successful alignment in a shared embedding space, DeSPITE and its variants (e.g., DeSPIE) allow novel and very useful applications between point cloud, IMU, and skeleton data: (a) Person Re-ID by matching a snippet of, e.g., IMU data to the correct point cloud snippet, or a skeleton snippet to point cloud sequences, (b) Temporal moment retrieval/search in a video with skeletons, IMU, or a point cloud as query, (c) pre-training modality specific encoders for human activity recognition, and (d) retrieval of each modality through each modality from a large motion database. To train and evaluate DeSPITE, we construct LIPD-Babel, a dataset aligning point clouds, IMU, skeletons, and text by integrating the LIPD dataset [ 48 ] with the text annotations from Babel [ 45 ] . LIPD-Babel enables two key evaluations: (1) LIPD-Babel-v1 for cross-modal matching and retrieval, demonstrating DeSPITE and DeSPIE’s ability to align point clouds, skeletons, and IMU signals, and (2) LIPD-Babel-v2 for contrastive pre-training, where DeSPITE and DeSPIE improve single-modality HAR, surpassing SOTA on HMPEAR [ 29 ] and MSR-Action3D [ 27 ] . Our contributions are as follows. (i) DeSPITE: A D e ep S keleton- P ointcloud- I MU- T ext E mbedding model that enables cross-modal matching and retrieval tasks between point clouds, IMU, and skeletons, unlocking applications that were previously impossible, and we will release the resulting pre-trained encoders, code, and data for future research. (ii) We show that DeSPITE is an effective contrastive pre-training strategy for single-modality HAR, demonstrating new state-of-the-art performance. (iii) LIPD-Babel, a new dataset for privacy-preserving multi-modal learning, with versions LIPD-Babel-v1 tailored for matching, retrieval tasks, and LIPD-Babel-v2 tailored for HAR. 2 Related Work 2.1 Multi-Modal Contrastive Learning for Human Activity Understanding Recent works have explored unified embedding spaces across sensor modalities. ImageBind [ 15 ] binds six modalities using image-text pairs, while IMU2CLIP [ 39 ] and MotionCLIP [ 54 ] align IMU and skeleton data with CLIP’s image-text space. LAVIMO [ 63 ] improves skeleton-video-text retrieval, and BabelTower [ 8 ] incrementally aligns six sensing modalities, reducing reliance on RGB. Unlike these works, we focus exclusively on privacy-preserving modalities, introducing LiDAR into a joint embedding space with IMU and skeletons. This enables novel retrieval tasks (LiDAR \leftrightarrow IMU, LiDAR \leftrightarrow Skeleton) and serves as an effective pre-training strategy for point cloud-based HAR. 2.2 Pre-Training for Point Cloud Human Activity Recognition While countless general purpose embedding models and foundation models have emerged in the last years

Read the complete text