TouchMap-OR: Multi-View 3D Mapping of Hand–Surface Contacts

Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax, [0.5em] ETH Zurich · 2026

Read TouchMap-OR: Multi-View 3D Mapping of Hand–Surface Contacts by Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax, [0.5em] ETH Zurich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17638.txt.

Introduction

Sophokles Ktistakis and colleagues introduce TouchMap-OR, a multi-view system for hand-surface contact reconstruction in operating rooms. Hand-surface contacts drive pathogen transmission yet stay largely unobserved. The system builds multi-person body tracks across cameras and estimates MANO hand meshes from RGB views aligned to depth. Fused reconstructions attach to clinicians, yielding hand paths mapped onto a 3D room model. Timing closeness infers contact episodes naming clinician, surface and time. On three anesthesia cases it beats tracking-based baselines on contact F1 and identity scores.

Catalog record

CREATORSophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax, [0.5em] ETH Zurich
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36091
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TouchMap-OR: Multi-View 3D Mapping of Hand–Surface Contacts Sophokles Ktistakis, Rui Wang, Bastian Grande, Hugo Sax, [0.5em] ETH Zurich Abstract Hand–surface interactions between clinicians, patients, and medical equipment play a central role in pathogen transmission during medical procedures. However, these interactions remain largely unobserved, as current infection-prevention practices rely on manual observation and cannot reconstruct detailed contact histories. In this work we formulate the problem of identity-resolved hand–surface interaction reconstruction in operating rooms and introduce TouchMap-OR, a multi-view RGB-D vision system that models clinicians, articulated hand geometry, and the semantic structure of the clinical environment to infer when and where contacts occur. The system reconstructs globally consistent multi-person 3D skeleton tracks across cameras while estimating articulated MANO hand meshes from RGB observations aligned to depth data. Multi-view hand reconstructions are fused and associated with tracked clinicians to obtain consistent left and right hand trajectories. A semantic 3D model of the operating room is built from multi-view segmentation and depth fusion, enabling reconstructed hand trajectories to be mapped to specific surfaces, including medical equipment, movable objects, and patient body sites. Temporal hand–surface proximity is used to infer contact episodes describing which clinician touched which surface and when. We evaluate TouchMap-OR on recordings from three real anesthesia inductions with manually annotated contact events. TouchMap-OR achieves 0.75 binary contact F1, outperforming tracking-based baselines while maintaining comparable multi-person tracking accuracy and achieving 0.96 identity attribution accuracy. Figure 1. From synchronized multi-view RGB-D observations (left), our system reconstructs a metric 3D representation of the operating room (right) that jointly models clinicians, articulated hands, and the semantic structure of the environment. This representation infers temporally consistent hand–surface contact episodes, identifying which clinician touched which surface, where, and for how long, thereby producing a detailed map of interactions among clinicians, patients, and medical equipment during clinical procedures. 1 Introduction Preventing pathogen transmission during medical and surgical procedures remains a major challenge. During routine clinical workflows, multiple clinicians interact with shared instruments, monitors, and the patient 27 . These interactions generate dense sequences of hand–surface contacts, creating opportunities for indirect pathogen transmission through contaminated surfaces (fomite transmission) 26 . Understanding how these contacts unfold in time and space is therefore critical for studying transmission pathways and evaluating infection prevention strategies. However, the dynamics of hand–surface interactions in clinical environments remain poorly characterized. Despite decades of infection prevention research, the current gold standard for assessing hand hygiene compliance remains manual observation, which is labor-intensive, prone to observer bias, and limited in temporal resolution 14 . Recent work in automated hand hygiene monitoring has explored computer vision as a scalable alternative to manual observation. Systems based on RGB or depth cameras detect dispenser activations or recognize hand hygiene gestures, often achieving accuracy comparable to human observers. Examples include multi-camera systems for detecting hygiene gestures 11 , CNN-based recognition of dispenser usage 30 , 3D convolutional models capturing temporal motion patterns during anesthesia induction 16 , and privacy-preserving approaches using depth-only sensing 8 . While these systems can determine when hand hygiene events occur, they do not observe the contact sequences between hands, surfaces, and patients that govern potential transmission pathways. As a result, they cannot reconstruct contact histories or analyze how contamination may propagate across surfaces and clinical actors. Outside healthcare, several computer vision approaches have been proposed to estimate hand–object or hand–surface contact. Early work inferred contact using CNN-based hand segmentation 2 ; 19 , while more recent methods leverage 3D hand-mesh reconstruction to predict fine-grained contact regions 12 ; 28 . However, these approaches typically assume visible skin, minimal occlusion, and controlled environments. Such assumptions rarely hold in operating rooms, where surgical gloves obscure skin cues and hands are frequently occluded by instruments, clinicians, and medical equipment. To address this gap, we introduce TouchMap-OR, a multi-view RGB-D vision system that reconstructs identity-resolved hand–surface contact events in real operating rooms. Our approach jointly models three components of the clinical scene: (i) the 3D trajectories of clinicians, (ii) metrically aligned articulated hand meshes, and (iii) a semantic 3D representation of the operating room surfaces. Using synchronized RGB-D cameras, we first estimate multi-person 3D skeleton tracks with stable identities across views. We then reconstruct articulated MANO hand meshes from each camera view and align them metrically to depth observations. These hand reconstructions are fused across cameras and associated with tracked clinicians to obtain consistent left and right hand trajectories for each person. Finally, these reconstructed hand trajectories are integrated with a semantically labeled 3D model of the operating room, obtained by multi-view fusion of segmented RGB-D observations. By evaluating the proximity between hand anchors and semantic surfaces over time, the system infers temporally consistent hand–surface contact episodes that specify which clinician touched which surface, at what location, and for how long. This enables reconstruction of contact histories between clinicians, instruments, and patient surfaces, providing a computational basis for analyzing transmission pathways. Our key contributions are: – Identity-resolved 3D contact reconstruction in operating rooms. We introduce a multi-view RGB-D pipeline that reconstructs hand–surface contact episodes between multiple clinicians and semantically labeled operating room surfaces. – Multi-view 3D hand reconstruction and person association. We fuse MANO-based hand reconstructions across multiple cameras and associate each left and right hand with a persistent clinician identity using multi-view 3D skeleton tracking. – Semantic 3D contact mapping. We map reconstructed hand trajectories to semantically labeled scene surfaces, enabling the reconstruction of temporally consistent hand–surface contact episodes. 2 Related Works 2.1 Vision-Based Clinical Workflow Monitoring Computer vision has increasingly been explored for understanding clinical workflows, with applications including activity recognition, mobility assessment, hygiene monitoring, and operating room scene analysis. Early work focused on recognizing surgical actions and workflow phases from video to support context-aware assistance systems in the operating room 32 . More recent approaches model the operating room through structured representations of actors, objects, and their relationships using semantic scene models and activity graphs 21 . Within infection prevention, several works have explored automated hand hygiene monitoring using visual sensing. Multi-camera hospital systems have been proposed to detect hygiene events and track healthcare workers using depth sensing and visual tracking 11 . Other approaches detect alcohol dispenser usage or hygiene gestures using convolutional or spatio-temporal models on RGB or depth data 30 . Vision-based systems have also been used to assess hand hygiene quality and measure healthcare worker–patient interactions or personal protective equipment compliance 1 ; 7 . However, these methods primarily detect discrete hygiene events or coarse interactions. In contrast, our work reconstructs identity-resolved hand–surface contact episodes across multiple clinicians and objects, enabling detailed analysis of interaction networks and potential contamination pathways in operating rooms. 2.2 Multi-View Multi-Person Pose Estimation and Tracking Human pose estimation has progressed rapidly with deep learning methods for single-view 2D pose estimation, where convolutional networks predict body joint heatmaps from RGB images. Early deep architectures 31 ; 20 demonstrated the effectiveness of hierarchical feature refinement for human pose prediction, while bottom-up formulations such as OpenPose 6 enabled real-time multi-person pose estimation by jointly detecting body parts and their associations. Building on these advances, monocular 3D pose estimation methods attempt to recover 3D joint coordinates from single images or videos. Representative approaches include direct 2D-to-3D pose lifting 18 , volumetric heatmap regression 22

Read the complete text