Seeing Without Eyes: 4D Human–Scene Understanding from Wearable IMUs

Hao-Yu Hsu *, Tianhang Cheng *, Jing Wen, Alexander G. Schwing, Shenlong Wang · 2026

Read Seeing Without Eyes: 4D Human–Scene Understanding from Wearable IMUs by Hao-Yu Hsu *, Tianhang Cheng *, Jing Wen, Alexander G. Schwing, Shenlong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21926.txt.

Introduction

Hsu and four colleagues present IMU-to-4D, a framework for 4D human-scene understanding from wearable sensors, without cameras. Cameras pose challenges in privacy, safety, energy efficiency, and scalability. The framework repurposes large language models for non-visual understanding of human-scene dynamics. Using data from a few inertial sensors in earbuds, watches, or smartphones, it predicts detailed 4D human motion together with coarse scene structure. Across diverse datasets, IMU-to-4D yields clearer and more stable results over time than cascaded pipelines.

Catalog record

CREATORHao-Yu Hsu *, Tianhang Cheng *, Jing Wen, Alexander G. Schwing, Shenlong Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58661
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Without Eyes: 4D Human–Scene Understanding from Wearable IMUs Hao-Yu Hsu *, Tianhang Cheng *, Jing Wen, Alexander G. Schwing, Shenlong Wang Abstract Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to reconstruct human motion and 3D scene layouts purely from everyday wearable sensors. For this we introduce IMU-to-4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-scene dynamics. IMU-to-4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure. Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support rich 4D understanding. Project page: https://tianhang-cheng.github.io/IMU4D/ * * footnotetext: Equal contribution. Figure 1 : IMU-to-4D reconstructs full 4D human–scene dynamics from only a few everyday wearable IMUs. Given sparse inertial signals from devices such as earbuds, smartphones, or watches, our method predicts SMPL-X body motion, generates textual activity descriptions, and recovers a coarse 3D scene layout with object identities. 1 Introduction Recent advances in multimodal foundation models have expanded how AI perceives and assists in daily life. Yet their understanding of the physical world remains limited: most rely on vision-centric sensing such as cameras or LiDAR, which, while informative, introduces persistent challenges w.r.t. privacy, safety, energy efficiency, and scalability. As AI systems become more integrated into daily life, there is an increasing demand for non-visual, privacy-preserving, and low-power modalities that can still capture the richness of human-environment interaction. Imagine an assistant that can recall how many times you drank water today, where you left your keys, or whether you’ve been sitting too long—without recording a single image. Such a system would extend AI’s utility to personal health, memory, and accessibility. This motivates a research question: How much can everyday motion sensors embedded in earbuds, watches, and smartphones reveal about our activities and environments? To answer this question, we propose IMU-to-4D , a framework for vision-free 4D perception that reconstructs detailed human motion and the surrounding 3D environment purely from inertial signals. The key insight behind IMU-to-4D: motion, human activities, and environment are inherently coupled —everyday inertial traces encode not only how the body moves but also what surrounds it, e.g . , a wrist acceleration may indicate placing a cup, while a brief free fall of an earbud reveals a nearby surface. Recent multimodal foundation models have demonstrated that such coupled structure across modalities can be effectively captured by leveraging rich pre-trained representations from LLMs and extending them to new modalities through unified architectures. Building on this intuition, IMU-to-4D repurposes large language models (LLMs) as spatiotemporal reasoners, jointly modeling the distribution of sequential inertial signals, human motions, 3D layouts, and semantic activities. To achieve this, we develop a novel tokenization scheme for each modality (IMU, motion, and scene layout) that balances expressiveness, alignment, and compactness. We then use the input and output embeddings to formulate a sequential prediction problem within an LLM and jointly finetune the model with a multi-task loss in a two-stage training process. This unified formulation allows the model to infer coherent, context-aware 4D representations from as few as 2-3 inertial sensors—substantially improving temporal and spatial consistency compared with cascaded pipelines (IMU \rightarrow motion \rightarrow scene). Through the compositional reasoning capacity of LLMs, IMU-to-4D bridges physical dynamics with spatial and semantic context, enabling a holistic understanding of human-scene interaction without relying on visual input. Experiments on diverse datasets show that IMU-to-4D achieves more coherent and physically consistent reconstructions than prior approaches. Our results demonstrate that ubiquitous wearable sensors alone can support rich, privacy-preserving 4D understanding of human activities and their environments—paving the way for embodied AI that perceives the world through motion rather than sight. Our contributions are threefold: • We introduce the first framework that recovers 4D human motion, scene layouts, and textual descriptions from as few as three IMU sensors by repurposing LLMs for cross-modal structural reasoning. • We design novel tokenization schemes tailored for IMU and human motions for token-efficient yet expressive sequence modeling. • Experiments show that our joint reasoning framework outperforms pipelined SoTA methods and enables ambient scene understanding without vision. 2 Related Work Table 1 : Motion- and scene-understanding task comparison. Task Input Modality Output Modality Raw IMU GT Motion Visuals Motion Text 3D Scene Motion-to-Text [ 6 , 10 ] ✓ ✓ ✓ Motion-Chat [ 121 , 16 ] ✓ ✓ ✓ Motion-to-Scene [ 98 , 64 , 48 ] ✓ ✓ Ego-Understand [ 30 , 22 ] ✓ ✓ ✓ ✓ IMU-to-Motion [ 97 , 63 , 103 ] ✓ ✓ IMU-to-4D (Ours) ✓ ✓ ✓ ✓ IMU-Based Human Motion Estimation. We are not the first to approach human pose estimation from wearable IMUs. Earlier works [ 76 , 82 , 32 , 104 , 13 , 103 , 36 , 39 , 123 , 105 , 80 , 92 , 102 , 122 ] explored IMU-based motion capture as a cost-effective alternative to optical tracking systems [ 62 ] . These methods typically rely on specialized, low-noise IMUs attached to the body via custom suits to achieve accurate reconstruction [ 74 ] . While effective in controlled settings, such setups are costly, intrusive, and impractical for everyday users. Subsequent studies [ 63 , 97 , 110 ] shifted toward commodity devices—smartphones, smartwatches, and wireless earbuds—to infer human motion from sparse, noisy IMU signals. This direction motivated IMU-to-4D as a path toward ubiquitous, low-cost human sensing. Yet, existing models remain task-specific and small-scale, often exhibiting temporal drift, weak global consistency, and no scene awareness. As a fix, recent efforts have incorporated egocentric video alongside IMU signals [ 30 , 111 , 99 , 44 , 66 ] , which injects high-level semantic cues about activities and surroundings. However, such vision-augmented methods reintroduce privacy risks, increase sensing costs, and break the goal of lightweight, always-on perception. In contrast, our IMU-to-4D studies foundation models that jointly reason about human motion, scene geometry, and semantics—using as input only a few everyday inertial sensors and no visual data. By coupling human motion and environment, IMU-to-4D delivers higher spatio-temporal coherence and richer scene understanding than prior IMU-based approaches. Motion-Scene and Motion-Object Synthesis. Another line of research focuses on generating scene layouts or object configurations from human motion. The key insight: human motion and surrounding environments exhibit strong semantic and geometric correlations—for example, specific actions often co-occur with particular spatial contexts. To capture these relationships, several paired motion-scene and motion-object datasets [ 27 , 25 , 1 , 47 , 3 , 100 , 28 , 109 , 57 , 40 , 70 ] have been developed, enabling bidirectional synthesis: generating human motion conditioned on scenes [ 26 , 83 , 84 , 115 , 42 , 88 , 118 , 38 ] or generating plausible scene layout from motion trajectories [ 98 , 101 , 48 , 33 , 64 ] . Early methods based on motion planning, inverse kinematics, or hand-crafted heuristics synthesized feasible human-scene interactions [ 61 , 118 ] , but

Read the complete text