Understanding Dynamic Scenes in Ego Centric 4D Point Clouds

Junsheng Huang, Shengyu Hao, Bocheng Hu, Gaoang Wang · 2025

Read Understanding Dynamic Scenes in Ego Centric 4D Point Clouds by Junsheng Huang, Shengyu Hao, Bocheng Hu, Gaoang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07251.txt.

Introduction

Understanding a moving scene from the wearer's own viewpoint, tracking how objects and people shift over time, matters for robots and smart assistants. Junsheng Huang and co-authors introduce EgoDynamic4D in 2025, a question-answering benchmark built from first-person video with depth, camera paths, object masks, and 3D boxes over time. It holds about 927,000 questions with written reasoning steps.

Twelve task types probe motion, human-object contact, path prediction, relations, and cause over time. The team also offers a model that packs instance-aware features, time and camera cues, and smart compression into tokens a language model can read. It beat rival systems across the board, showing the value of joining static and moving scene cues.

Catalog record

CREATORJunsheng Huang, Shengyu Hao, Bocheng Hu, Gaoang Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07251

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Dynamic Scenes in Ego Centric 4D Point Clouds Junsheng Huang, Shengyu Hao, Bocheng Hu, Gaoang Wang Abstract (a) Data sources that provide videos captured in different scenes. (b) Duration of sequences. Peaks at 10-30s (38 sequences) and 90-120s (224 sequences). (c) QA pairs in different tasks. We define 12 tasks in 3 domains. Figure 2: Distribution of data. Understanding dynamic 4D scenes from an egocentric perspective—modeling changes in 3D spatial structure over time—is crucial for human–machine interaction, autonomous navigation, and embodied intelligence. While existing egocentric datasets contain dynamic scenes, they lack unified 4D annotations and task-driven evaluation protocols for fine-grained spatio-temporal reasoning, especially on motion of objects and human, together with their interactions. To address this gap, we introduce EgoDynamic4D , a novel QA benchmark on highly dynamic scenes, comprising RGB-D video, camera poses, globally unique instance masks, and 4D bounding boxes. We construct 927K QA pairs accompanied by explicit Chain-of-Thought (CoT), enabling verifiable, step-by-step spatio-temporal reasoning. We design 12 dynamic QA tasks covering agent motion, human–object interaction, trajectory prediction, relation understanding, and temporal–causal reasoning, with fine-grained, multidimensional metrics. To tackle these tasks, we propose an end-to-end spatio-temporal reasoning framework that unifies dynamic and static scene information, using instance-aware feature encoding, time and camera encoding, and spatially adaptive down-sampling to compress large 4D scenes into token sequences manageable by LLMs. Experiments on EgoDynamic4D show that our method consistently outperforms baselines, validating the effectiveness of multimodal temporal modeling for egocentric dynamic scene understanding. Figure 1: We introduce a novel QA benchmark EgoDynamic4D and an end-to-end spatio-temporal reasoning framework. Figure 3: QA generation pipeline. Given RGB-D sequences with aligned 3D bounding boxes and poses, we extract spatial-temporal properties, apply template-based CoT reasoning, and refine questions via LLMs and human validation. 1 Introduction With the rapid advancement of embodied AI and human–machine interaction technologies, understanding dynamic 4D scenes from an egocentric viewpoint—integrating 3D spatial dimensions and the temporal dimension—has emerged as a pivotal research challenge. Unlike conventional third-person video analysis ( Pang et al. 2021 ; Song et al. 2021 ; Pan et al. 2021 ; Li, Niu, and Zhang 2022 ) , egocentric videos exhibit high dynamics, frequent scene changes, and rich interactive behaviors, requiring models not only to capture the wearer’s movement but also to perceive and reason about surrounding people, objects, and their evolving relationships ( Li et al. 2021 ; Yu et al. 2023 ; Fan et al. 2017 ; Lee, Lee, and Choi 2023 ; Lin et al. 2022 ; Pramanick et al. 2023 ) . Applications such as robotic perception, AR, and autonomous driving demand efficient and accurate ego-centric scene understanding to empower the next generation of embodied agents. Although landmark egocentric datasets Ego4D ( Grauman et al. 2022 ) , EgoExo4D ( Grauman et al. 2024 ) , HD-Epic ( Perrett et al. 2025 ) , and 3D datasets ( Dai et al. 2017 ; Wang et al. 2024a ; Chen, Chang, and Nießner 2020 ) have driven progress in action recognition, object locating, and 3D scene analysis, they suffer from the following key limitations: (1) Incomplete 4D annotations and lack of dynamic content: Egocentric datasets often lack temporally aligned 3D bounding boxes and trajectories, while 3D datasets focus on static scenes without moving objects or agents—limiting the study of real-world dynamics. (2) Limited evaluation of temporal reasoning: Existing benchmarks emphasize short-term or moment-based tasks and lack protocols to assess reasoning over continuous object motion or interaction. (3) Incomplete multimodal evaluation: While some works explore temporal scene graphs ( Yang et al. 2023 ) , they focus on representation construction rather than end-to-end multimodal reasoning, and do not support QA-based evaluation of dynamic 4D scenes. We propose EgoDynamic4D , the first egocentric QA benchmark for highly dynamic 4D scene understanding. By refining ADT ( Pan et al. 2023 ) and THUD++ ( Tang et al. 2024 ) annotations, we create a unified, multimodal dataset with RGB-D video, camera poses, globally unique instance masks, and 4D bounding boxes across indoor scenes. It includes 12 QA tasks covering scene descriptions, momentary and durative dynamics, with rich evaluation metrics. To improve reasoning and transparency, we equip QA pairs with explicit Chain-of-Thought (CoT), enabling step-by-step spatio-temporal explanations that aid training and allow interpretable intermediate results. We also design an end-to-end spatio-temporal reasoning model that compresses long 4D sequences into LLM-compatible tokens via instance-aware encoding and adaptive down-sampling. Experiments show our method with CoT supervision outperforms prior baselines, setting a new standard for egocentric dynamic scene understanding. Our main contributions are: 1. We introduce EgoDynamic4D , a novel highly dynamic 4D scene QA benchmark with 927K QA pairs and explicit CoT, covering 12 task types and multimodal data. 2. We propose an end-to-end spatiotemporal reasoning framework using instance-aware feature encoding, time and camera encoding, and adaptive down-sampling to efficiently handle large 4D data for LLMs. 3. We conduct extensive experiments on EgoDynamic4D using representative models. Our method significantly improves performance and provides interpretable reasoning, establishing a strong baseline for future research. 2 Related Work Egocentric Video Datasets Egocentric datasets have catalyzed advances in action recognition and object tracking. EpicKitchens ( Damen et al. 2018 ) provides first-person kitchen activity videos but lacks 3D spatial annotations. HD-Epic ( Perrett et al. 2025 ) adds some object localization and short-term action data, but offers only sparse MPS point clouds without dense depth videos, limiting 4D reconstruction. Large-scale datasets like Ego4D ( Grauman et al. 2022 ) , EgoExo4D ( Grauman et al. 2024 ) and EgoLife ( Yang et al. 2025 ) omit 3D bounding boxes per frame, which constrains dynamic object reasoning. ( Linghu et al. 2024 ) focuses on contextual reasoning in 3D scenes rather than dynamic objects. In contrast, ADT ( Pan et al. 2023 ) and THUD++ ( Tang et al. 2024 ) annotate each frame with RGB-D data, object poses, 2D/3D bounding boxes, and camera trajectories, supporting dynamic scene understanding. EgoDynamic4D builds reorganizing these 2 datasets to create a unified benchmark for dynamic 4D QA tasks. 3DLLMs 3DLLM ( Hong et al. 2023 ) and 3UR-LLM ( Xiong et al. 2025 ) take point clouds as inputs for tasks like description and localization. LLaVA3D ( Zhu et al. 2024 ) and Video3DLLM ( Zheng, Huang, and Wang 2025 ) introduce ”3D patches” by embedding 2D CLIP features ( Radford et al. 2021 ) with 3D positional encodings. GPT4Scene ( Qi et al. 2025 ) uses BEV imagery ( Liu et al. 2022 ; Li et al. 2022a ) and consistent object IDs to reason about spatial relations. LSceneLLM ( Zhi et al. 2025 ) employs LLM attention to identify relevant regions for fine details. Chat-3D ( Wang et al. 2023 ) and Chat-Scene ( Huang et al. 2024 ) use object identifiers to enable interactive understanding. While effective on static 3D scenes, these approaches lack mechanisms to capture temporal dynamics for 4D reasoning. 4D Scene Understanding Models Dynamic 4D scene comprehension requires integrating RGB-D information and facilitating cross-time reasoning. PSG4D ( Yang et al. 2023 ) offers a structured 4D scene representation, ( Wu et al. 2025 ) leverages 2D annotations to enhance 4D learning via a 3D mask decoder–based LLM. Both focus on graph generation without

Read the complete text