4D Synchronized Fields : Motion-Language Gaussian Splatting for Temporal Scene Understanding

Mohamed Rayan Barhdadi, Samir Abdaljalil, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban · 2026

Read 4D Synchronized Fields : Motion-Language Gaussian Splatting for Temporal Scene Understanding by Mohamed Rayan Barhdadi, Samir Abdaljalil, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14301.txt.

Introduction

Barhdadi and co-authors present 4D Synchronized Fields, a representation that joins shape, motion, and language for moving scenes. Older methods split these parts: rebuilds drop motion structure, and language tags are pasted on after motion is learned. Each object's motion is factored into a shared object path plus a small residual. A ridge map tied to that motion predicts how object meanings shift over time. On HyperNeRF, the method posts the best scores among language and motion rivals, and finds target moments well.

Catalog record

CREATORMohamed Rayan Barhdadi, Samir Abdaljalil, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76556
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14301

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

4D Synchronized Fields : Motion-Language Gaussian Splatting for Temporal Scene Understanding Mohamed Rayan Barhdadi, Samir Abdaljalil, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban Abstract Current 4D representations decouple geometry, motion, and semantics: reconstruction methods discard interpretable motion structure; language-grounded methods attach semantics after motion is learned, blind to how objects move; and motion-aware methods encode dynamics as opaque per-point residuals without object-level organization. We propose 4D Synchronized Fields , a 4D Gaussian representation that learns object-factored motion in-loop during reconstruction and synchronizes language to the resulting kinematics through a per-object conditioned field. Each Gaussian trajectory is decomposed into shared object motion plus an implicit residual, and a kinematic-conditioned ridge map predicts temporal semantic variation, yielding a single representation in which reconstruction, motion, and semantics are structurally coupled and enabling open-vocabulary temporal queries that retrieve both objects and moments. On HyperNeRF, 4D Synchronized Fields achieves 28.52 dB mean PSNR, the highest among all language-grounded and motion-aware baselines, within 1.5 dB of reconstruction-only methods. On targeted temporal-state retrieval, the kinematic-conditioned field attains 0.884 mean accuracy, 0.815 mean vIoU, and 0.733 mean tIoU, surpassing LangSplat (0.415, 0.304, and 0.262) and 4D LangSplat (0.620, 0.433, and 0.439 respectively). Ablation confirms that kinematic conditioning is the primary driver, accounting for + + 0.45 tIoU over a static-embedding-only baseline. 4D Synchronized Fields is the only method that jointly exposes interpretable motion primitives and temporally grounded language fields from a single trained representation. Code will be released. Available_soon Figure 1: 4D Synchronized Fields learns a deformable 4D Gaussian scene whose per-Gaussian trajectories are decomposed in-loop into shared object motion plus implicit residuals. The resulting synchronized tracks and kinematics condition an object-time language field, trained from projected object crops and a per-object ridge map, enabling open-vocabulary temporal queries that retrieve both objects and moments , grounded in learned motion structure. 1 Introduction Building faithful representations of the world is a long-standing goal of visual computing, pursued through stages that successively close the gap between what is represented and what the world actually contains. Early work focused on what could be extracted from a single image: edges and local features [ 37 , 35 ] , part-based shape models [ 2 , 12 ] , and learned hierarchies for recognition at scale [ 25 ] , yet recognizing what is in an image leaves open the question of where things are in three dimensions. Multi-view geometry and structure-from-motion [ 18 , 46 ] recovered 3D structure from image collections; neural radiance fields [ 38 ] encoded scenes as continuous volumetric functions for photorealistic novel-view synthesis [ 1 , 39 ] ; and 3D Gaussian Splatting [ 22 ] replaced implicit volumetric queries with explicit, rasterizable primitives at real-time rates. #1: the world can now be reconstructed and rendered with high fidelity, but the representation is blind to what it depicts. Language-grounded methods [ 23 , 43 , 59 , 62 , 6 ] attached open-vocabulary semantics to 3D structures. 4D LangSplat [ 29 ] and related works [ 36 , 20 ] extended this to temporal representations, but language features are typically distilled after motion has been optimized, so the semantic field inherits no structured knowledge of how objects move, only a static association with what is present. #2: language grounding lets the representation know what it depicts, but it treats motion as a black box and cannot express how things move. Knowing what is not enough when the world is not static: objects move, interact, and change state. Deformation-based NeRFs [ 42 , 40 , 41 ] , scene-flow methods [ 31 , 15 , 34 ] , factored space-time representations [ 14 , 3 ] , and 4D Gaussian Splatting [ 57 , 58 , 32 , 30 ] brought dynamic reconstruction within reach, yet motion is encoded as per-point deformation optimized purely for photometric error, exposing no interpretable object-level structure. #3: motion-aware methods model dynamics, but motion remains an opaque per-point residual with no object-level structure or semantic coupling. The absence of object-level structure is not just an engineering limitation; it conflicts with how biological perception is organized. From the first months of life, infants perceive objects as cohesive bodies by attending to how surfaces move, well before relying on appearance [ 52 , 53 , 4 ] , and objects individuated through motion serve as the hubs around which perception, action, and language coalesce [ 49 , 50 ] . World models that must predict, plan, and explain in terms of objects require representations exposing motion as a first-class, interpretable quantity [ 54 , 48 , 17 , 51 ] . 1 1 1 We adopt a perception-first viewpoint: the representation is learned from visual supervision; language serves as an interface to the learned structure over space and time. This mirrors developmental findings in which infants first build core object representations from spatiotemporal cues and only later integrate linguistic labels [ 4 , 56 ] . #4: closing the loop requires synchronization : motion must be factored by object, and semantics must be conditioned on that factored motion, so that reconstruction, language, and dynamics are unified within a single representation. We present 4D Synchronized Fields , a 4D scene representation built on the synchronization principle of #4. 4D Synchronized Fields learns renderable Gaussians while fitting an object-consistent motion model in-loop during reconstruction, so that motion structure, object identity, and semantics emerge together rather than being bolted on in sequence. In summary, our contributions are as follows: A synchronized 4D scene representation. We propose the first representation that synchronizes reconstruction, object-factored motion, and language within a unified Gaussian representation via staged training: motion structure is learned first, and language is then grounded on the resulting kinematics, culminating the progression of #1 to #3 and instantiating the synchronization principle of #4. In-loop motion decomposition. Each Gaussian’s trajectory is decomposed into shared object motion (SE(3) or affine) plus an implicit residual, with a residual-adaptive modulation scheme that relaxes regularization on persistently non-rigid Gaussians while leaving the forward renderer unchanged. Kinematic-conditioned language field. We train a per-object ridge map from a kinematic feature vector to semantic residuals, yielding an object-time embedding field that supports open-vocabulary temporal queries and surpasses existing language-grounded baselines on state retrieval. Structured temporal scene understanding. The representation produces synchronized object tracks, motion primitives, interaction graphs, and language slots that a multimodal LLM can directly consume at inference time for temporal reasoning, without having entered the training loop or requiring task-specific retraining. 2 Related Work We organize prior work along four capability axes and assign each a visual marker used consistently throughout the paper, in section headers, method stages, tables, and result discussions: Reconstruction Language grounding Motion structure Synchronized (ours) . No existing method spans all four; 4D Synchronized Fields is the first to do so within a single trained representation. 3D and 4D reconstruction. NeRFs [ 38 , 1 , 39 ] and 3D Gaussian Splatting [ 22 ] achieve photorealistic novel-view synthesis from static scenes. Dynamic extensions, including deformation-based NeRFs [ 42 , 40 , 41 ] , scene-flow methods [ 31 , 15 , 34 ] , factored representations [ 14 , 3 ] , and 4D Gaussian variants [ 57 , 58 ,

Read the complete text