\BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp : Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video
Jerrin Bright, John Zelek · 2026
Read \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp : Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video by Jerrin Bright, John Zelek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28820.txt.
Introduction
Bright and Zelek present CoGS, a method for rebuilding a moving scene of a person handling an object from a single ordinary video. One camera view makes the task hard, because the person, the object and the room follow different motion rules while sharing the same pixels. Earlier methods often let the object's motion leak into the person or the background, and parts rarely seen stay poorly constrained.
CoGS splits the scene into three coordinated parts: a jointed human body started from a complete template, a rigid object field following its estimated path, and a still background field. A six-stage schedule steadies the person and object separately before merging them with the room under checks on silhouettes, motion and visibility. On two public benchmarks the system improved both the interaction scenes and wild footage of people in real places.
Catalog record
| CREATOR | Jerrin Bright, John Zelek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28820
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp : Compositional Dynamic Human-Object Scenes Gaussian Splatting from Monocular Video Jerrin Bright, John Zelek Abstract Reconstructing dynamic human–object interaction scenes from monocular video is difficult because the human, manipulated object, and background obey different motion models while sharing the same pixels. Existing dynamic radiance-field and Gaussian-splatting methods often entangle these components, causing object motion to leak into the human or static scene, and monocular human reconstruction remains underconstrained in regions that are rarely observed. We present \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp , a compositional Gaussian-splatting framework for monocular human–object scene reconstruction. \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp decomposes the video into three coordinated branches: an articulated human initialized from a complete canonical prior, a rigid object field driven by an estimated object trajectory, and a static scene field regularized by weak scene-only planar primitives when available. A six-stage optimization schedule first stabilizes the human and object independently, then fuses them with the scene under full-image supervision, visibility-aware human anchoring, object silhouette and motion constraints, and delayed scene regularization. This design keeps each component responsible for its own geometry and motion while allowing photometric evidence to correct the final composite. Experiments on HOSNeRF and NeuMan show that \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp improves both human–object interaction reconstruction and in-the-wild human–scene rendering, achieving stronger fidelity and perceptual quality across full-frame and human-focused evaluations. Code will be released upon publication. 1 Introduction Reconstructing a dynamic human, the object they manipulate, and the surrounding scene from monocular video is a tightly coupled inverse problem studied by human-scene and human-object neural rendering methods 12 ; 9 ; 16 . A single camera provides incomplete geometry, frequent occlusion, and ambiguous depth, while the foreground is not governed by one motion model 12 ; 37 ; 36 . The human articulates according to body pose 18 ; 25 , the manipulated object follows its own motion 16 , and the remaining environment is typically modeled as a static world component observed through the moving camera 12 ; 9 . Treating all pixels as one deforming field can fit training views 22 ; 23 ; 34 ; 41 , but in human–object videos this can entangle motions across components, motivating explicit separation of the body, manipulated object, and scene 16 . Gaussian splatting makes this problem both promising and fragile. Its explicit anisotropic representation supports efficient rendering and direct optimization 13 , and dynamic Gaussian methods extend it to time-varying monocular scenes 34 ; 41 ; 1 ; 15 ; 17 . However, monocular supervision remains underconstrained: human Gaussian/avatar methods rely on articulated priors to stabilize unseen or weakly observed body regions 14 ; 21 ; 36 , object motion in interaction videos requires object-specific supervision 16 , and depth or geometric priors are often used to stabilize ambiguous scene structure 17 ; 31 . A robust monocular method therefore needs more than a single photometric objective. Our key idea is to reconstruct the scene compositionally. We separate the representation into an articulated human branch, a rigid object branch, and a static scene branch, then train each component only when its supervision is reliable. This separation lets the human prior complete weakly observed body regions, keeps manipulated objects independent from the body and scene, and stabilizes large static surfaces without forcing all pixels into one motion model. We introduce \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp , a monocular human–object Gaussian-splatting framework built around this decomposition. Preprocessing aligns COLMAP cameras 31 , SMPL motion from monocular pose estimation 18 ; 24 , SAM-based masks 28 , object trajectory initialization, canonical human initialization, and optional scene planar primitives into a shared coordinate frame. Optimization then proceeds in six stages: human base fitting, human alpha/boundary refinement, object pose and appearance fitting, object silhouette/rotation refinement, full compositional warmup, and true-resume refinement. This schedule reduces competition between branches and delays aggressive density changes until the corresponding component has stable support. The resulting representation supports both full composite rendering and branch-level inspection. At inference time, a target camera, human pose, and object pose are rendered by composing the static scene, posed human, and posed object. This enables evaluation on human–object interaction videos from HOSNeRF 16 as well as human–scene benchmarks such as NeuMan 12 , where the object branch is not the primary challenge. Across both benchmarks, \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp improves reconstruction fidelity and perceptual quality, especially in settings with occlusion, moving foreground content, and weakly observed human regions. The main contributions of this work are as follows: • We introduce a compositional Gaussian-splatting formulation for monocular human–object scene reconstruction, separating the video into an articulated human branch, a rigid manipulated-object branch, and a static scene branch with distinct motion models. • We propose reliability-aware supervision for noisy monocular inputs: visibility-aware anchoring preserves weakly observed human regions. • We design a six-stage optimization curriculum that stabilizes the human and object branches before full compositional fusion with the scene, reducing competition between components that share pixels. • We validate the method on HOSNeRF and NeuMan, showing improved full-frame, human-focused, and human–object reconstruction quality. 2 Related Work 2.1 Dynamic Gaussian Splatting Neural radiance fields established high-quality view synthesis from posed images 19 , while 3D Gaussian Splatting (3DGS) replaced dense implicit evaluation with an explicit anisotropic Gaussian representation and visibility-aware rasterization for real-time rendering 13 . Dynamic scene methods extend this idea by modeling time-varying radiance fields with deformation fields, factorized space-time grids, or explicit dynamic Gaussians 22 ; 23 ; 35 ; 8 ; 5 ; 34 ; 41 ; 1 ; 15 ; 17 . These methods are effective for generic deforming scenes, but monocular human–object videos contain components with different physical constraints: articulated body motion, approximately rigid object motion, and mostly static scene structure. \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp therefore uses separate human, object, and scene branches rather than asking a single dynamic field to explain all pixels. 2.2 Human Gaussian Splatting Human-specific neural rendering uses body models such as SMPL/SMPL-X to factor pose from appearance 18 ; 25 ; 3 . NeRF-based methods such as Neural Body, HumanNeRF, NeuMan, and Vid2Avatar combine articulated human priors with radiance fields for monocular free-viewpoint rendering 26 ; 33 ; 12 ; 9 . More recent Gaussian approaches improve efficiency and editability by learning canonical human Gaussians that are deformed into posed frames 14 ; 11 ; 10 ; 27 ; 4 . Human-scene methods further reconstruct the moving person and static environment jointly 40 ; 42 . These methods retain animatability and achieve strong rendering quality, but monocular training still leaves persistently hidden body regions underconstrained, especially when objects or scene structure occlude the body. Occlusion-aware methods address this with explicit occluder or visibility modeling 37 ; 36 ; \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp instead uses a complete canonical human prior and modulates its influence by per-Gaussian visibility, allowing observed regions to adapt while preserving plausible completion in unobserved areas. Overall, human Gaussian methods show that body-model guidance and canonical human Gaussians are strong priors for animatable reconstruction, but monocular videos still require careful handling of occlusion and weakly observed regions. This motivates the visibility-aware human branch in \BeginAccSupp method=pdfstringdef,unicode,ActualText= CoGS CoGS \EndAccSupp : the canonical prior supports initialization and