GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting
Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker · 2026
Read GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting by Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18912.txt.
Introduction
Aboukhadra and co-authors rebuild hand-object action from a single color video. Their GHOST system uses flat Gaussian discs for both hands and objects, with no preset object class. A prior-based step fills in hidden object parts. A grasp-aware step fixes hand placement and object scale for real contact. A background rule stops hidden object zones from being punished. The system runs about ten times faster than past class-free methods. Tests on three data sets show top scores for 3D shape and image quality.
Catalog record
| CREATOR | Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40317 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18912
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini, Abdalla Arafa, Jameel Malik, Ahmed Elhayek, Didier Stricker Abstract Understanding realistic hand–object interactions from monocular RGB videos is essential for AR/VR, robotics, and embodied AI. Existing methods rely on category-specific templates or heavy computation, yet still produce physically inconsistent hand–object alignment in 3D. We introduce GHOST ( G aussian H and- O bject S pla T ting ) \textbf{GHOST}\ (\textit{{G}aussian {H}and-{O}bject {S}pla{T}ting}) , a fast, category-agnostic framework for reconstructing dynamic hand-object interactions using 2D Gaussian Splatting . GHOST represents both hands and objects as dense, view-consistent Gaussian discs and introduces three key innovations: (1) a geometric-prior retrieval and consistency loss that completes occluded object regions, (2) a grasp-aware alignment that refines hand translations and object scale to ensure realistic contact, and (3) a hand-aware background loss that prevents penalizing hand-occluded object regions. GHOST achieves complete, physically consistent, and animatable reconstructions from a single RGB video while running an order of magnitude faster than prior category-agnostic methods. Extensive experiments on ARCTIC, HO3D, and in-the-wild datasets demonstrate state-of-the-art accuracy in 3D reconstruction and 2D rendering quality, establishing GHOST as an efficient and robust solution for realistic hand-object interaction modeling. Code is available at https://github.com/ATAboukhadra/GHOST . Figure 1 : Our method reconstructs complete 3D hand–object interactions from a single monocular RGB video—recovering full object surfaces and realistic hand contact even under severe occlusions—while enabling fast, accurate, category-agnostic reconstruction and novel-view rendering. 1 Introduction Hands enable nearly every form of physical interaction we perform, such as grasping, manipulating objects, operating devices, or expressing intent 16 ; 21 ; 3 ; 51 ; 54 . As a result, reconstructing realistic 3D hand–object interactions from monocular RGB videos has become fundamental for virtual reality (VR), augmented reality (AR), teleoperation, and embodied AI. Achieving this, however, remains challenging: mutual occlusions between hands and objects, large variation in object topologies, and inherent depth ambiguities in monocular settings often lead to unstable scales and physically inconsistent contact. Although significant progress has been made, existing hand-object reconstruction methods typically fall into two categories. Template-based approaches 17 ; 18 ; 6 can generate high-quality meshes, but are restricted to a small, fixed set of known object shapes, restricting their ability to generalize to novel or unconventional objects encountered in real-world scenarios. Category-agnostic methods remove this constraint but are often computationally expensive 13 ; 35 , struggle to recover missing geometry under occlusion 1 , or require multi-view supervision 39 . NeRF-based pipelines such as HOLD 13 achieve photorealistic rendering, yet demand hours of per-sequence optimization. Recent Gaussian-splatting frameworks improve runtime and provide explicit 3D representations 35 ; 39 . However, these methods still tend to produce unrealistic hand–object contact under severe occlusions, and their optimization remains time-consuming, limiting practical applicability. To address these challenges, we present GHOST , a fast, category-agnostic framework for reconstructing realistic bimanual hand-object interactions from monocular RGB videos. GHOST represents hands and objects as dense, view-consistent Gaussian discs, enabling complete and physically consistent reconstructions even under heavy occlusions. By integrating grasp-aware reasoning with geometric object priors to recover hidden surfaces, GHOST achieves high-quality, photorealistic rendering and produces animatable reconstructions. The framework operates in three stages: (1) Preprocessing, which initializes hand meshes and retrieves object priors; (2) hand-object alignment, which refines scale and hands translation through grasp-aware reasoning; and (3) Gaussian-splatting optimization, which jointly reconstructs hands and objects with occlusion-aware consistency. Together, these components enable GHOST to achieve state-of-the-art performance in both 3D accuracy and 2D rendering fidelity (See Fig. 1 ) while running over an order of magnitude faster than previous category-agnostic methods. Our contributions are summarized as follows: • A fast, category-agnostic framework for reconstructing animatable bimanual hand-object interactions from monocular RGB sequences. • A prior-aware reconstruction strategy that fills occluded object regions using geometric consistency from retrieved priors. • A grasp-aware alignment that ensures realistic and stable hand-object contact. • A hand-aware background loss that preserves valid object regions despite persistent occlusion. • State-of-the-art performance on both 3D reconstruction and 2D rendering metrics, while achieving over 13× faster runtime than prior category-agnostic approaches. We conduct an extensive evaluation across three datasets: ARCTIC, HO3D, and in-the-wild videos, covering both 3D geometric and 2D photometric metrics. GHOST consistently surpasses previous methods in hand-object reconstruction accuracy and rendering quality, establishing a new benchmark for fast and realistic category-agnostic hand-object reconstruction. 2 Related Work The problem of realistic hand–object reconstruction spans multiple subdomains, including monocular hand reconstruction under occlusion, animatable avatars, and category-agnostic interaction reconstruction. 3D Hand Reconstruction from RGB under Occlusions. Monocular hand reconstruction has been extensively studied in 38 ; 40 ; 12 ; 1 . Recent Transformer-based approaches 37 ; 29 ; 38 ; 40 ; 1 significantly improve robustness under occlusions, achieving accurate 3D hand meshes. HaMeR 38 , in particular, provides a strong baseline trained on a large-scale dataset and generalizes well to in-the-wild scenarios, making it suitable for initializing hands in interaction pipelines. Neural and Gaussian-based Avatars. Neural implicit representations such as NeRFs 33 , and the more recent explicit formulations like Gaussian Splatting 25 , have become standard for photorealistic, animatable human avatars 5 ; 41 ; 36 . Recent works 34 ; 27 ; 36 ; 49 initialize a set of 3D Gaussians on the surface of a canonical SMPL 31 or FLAME 28 mesh and animate them through parameter-controlled skeletal motion. Rendering is performed by projecting the Gaussians onto the image plane, where they are blended to produce smooth, photorealistic images of the avatar. Hand-Object Interaction Reconstruction. Objects, unlike hands, lack consistent topology or parametric structure and exhibit wide variations in shape, material, and appearance. Early approaches 19 ; 17 ; 18 jointly reconstructed hand and object meshes from RGB inputs, enforcing contact through attraction–repulsion losses and leveraging photometric or silhouette consistency. However, these methods rely on fixed object templates, limiting generalization. Template-free representations such as THOR-Net 1 and ShapeGraformer 2 addressed this limitation by modeling objects as spherical deformations with shared topology, enabling category-agnostic learning. Category-agnostic Hand-Object Reconstruction. Building on implicit neural fields 33 , several category-agnostic pipelines 13 ; 52 ; 53 reconstruct hand–object interactions directly from monocular sequences. HOLD 13 uses off-the-shelf hand pose estimators 29 ; 38 , SAM-based object segmentation 26 ; 9 , and HLoc-based SfM 46 ; 47 ; 45 to recover geometry via NeRF optimization, but requires hours per sequence. BIGS 35 replaces the NeRF stage with Gaussian Splatting 25 , slightly improving runtime and explicitness of the representation. However, these methods still produce unrealistic contact under occlusions and high computational cost, motivating our efficient and physically consistent Gaussian-based framework. Figure 2 : (Top) Overview of our pipeline, which consists of three stages. In preprocessing , we extract hand meshes, camera poses, and object information (i.e. mask, point cloud, and geometric prior). During hand–object alignment , object’s scale and hand translations are optimized using grasp-aware and temporal reasoning. In the Gaussian Splatting stage , hands and objects are jointly reconstructed with occlusion-aware losses. (Bottom) Retrieval and alignment pipeline used to obtain the object’s geometric prior. 3 Method Our method reconstructs hand–object interactions from monocular RGB videos. It consists of three main stages: