TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos DOI: XXXXXXX.XXXXXXX Conference: ACM Multimedia 2026; November 10–14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2026/06 3214 CCS: Computing methodologies Tracking CCS: Computing methodologies Video segmentation CCS: Computing methodologies 3D imaging CCS: Computing methodologies Trajectory modeling
Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria, Rainer Lienhart, S. Shankar Sastry · 2026
Read TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos DOI: XXXXXXX.XXXXXXX Conference: ACM Multimedia 2026; November 10–14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2026/06 3214 CCS: Computing methodologies Tracking CCS: Computing methodologies Video segmentation CCS: Computing methodologies 3D imaging CCS: Computing methodologies Trajectory modeling by Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria, Rainer Lienhart, S. Shankar Sastry (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01234.txt.
Introduction
Nima Rahmanian and five co-authors present TT4D, a pipeline and dataset for four-dimensional reconstruction of table tennis from monocular broadcast videos. The dataset provides well over a hundred hours of reconstructed singles and doubles gameplay, with camera calibrations, precise three-dimensional ball positions, ball spin, time segmentation, and human meshes over time. Such data can support virtual replay, player analysis, and robot learning.
Earlier methods segmented shots from the two-dimensional ball track before reconstruction, which collapses under occlusion and varied viewpoints. TT4D inverts the order: a learned network first lifts the full unsegmented track to three dimensions, also inferring spin and handling unreliable detections, and segmentation follows reliably. Downstream tests estimate racket pose and velocity at impact and train a generative rally model.
Catalog record
| CREATOR | Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria, Rainer Lienhart, S. Shankar Sastry |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01234
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos DOI: XXXXXXX.XXXXXXX Conference: ACM Multimedia 2026; November 10–14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2026/06 3214 CCS: Computing methodologies Tracking CCS: Computing methodologies Video segmentation CCS: Computing methodologies 3D imaging CCS: Computing methodologies Trajectory modeling Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria, Rainer Lienhart, S. Shankar Sastry Abstract . We present TT4D , a large-scale, high-fidelity table tennis dataset. It provides 140+ hours of reconstructed singles and doubles gameplay from monocular broadcast videos, featuring multimodal annotations like high-quality camera calibrations, precise 3D ball positions, ball spin, time segmentation, and 3D human meshes over time. This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning. The dataset’s combination of scale and precision is achieved through a novel reconstruction pipeline . Prior methods first partition a game sequence into individual shot segments based on the 2D ball track, and only then attempt reconstruction. However, 2D-based time segmentation collapses under occlusion and varied camera viewpoints, preventing reliable reconstruction. We invert this paradigm by first lifting the entire unsegmented 2D ball track to 3D through a learned lifting network. This 3D trajectory then allows us to reliably perform time segmentation. The learned lifting network also infers the ball’s spin, handles unreliable ball detections, and successfully reconstructs the ball trajectory in cases of high occlusion. This lift-first design is necessary, as our pipeline is the only method capable of reconstructing table tennis gameplay from general-view broadcast monocular videos. We demonstrate the dataset’s fidelity through two downstream tasks: estimating the racket’s pose & velocity at impact, and training a generative model of competitive rallies. Figure 1 . “Lift-First Pipeline" to generate TT4D, a massive multimodal 140+ hour dataset that recovers camera parameters, 3D ball positions, ball spin, and 3D human meshes over time from general-view broadcast videos. Inverting traditional logic, we directly lift entire sequences from 2D ball tracks to 3D trajectories, bypassing fragile 2D-based time segmentation. Two side-by-side 3D visualizations demonstrating the multimodal output of the Lift-First Pipeline. Each visualization shows a 2D broadcast video frame projected into a 3D grid space, originating from a camera coordinate axis. Within the 3D space, a green wireframe table tennis table is visible. A sequence of grey spheres maps the 3D ball positions across the table. Colored 3D human meshes are rendered with multiple overlapping translucent poses to illustrate their movement over time. The left visualization depicts a doubles match with four human meshes, while the right visualization depicts a singles match with two human meshes. 1. Introduction Online platforms host a vast and growing collection of high-quality competitive sports footage. This abundance of broadcast video makes 4D monocular-view reconstruction a scalable task enabling virtual replay, player analytics, and robot learning. Table tennis, in particular, serves as a challenging testbed due to its high-speed and dynamic nature. A complete analysis goes beyond human mesh recovery and ball position reconstruction: it includes estimating ball spin, which strongly influences both flight (via the Magnus effect) and bounce behavior. Extracting these signals at scale from broadcast video is difficult: the ball is small, moves rapidly, and is routinely occluded by players. This constant occlusion makes time segmentation, the task of identifying the exact moment of a hit to obtain the individual shot segments, the biggest challenge for reconstruction. Existing pipelines ( 10 ; 17 ; 9 ; 22 ; 21 ) follow a 2D-based strategy: First, use the 2D ball track to partition the sequence into individual shot segments. Then, lift each segment to 3D. This approach has limitations. Methods relying on automated 2D-based time segmentation, such as LATTE-MV ( 10 ) and TT3D ( 17 ) , often fail when the 2D ball track is interrupted by occlusions or corrupted by misdetections. Manual segmentation ( 22 ; 21 ) may be used for precise benchmarking, but is unscalable and only feasible for small test sets. In this work, we introduce the Lift-First Pipeline , which fundamentally reverses this logic. Our pipeline directly lifts the entire unsegmented 2D ball track to a continuous 3D trajectory using a learned model. Only then do we perform time segmentation in the unambiguous 3D domain. Once this continuous 3D trajectory is available, hit and bounce events can be reliably identified. This 3D-first approach is enabled by our core technical contribution: a novel Full-Sequence Lifting Network . As the first method capable of processing long and complex unsegmented sequences, it is the key enabler for our Lift-First Pipeline, making a 3D-first approach to table tennis reconstruction possible for the first time. This network is trained on a massive-scale synthetic dataset of 3 million rallies. We summarize our contributions as follows: • The “Lift-First" Reconstruction Pipeline: A new paradigm that decouples 3D reconstruction from fragile 2D-based time segmentation by first lifting the entire unsegmented sequence to 3D. • A Novel Full-Sequence Lifting Network: The core technical method, enabled by a 3M points synthetic dataset. The network is the first to process unsegmented rallies and outputs the full 3D trajectory and dense per-frame 3D spin vectors. • The TT4D Dataset: A 140+ hour multimodal dataset generated with our pipeline, featuring precise 3D ball trajectories, 3D human meshes, and two annotations previously unavailable at scale: dense ball spin and robust 3D-derived time segmentations. • Novel Downstream Applications: We demonstrate our dataset’s high fidelity by (a) introducing a new racket stroke estimation method that recovers the racket’s pose and velocity at impact and (b) training a generative Flow Matching ( 28 ) model on competitive gameplay. We will release the TT4D dataset, paving the way for new applications in computational sports science and robotics ( 10 ; 52 ; 53 ) . 2. Related Work 2D ball tracking in table tennis is challenging due to its small size, fast motion, frequent occlusions, and motion blur. Recent methods rely on deep detectors ( 19 ; 56 ; 24 ; 40 ) , with the Multiple-Input Multiple-Output (MIMO) formulation from TrackNetV2 ( 40 ) being a key breakthrough. This MIMO strategy has been adopted by subsequent works using different backbones ( 42 ; 30 ; 4 ; 20 ; 21 ) . Attention mechanisms have also been incorporated to enhance temporal feature fusion ( 18 ; 15 ; 37 ) . Methods for 3D trajectory lifting can be split into physics-based optimization and learned networks. Optimization-based methods like TT3D ( 17 ) , LATTE-MV ( 10 ) and MonoTrack ( 30 ) minimize reprojection error. These non-learning methods are inherently limited to the “Traditional Pipeline" and hence do not scale. Their optimization is already unstable for single, shot segments; extending them to a “3D-first" approach that jointly solves for the trajectory, all unknown bounce points, and all unknown hit-points is computationally infeasible. This necessitates a learning-based approach to make the Lift-First Pipeline viable. Recent learned approaches train a network to lift 2D tracks to 3D trajectories. SynthNet ( 9 ) and ( 36 ) both tackle this for tennis. Most relevant is the work of 22 , which proposes a transformer that lifts single shot segments from 2D to 3D with zero-shot generalization from synthetic to real data. This was extended in