LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos
Daniel Etaat, Dvij Kalaria, Nima Rahmanian, S. Shankar Sastry · 2025
Read LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos by Daniel Etaat, Dvij Kalaria, Nima Rahmanian, S. Shankar Sastry (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20936.txt.
Introduction
Daniel Etaat and co-authors teach a system to anticipate table tennis hits from single camera video. Their 2025 paper notes that champions win time by reading an opponent's intent early, not just by moving fast. Past robot players often lacked that anticipation, and past forecasting work was limited by small, narrow datasets.
The team offers two parts: a scalable pipeline that rebuilds monocular match video in 3D, and a controller that anticipates opponent actions while tracking its own uncertainty. In simulation the policy raised the ball return rate against high speed hits from about half to 59 percent, against a baseline policy with no anticipation. The work is a step toward an anticipatory playing agent.
Catalog record
| CREATOR | Daniel Etaat, Dvij Kalaria, Nima Rahmanian, S. Shankar Sastry |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LATTE-MV: Learning to Anticipate Table Tennis Hits from Monocular Videos Daniel Etaat, Dvij Kalaria, Nima Rahmanian, S. Shankar Sastry Abstract Physical agility is a necessary skill in competitive table tennis, but by no means sufficient. Champions excel in this fast-paced and highly dynamic environment by anticipating their opponent’s intent – buying themselves the necessary time to react. In this work, we take one step towards designing such an anticipatory agent. Previous works have developed systems capable of real-time table tennis gameplay, though they often do not leverage anticipation. Among the works that forecast opponent actions, their approaches are limited by dataset size and variety. Our paper contributes (1) a scalable system for reconstructing monocular video of table tennis matches in 3D and (2) an uncertainty-aware controller that anticipates opponent actions. We demonstrate in simulation that our policy improves the ball return rate against high-speed hits from 49.9% to 59.0% as compared to a baseline non-anticipatory policy. Project website : https://sastry-group.github.io/LATTE-MV/ 1 Introduction Table tennis has served as a testbed for robotics research since the 1980s [ 4 ] , presenting unique challenges due to the game’s high speed and dynamic environment. The sport requires targeted perception, precise motor control, and strategic decision-making, pushing the boundaries of robotic capabilities in sensing, actuation, and real-time processing. Recent advancements in deep learning have propelled the development of robotic systems capable of playing table tennis [ 1 , 10 , 31 , 15 , 18 , 11 ] . These systems have achieved milestones such as cooperative rallying with humans [ 1 ] and competitive play [ 10 ] . Despite these achievements, current state-of-the-art systems struggle against professional players in fast-paced rallies. One explanation for this shortcoming is that the state-of-the-art systems are unable to anticipate their opponent’s future intent when returning high-speed hits. Figure 1 : Example of a reconstructed frame. Human-action anticipation has been widely studied in robotics [ 17 , 20 , 40 ] and sports [ 9 , 28 , 42 ] , where it is often regarded as a distinguishing characteristic between amateur and expert athletes [ 9 ] . Despite these findings, anticipation is yet to play a prominent role in strategic algorithms for table tennis robotics. Previous efforts to forecast opponent serves [ 43 ] and incorporate anticipatory action selection [ 41 , 39 ] have been constrained by small datasets. Access to a large-scale, high-quality dataset of competitive gameplay, a resource currently unavailable, may streamline the development of behavior-aware learning-based algorithms – such as an anticipatory mechanism. To address this gap, we introduce LATTE-MV ( L earning A nticipation for T able T ennis E xecution from M onocular V ideos) that primarily contributes a novel system for reconstructing monocular video of table tennis gameplay in 3D. By using monocular videos as input, our system scales to the wealth of publicly available footage of competitive gameplay. To the best of our knowledge, this work marks the first attempt at collecting a dataset of table tennis gameplay at scale without specialized recording equipment. We process \sim 50 hours of table tennis gameplay into 73,222 exchanges, surpassing the size of the largest collected dataset publicly referenced (14,241 segments in [ 10 ] ). We note that this is not a fair comparison, though, since our reconstruction algorithm does not recover racket poses and our reconstructions are likely less accurate. We also note that our data is composed of human-human gameplay featuring professional athletes, as opposed to human-robot gameplay or amateur human-human gameplay. With this dataset in hand, we train a transformer-based generative model [ 35 ] to learn the dynamics of table tennis gameplay. This model serves as a foundation for anticipatory action prediction, allowing us to forecast an opponent’s future shots based on their prior actions. We further reason about epistemic uncertainty in the model’s prediction through conformal bounds which can be used to filter out uncertain predictions. We demonstrate in simulation that our model’s predictions can be used to improve the ball return rate of a KUKA robot arm from 49.9% to 59.0% as compared to a baseline non-anticipatory policy. In conclusion, we make the following contributions: 1) we unify the use of various pre-trained models and develop a novel reconstruction system to extract expert gameplay data from large-scale open-world videos; 2) we apply this dataset to learn an uncertainty-aware anticipatory control algorithm. We deploy our controller on a simulated robot table tennis player and show that it improves the ball return rate from 49.9% to 59.0% . 2 Related Works Reconstruction of Sports Gameplay from Video Reconstructing sports footage is a common area of research within the computer vision community. Some works focus on entity tracking and segmentation on 2D images. [ 36 ] trains a neural network to perform real-time player, ball, and table segmentation and [ 8 ] trains a network for high-speed ball tracking. General purpose pre-trained object detection and image segmentation models like YOLO [ 26 ] or SAM [ 19 ] can also be used for these tasks. A more difficult problem is that of reconstructing 3D information from sports footage. While some works assume access to multiple camera feeds [ 31 , 10 ] or RGB-D camera feed [ 30 ] , many do not make this assumption. For instance, [ 13 ] uses pre-defined court models for camera calibration, [ 45 ] uses open-source models to reconstruct player joint positions, [ 7 , 29 ] reconstruct 3D ball trajectories by using simplistic projectile models, and [ 46 ] reconstructs 3D ball trajectories from low frame-rate footage. Very few works have combined these methods to provide a simultaneous reconstruction of the player and ball from monocular video. A series of papers in the mid-2000s approached this problem for soccer gameplay [ 3 , 44 , 22 ] . To the best of our knowledge, no one has attempted this for table tennis. Anticipation in Table Tennis Robotics Recent works have designed learning-based systems for various tasks in table tennis robotics [ 1 , 10 , 31 , 15 , 18 , 11 ] , but few have incorporated anticipation into their decision-making algorithm. [ 41 , 39 ] model the unknown human intent using IDDM (Intention-Driven Dynamics Models), a latent variable model, and [ 41 ] uses this information to improve the table tennis playing ability of a simulated robot. [ 43 ] trains an LSTM to forecast the landing point of a serve. These works all use relatively small datasets (less than 1000 table tennis exchanges) of gameplay collected in-house. We differ substantially from these works by leveraging a large dataset of expert gameplay. Uncertainty Estimation Quantifying uncertainty in anticipatory predictions is crucial for deploying a reliable table tennis policy [ 41 ] . The primary tools for performing uncertainty estimation with neural networks are ensemble-based methods [ 25 , 12 , 34 ] , which combine outputs from multiple independently trained models to reduce variance and improve robustness, and conformal prediction [ 37 , 2 ] , which provides calibrated confidence intervals around predictions with user-defined error rates. We combine both methods in this work. 3 Reconstructing Table Tennis Gameplay In this section, we present our