TRec: Learning Hand-Object Interactions through 2D Point Track Motion

Dennis Holzmann, Sven Wachsmuth · 2026

Read TRec: Learning Hand-Object Interactions through 2D Point Track Motion by Dennis Holzmann, Sven Wachsmuth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03667.txt.

Introduction

Dennis Holzmann and Sven Wachsmuth show that tracking plain points across video frames can substantially improve hand-object action recognition. Most systems rely on image appearance, body pose estimates, or both, and skip raw motion. The pair instead sample random points in the first frame and follow them with a tracker called CoTracker, feeding the paths plus frames into a transformer model.

Remarkably, gains appeared even when the model saw only the first frame plus the point tracks, without the full video. Adding tracks consistently beat the same model trained without motion information. The method needs no hand or object detectors at all. As a light and simple motion cue, point tracks could strengthen many video understanding systems, from assistive robots to activity monitoring.

Catalog record

CREATORDennis Holzmann, Sven Wachsmuth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TRec: Learning Hand-Object Interactions through 2D Point Track Motion Dennis Holzmann, Sven Wachsmuth Abstract We present a novel approach for hand-object action recognition that leverages 2D point tracks as an additional motion cue. While most existing methods rely on RGB appearance, human pose estimation, or their combination, our work demonstrates that tracking randomly sampled image points across video frames can substantially improve recognition accuracy. Unlike prior approaches, we do not detect hands, objects, or interaction regions. Instead, we employ CoTracker to follow a set of randomly initialized points through each video and use the resulting trajectories, together with the corresponding image frames, as input to a Transformer-based recognition model. Surprisingly, our method achieves notable gains even when only the initial frame and the point tracks are provided, without incorporating the full video sequence. Experimental results confirm that integrating 2D point tracks consistently enhances performance compared to the same model trained without motion information, highlighting their potential as a lightweight yet effective representation for hand-object action understanding. Keywords: action recognition, computer vision, motion-based recognition. 1 Introduction The task of action recognition involves identifying and classifying human actions from video data. While many prior works focus on generic third-person datasets that capture large-scale body movements, our study targets hand-object interaction scenarios, where meaningful actions are characterized by fine-grained manipulations rather than full-body motion. Such datasets can include both egocentric and third-person viewpoints, providing diverse perspectives on close-range interactions between hands and objects. This perspective introduces unique challenges, including strong camera motion, frequent occlusions, and limited field of view, all of which complicate the modeling of both actor and object dynamics. The prominent camera motion paired with the exceptional close and fast-moving background covering an extensive amount of the image pose an intricate challenge of modeling both actor and object dynamics. The interplay of camera motion combined with the rapidly moving and closely captured background dominating the visual field make it particularly challenging to model both actor and object dynamics. Many existing approaches rely on RGB-based features. While such methods have achieved strong results, they often depend on computationally expensive or error-prone semantic detection pipelines for hands, objects, or skeletons. In this work, we investigate whether simple, semantically agnostic motion cues, specifically random 2D point tracks, can provide comparable or even superior information for recognizing hand-object interactions. By tracking multiple points on hands, objects, and background regions, our method captures diverse motion signals, including camera movement and object deformation. The difficulty of distinguishing object categories is reduced, since our method directly tracks the motion of the object in the scene. We generate these tracks using CoTracker. The classification pipeline uses the ResNet18 as backbone and a Transformer based module on top for classification. Surprisingly, our experiments show that a Transformer model trained with such point tracks, even when only the initial frame is used, can recognize actions more accurately than the RGB-only baseline using the whole video. Our main contributions are as follows: • We propose a simple and efficient method for hand-object action recognition that integrates 2D point tracks with image features. • We demonstrate that these tracks capture meaningful motion patterns spanning hand, object, and head movements, without requiring explicit semantic detection. • We show consistent performance improvements over the RGB-only baseline on an hand-object interaction benchmark. 2 Related Work Trajectory-based methods model motion by tracking key points or regions across consecutive frames [ 18 , 12 , 20 ] . They have been explored in exocentric (third-person) action recognition, where stable camera viewpoints simplify trajectory estimation. To the best of our knowledge, no prior work has investigated the use of 2D point tracks for hand-object action recognition without filtering background points, where strong camera motion and frequent occlusions make trajectory-based modeling more challenging. Additionally, in contrast to these methods we utilize a relatively sparse set of points to track within the image compared to methods using optical flow approaches. Wang et al. [ 18 ] introduced Improved Dense Trajectories , which compensate for camera motion using RANSAC-based estimation and employ a human detector to distinguish foreground motion from background motion. This approach established a strong pre-deep-learning baseline for video action recognition. Nguyen et al. [ 12 ] proposed Denser Trajectories of Anchor Points for Action Recognition , which anchor trajectories to more stable points to better capture fine-grained motion variations and increase robustness to camera movement. Hand-object detectors can improve hand-object interaction recognition. Several methods utilize some kind of hand-object detector and integrate this information into their model [ 3 , 14 , 13 , 11 ] . Chatterjee et al. [ 3 ] expanded the vocabulary of egocentric actions by linking detected hands and objects with fine-grained action categories. Shiota et al. [ 14 ] explicitly modeled hand-object contact and object state transitions, while Peirone et al. [ 13 ] proposed a zone-aware recognition frameworks which learns environmental affordances to extract environment and action information separately to train an action recognition model. Materzyńska et al. [ 11 ] proposed Something-Else , a compositional model that builds spatial-temporal interaction graphs between hands and objects, achieving strong performance on the Something-Something-v2 dataset. Although effective, these methods depend heavily on accurate hand and object detection, which can fail under occlusions or strong motion. RGB-based approaches [ 15 , 1 , 9 , 5 , 16 , 19 ] rely solely on raw visual appearance for action recognition, without incorporating explicit motion or geometric cues. Many of these high-performing models are trained on large-scale datasets, often leveraging extensive pretraining and carefully designed architectures. In contrast, we deliberately adopt a simple model design to isolate and clearly demonstrate the effectiveness of incorporating 2D point tracks. Unlike RGB-based features that primarily encode texture and color statistics, point trajectories provide explicit geometric and temporal information, capturing object and camera motion directly from the image sequence. Girdhar et al. [ 5 ] propose the Anticipative Video Transformer , which models temporal dependencies in egocentric videos to predict upcoming actions. Their model implicitly attends to image regions containing hands and objects, and its performance improves with increased temporal context. However, the paper does not clarify whether motion is inferred implicitly through appearance changes between frames. In contrast, our approach introduces motion information explicitly by integrating 2D point tracks, enabling the model to reason directly about the displacement and dynamics of scene elements rather than relying on learned appearance differences. This explicit representation of motion complements the visual features and enhances interpretability in modeling fine-grained hand-object interactions and providing robustness to camera motion. Tong et al. [ 16 ] employ masked autoencoder pretraining to improve video representation learning, achieving strong performance on egocentric and other video benchmarks. Wang et al. [ 19 ] further extend pretrained video backbone models through an interactive prototype learning framework, enhancing active-object representations by leveraging actor motion cues within RGB inputs. Point tracking aims to estimate the trajectories of 2D points across video frames, providing fine-grained motion information that supports higher-level tasks such as action recognition and object interaction analysis. Recent advances in this area have focused on improving both tracking accuracy and computational efficiency. CoTracker [ 8 ] introduced a transformer-based architecture capable of jointly tracking multiple points across frames. Subsequent versions have

Read the complete text