STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans

Shashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swami, Shanmuganathan Raman · 2025

Read STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans by Shashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swami, Shanmuganathan Raman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13344.txt.

Introduction

Shashikant Verma and four co-authors present a 2025 paper on STEP, a system that tracks animals and people in video while estimating their poses. Watching body keypoints over time supports behavior study and action recognition. Most pose tools need fresh detections in every frame. STEP instead starts from a known target in the first frame and follows it, inspired by how brains track form and motion together.

Two added modules let a transformer model predict its own targets, removing the need for hand-fed keypoint states. The system outputs both the target's location and its keypoints frame by frame. Trained across several species, it beat existing methods in the team's tests. Skipping per-frame detection also speeds inference. The authors see uses in wildlife research, sports, and animal welfare monitoring. The design treats humans and animals in one frame.

Catalog record

CREATORShashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swami, Shanmuganathan Raman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans Shashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swami, Shanmuganathan Raman Abstract We introduce STEP, a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans. We are inspired by the fact that the human brain exploits spatiotemporal continuity and performs concurrent localization and pose estimation despite the specialization of brain areas for form and motion processing. Traditional discriminative models typically require predefined target states for determining model weights, a challenge we address through Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) Modules. These modules remove the necessity of keypoint target states as input, streamlining the process. Our method starts with a known target state in the initial frame of a given video sequence. It then seamlessly tracks the target and estimates keypoints of anatomical importance as output for subsequent frames. Unlike prevalent top-down pose estimation methods, our approach doesn’t rely on per-frame target detections due to its tracking capability. This facilitates a significant advancement in inference efficiency and potential applications. We train and validate our approach on datasets encompassing diverse species. Our experiments demonstrate superior results compared to existing methods, opening doors to various applications, including but not limited to action recognition and behavioral analysis. keywords Pose Estimation, Visual Tracking, Target Model Prediction. 1 Introduction Pose estimation is a fundamental and demanding task in computer vision, with diverse practical applications in the real world. When applied to video sequences, pose estimation proves invaluable across various uses, including but not limited to action recognition, behavioral analysis, robotics, and surveillance, demonstrating its wide-ranging utility. Analyzing poses across videos involves tackling specific challenges, such as object detection, precise frame-to-frame tracking, and anatomical keypoint localization. Motivation. Traditional top-down Pose Estimation Techniques require precise bounding box data, sourced either from automated detectors or manually annotated ground truths, to appropriately crop, center, and scale the images of the targets. These methods are designed with specific cropping standards that reflect established priors regarding the target shapes. Although these methods have been effective, their performance can diminish when encountering targets with non-standard aspect ratios, leading to less accurate pose estimation. To this end, we propose a transformer-based method to simultaneously handle tracking and pose estimation for the object within a video by concurrently estimating weights for three tasks: localization, tracking, and pose estimation. Integrating tracking eliminates the need for per-frame object detections used in existing top-down methods. Unlike these conventional approaches, our proposed method offers a robust alternative not hindered by variations in aspect ratios, enhancing its utility across diverse imaging conditions and target configurations. Pose Estimation. Pose estimation approaches can be categorized into two main paradigms: top-down and bottom-up. The top-down approaches Xu et al. (2022) Sun et al. (2019) first detect the object of interest employing object detectors and subsequently conduct pose estimation for each detection. In contrast, the bottom-up approaches Pishchulin et al. (2016) Ye et al. (2023) Insafutdinov et al. (2016) directly regress the keypoints and later concentrate on associating the detected keypoints to the object. The top-down approach achieves accuracy at a higher cost with an extra detection step, while the bottom-up paradigm, though more efficient, attains lower accuracy. Traditional top-down Pose Estimation Techniques require precise bounding box data typically, a 1.33 height-to-width ratio for human subjects to accommodate their vertical orientation and a 1.33 width-to-height ratio for animals to capture their broader horizontal spread. CNN-based models used for regressing keypoints in pose estimation, like Toshev and Szegedy (2014) Wei et al. (2016) Xiao et al. (2018) , face challenges in capturing image-specific dependencies because of static convolutional kernels. Furthermore, these models excel in revealing class-specific saliency maps rather than at establishing relationships between structural elements, such as the connection between anatomical structure and keypoint locations within the pose context Yang et al. (2021) . Visual Object Tracking. Deep neural networks have shown remarkable performance in object tracking and pose estimation tasks. Within the tracking domain, two predominant methodologies have emerged: Siamese networks Li et al. (2018) and networks integrating discriminative appearance modules Bhat et al. (2019) . Object Tracking deals with accurately identifying and tracking each target object within video streams. This field has progressed significantly due to advances in object detection technologies, with most modern trackers adopting the tracking-by-detection approach. This technique applies an object detector across successive frames to generate detections that are then used for tracking. Centernet Duan et al. (2019) is commonly employed in many tracking-by-detection systems due to its operational efficiency, whereas YOLOX Ge et al. (2021) has gained popularity for its exceptional detection accuracy. Typically, these methods incorporate the Kalman filter Welch et al. (1995) as a motion model to predict future tracklet positions. The Intersection over Union (IoU) metric is used to gauge the similarity between predicted and detected boxes for a reliable association. However, the Kalman filter is noted for its vulnerabilities, such as susceptibility to state noise and amplification of temporal discrepancies. Furthermore, the application of the Kalman filter, despite its widespread use, requires careful tuning of its parameters to mitigate the effects of its inherent vulnerabilities. To address these issues, some tracking systems have begun integrating more sophisticated deep learning-based motion models, which can offer improved robustness and accuracy in dynamic environments. Contribution. Existing top-down approaches utilize object detectors like Jiang et al. (2022) Li et al. (2022) to determine a bounding box around the object for pose estimation. Object detectors primarily focus on recognizing and localizing objects within individual images rather than maintaining a (tracking) identity across frames, creating a gap in pose analysis within video sequences. Our proposed approach addresses this gap through tracking capability while simultaneously estimating pose keypoints. Our approach uses a transformer-based method to simultaneously handle tracking and pose estimation for the object within a video by concurrently estimating weights for three tasks: localization, tracking, and pose estimation. Integrating tracking eliminates the need for per-frame object detections used in existing top-down methods. In summary, our contributions are as given below. • We conduct a comprehensive study on pose estimation task within a discriminative model prediction framework. We utilize a Transformer-based model prediction module while simultaneously tracking the object of interest— an unexplored area in previous research. • We introduce the Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) modules, which work together to accurately localize key points corresponding to the target state, ensuring both efficiency and precision in pose key point estimation. • In the inference phase, we propose a confidence-based memory update of our tracker. This update integrates previous frame outputs to supervise and estimate subsequent outputs effectively. We showcase the effectiveness of our approach through a case study on Awaji Monkey Center (AMC) video streams. We begin by reviewing relevant works on tracking and pose estimation closely related to ours in Section 2 . Following that, Section 3 provides a detailed explanation of our proposed methodology, including the network architecture and the steps involved in training STEP and performing inference. We demonstrate that our approach delivers competitive performance in pose estimation compared to state-of-the-art methods while also incorporating tracking. Notably, our method surpasses these approaches in top-down evaluations.

Read the complete text