Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction

Shilei Wang, Gong Cheng, Pujian Lai, Dong Gao, Junwei Han · 2025

Read Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction by Shilei Wang, Gong Cheng, Pujian Lai, Dong Gao, Junwei Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11531.txt.

Introduction

Shilei Wang, Gong Cheng, Pujian Lai, Dong Gao, and Junwei Han make a fast object tracker sharper by letting it think in several states at once. Their 2025 paper addresses the tax on efficient trackers: cutting compute usually weakens the features that pin down a target. Their Multi-State Tracker generates state representations at several stages of feature extraction instead of trusting one layer.

Lightweight enhancement modules polish each state for target-specific detail, and a cross-state module lets them trade information and merge strengths. Both add-ons use a lean state-space design costing a tenth of a gigaflop and two-thirds of a million parameters. Across benchmarks the tracker beat every prior efficient rival. On GOT-10K it topped the best earlier efficient score by 4.5 points while staying quick. Code is public.

Catalog record

CREATORShilei Wang, Gong Cheng, Pujian Lai, Dong Gao, Junwei Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11531

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-State Tracker: Enhancing Efficient Object Tracking via Multi-State Specialization and Interaction Shilei Wang, Gong Cheng, Pujian Lai, Dong Gao, Junwei Han Abstract . Efficient trackers achieve faster runtime by reducing computational complexity and model parameters. However, this efficiency often compromises the expense of weakened feature representation capacity, thus limiting their ability to accurately capture target states using single-layer features. To overcome this limitation, we propose Multi-State Tracker (MST), which utilizes highly lightweight state-specific enhancement (SSE) to perform specialized enhancement on multi-state features produced by multi-state generation (MSG) and aggregates them in an interactive and adaptive manner using cross-state interaction (CSI). This design greatly enhances feature representation while incurring minimal computational overhead, leading to improved tracking robustness in complex environments. Specifically, the MSG generates multiple state representations at multiple stages during feature extraction, while SSE refines them to highlight target-specific features. The CSI module facilitates information exchange between these states and ensures the integration of complementary features. Notably, the introduced SSE and CSI modules adopt a highly lightweight hidden state adaptation-based state space duality (HSA-SSD) design, incurring only 0.1 GFLOPs in computation and 0.66 M in parameters. Experimental results demonstrate that MST outperforms all previous efficient trackers across multiple datasets, significantly improving tracking accuracy and robustness. In particular, it shows excellent runtime performance, with an AO score improvement of 4.5% over the previous SOTA efficient tracker HCAT on the GOT-10K dataset. The code is available at https://github.com/wsumel/MST. 1. Introduction Figure 1. Comparison with the state-of-the-art trackers in terms of AO performance and CPU speed on GOT-10K. Following the VOT real-time standard, we define real-time performance as achieving at least 20 frames per second (FPS). Visual object tracking is a fundamental task in computer vision, with broad applications in areas such as surveillance, autonomous driving, and human-computer interaction ( Li et al. 2019 ; Chen et al. 2021 ; Mayer et al. 2021 ; Wang et al. 2024 ; Lai et al. 2023 ; Lai et al. 2025 ; Sun et al. 2024 ; Kang et al. 2024 ) . The goal is to accurately track an object across video frames, even under challenging conditions such as occlusions, appearance changes, and varying motion dynamics. Efficient object tracking methods are essential for real-time applications, particularly in resource-constrained environments like edge devices ( Yan et al. 2021b ; Henriques et al. 2014 ; Bertinetto et al. 2016 ; Kang et al. 2023 ; Cui et al. 2024 ; Wang et al. 2025 ) . These methods typically rely on lightweight models with fewer parameters to achieve fast processing speed. However, this simplification often weakens its feature extraction ability and makes it difficult to capture the complete state of the target using a single-layer representation, thus limiting its ability to handle complex tracking scenarios. To address these limitations, we propose the Multi-State Tracker (MST), a novel tracking framework designed to enhance the robustness of efficient tracking in challenging environments. MST leverages multiple state representations to provide a more comprehensive understanding of the tracked object, enabling it to better capture variations in appearance, occlusions, and motion blur. Crucially, MST is designed to significantly improve feature representation and tracking performance while introducing minimal computational and parameter overhead, making it highly suitable for real-time and resource-constrained applications. Unlike traditional methods using a single state, MST fuses multiple states for more accurate and reliable tracking. At the core of MST are three key modules: multi-state generation (MSG), state-specific enhancement (SSE), and cross-state interaction (CSI). The MSG module generates diverse state representations by modeling spatial relationships among patches extracted from both the template and the search region. These representations are then refined by the SSE module, which employs a global enhancement mechanism to emphasize target-specific characteristics, thereby making the tracker more sensitive to subtle variations in the object’s appearance. The CSI module facilitates information exchange between these states, integrating complementary features to further improve tracking performance. A key advantage of MST is its efficiency. It’s core modules SSE and CSI are both designed based on the hidden state adaptation-based state space duality (HSA-SSD), which features linear time complexity and minimal resource consumption. As a result, MST is capable of maintaining high tracking speeds even in real-time applications ( Li et al. 2022 ; Lee et al. 2025 ) . Despite its lightweight architecture, MST outperforms all previous efficient trackers across multiple datasets, significantly improving tracking accuracy and robustness. As shown in Figure 1 , MST is five times faster than the traditional tracker OSTrack (-256) ( Ye et al. 2022 ) and surpasses the previous best-performing efficient tracker, HCAT ( Chen et al. 2023 ) , by 4.5% in AO score on the GOT-10K ( Huang et al. 2019 ) dataset. Our approach provides a novel solution to balancing efficiency and robustness, enabling more reliable real-time tracking in complex scenarios. The main contributions are: • Multi-State Tracking Architecture: We introduce the Multi-State Tracker (MST), which leverages multiple state representations to enhance tracking accuracy and robustness, enabling better handling of variations in appearance, occlusions, and motion blur. • Key Technical Innovations: We propose three key modules Multi-State Generation (MSG), State-Specific Enhancement (SSE), and Cross-State Interaction (CSI), with the latter two built upon the Hidden State Adaptation-based State Space Duality (HSA-SSD). Together, these modules enable the generation of diverse state-aware features, the refinement of individual state representations, and effective information exchange across states. Importantly, they enhance tracking performance while introducing only minimal computational overhead. • State-of-the-Art Performance: Leveraging its innovative architectural design and key advancements, MST not only maintains exceptional processing speed but also delivers strong tracking performance, as demonstrated by extensive experimental evaluations across several benchmark datasets. 2. Related Work Efficient Object Tracking. Efficient object tracking has become increasingly important for real-world applications, particularly on edge devices. Early methods, such as ECO ( Danelljan et al. 2017 ) and ATOM ( Danelljan et al. 2019 ) , demonstrated real-time capabilities, but their tracking accuracy often fell short. Subsequent efforts have sought to better balance speed and performance. For example, LightTrack ( Yan et al. 2021b ) , FEAR ( Borsuk et al. 2022 ) , and HCAT ( Chen et al. 2023 ) have explored lightweight network architectures and efficient design strategies to reduce computational complexity while still delivering competitive performance. With the advent of one-stream architectures, one-stream efficient tracking frameworks such as MixFormerV2 ( Cui et al. 2024 ) and HiT ( Kang et al. 2023 ) integrate feature extraction and interaction into a unified process, achieving impressive accuracy through techniques like distillation, pruning, and innovative architectural design. However, these methods all rely on a single-layer feature representation to capture the target’s state. In contrast, our proposed Multi-State Tracker (MST) leverages multiple state representations and interactive aggregation to robustly capture diverse target variations, thereby significantly enhancing tracking robustness while maintaining extremely high computational efficiency. State Space Model. State Space Models (SSMs) ( Gu and Dao 2023 ) have recently gained considerable attention as an efficient and scalable solution for sequence modeling, offering linear time complexity while capturing long-range dependencies. These models, originating from the Kalman filter, have been widely adopted in various domains, particularly in natural language processing (NLP), where models like S4, DSS, and Mamba have demonstrated their ability

Read the complete text