History-Aware Transformation of ReID Features for Multiple Object Tracking
Ruopeng Gao, Yuyao Wang, Chunxu Liu, Limin Wang 🖂 · 2025
Read History-Aware Transformation of ReID Features for Multiple Object Tracking by Ruopeng Gao, Yuyao Wang, Chunxu Liu, Limin Wang 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12562.txt.
Introduction
Ruopeng Gao, Yuyao Wang, Chunxu Liu, and Limin Wang argue that tracking features should be tailored to each video, not borrowed unchanged from re-identification. Multi-object tracking leans on appearance features to link detections across frames. Those features are usually applied uniformly, with simple similarity scores, as if the task were generic retrieval. Within one video, though, the only rivals that matter are that video's own targets.
Their training-free method builds context from the historical features of established tracks. A Fisher linear discriminant fitted to that context projects features into a sequence-specific space where the actual targets separate better. Experiments across diverse re-identification backbones show marked, consistent tracking gains. No retraining is needed. The moral: discrimination is local, and features should be too. Code is released.
Catalog record
| CREATOR | Ruopeng Gao, Yuyao Wang, Chunxu Liu, Limin Wang 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
History-Aware Transformation of ReID Features for Multiple Object Tracking Ruopeng Gao, Yuyao Wang, Chunxu Liu, Limin Wang 🖂 Abstract In Multiple Object Tracking (MOT), Re-identification (ReID) features are widely employed as a powerful cue for object association. However, they are often wielded as a one-size-fits-all hammer, applied uniformly across all videos through simple similarity metrics. We argue that this overlooks a fundamental truth: MOT is not a general retrieval problem, but a context-specific task of discriminating targets within a single video. To this end, we advocate for the adjustment of visual features based on the context specific to each video sequence for better adaptation. In this paper, we propose a history-aware feature transformation method that dynamically crafts a more discriminative subspace tailored to each video’s unique sample distribution. Specifically, we treat the historical features of established trajectories as context and employ a tailored Fisher Linear Discriminant (FLD) to project the raw ReID features into a sequence-specific representation space. Extensive experiments demonstrate that our training-free method dramatically enhances the discriminative power of features from diverse ReID backbones, resulting in marked and consistent gains in tracking accuracy. Our findings provide compelling evidence that MOT inherently favors context-specific representation over the direct application of generic ReID features. We hope our work inspires the community to move beyond the naive application of ReID features and towards a deeper exploration of their purposeful customization for MOT. † † footnotetext: 🖂 : Corresponding author. 1 Introduction Multiple Object Tracking (MOT) is a fundamental computer vision task that aims to detect objects and maintain their identities across video frames. Its primary goal is to generate a distinct trajectory for each target by associating its corresponding detections over time [ 32 ] . As a critical component for understanding dynamic scenes, MOT serves as an essential prerequisite for a wide range of downstream applications, such as autonomous driving [ 23 ] , human behavior analysis [ 9 ] , trajectory forecasting [ 25 ] , and public surveillance. The tracking-by-detection paradigm [ 3 , 63 , 5 ] has long been the dominant and most widely adopted approach in the field of multiple object tracking. According to the task definition, it decouples the complex tracking problem into two sequential subtasks: first, an object detector localizes all targets within each frame, and second, an association algorithm links these detections across frames to form individual trajectories. As the former step is well-addressed by powerful detectors [ 19 , 51 ] , the crux of this paradigm lies in the association stage. To solve this association problem, most methods [ 64 , 5 , 13 ] model existing trajectories with discriminative cues and then allocate identities by minimizing the matching cost. Given that distinct targets often exhibit unique visual characteristics, appearance has emerged as a powerful and prevalent discriminative feature for trajectory modeling. In practice, visual features are typically extracted using off-the-shelf Re-Identification (ReID) models [ 35 ] , and a cost matrix is then formulated by the cosine distance. Despite its demonstrated success [ 56 , 64 , 1 , 60 ] , a latent contradiction persists within this paradigm. According to the definition, the goal of a general ReID model is to learn a universal feature representation capable of distinguishing any given identity from a large, open set. In contrast, the challenge within MOT is to discriminate only between the limited set of targets appearing in a specific video, which constitutes a more nuanced, expert-level requirement. As illustrated in Fig. 1(a) , targets within the same video sequence always exhibit a high degree of similarity, making them difficult to distinguish in a generic, globally-trained feature space [ 35 ] . Furthermore, this intra-sequence similarity causes their representations to cluster within a confined subspace of the original space, leading to redundancy, as shown in Fig. 1(b) . Based on the foregoing observations, a natural question arises: can we seek a specialized representation subspace for MOT, one that is more focused on distinguishing identities within the constrained set of a given sequence? (a) High similarity between the ReID features of different trajectories can create potential ambiguity. (b) ReID features of individual sequences form compact clusters in the original space. Figure 1 : Visualization of ReID features in the original representation space [ 35 ] . In this paper, we first confirm the significant influence of representation discriminability on tracking performance. Accordingly, we posit that an ideal representation space should be structured for each sequence to minimize intra-trajectory distances while maximizing inter-trajectory distances. Such a configuration would effectively increase the separability between positive and negative pairs, thereby enhancing the tracking ability. Coincidentally, this principle is conceptually analogous to the objective of Fisher Linear Discriminant (FLD) [ 16 ] , where trajectories are viewed as different categories. Since tracking is an online, continuous inference process, the established historical context can be regarded as a dynamic approximation of the data distribution for the current sequence. Building on these discussions, in practice, we feed the existing trajectories as conditional input into the FLD algorithm. By computing the closed-form solution, we derive a projection matrix that maps the original features onto a more discriminative subspace tailored for separating different trajectories. The experiment results reveal that this simple yet effective feature transformation substantially enhances discriminability and boosts overall tracking performance. Nevertheless, we revisit this process and argue that MOT possesses some task-specific demands that should not be overlooked. Firstly, since a target’s features gradually change over time in online tracking, we use a temporally weighted average to construct the trajectory’s center, rather than the naïve averaging. This makes the resulting representation better suited for the similarity assessment required at the current timestep. Secondly, historical trajectories are not always reliable due to occlusions and tracking errors. Moreover, the tracker needs to handle newborn objects, which are not considered in the transformed space. These challenges underscore the importance of retaining the original feature space with its strong robustness and generalization capabilities. To this end, we combine the similarity scores from both the general and specialized representations, thereby leveraging the complementary strengths of each. To clearly validate the impact of our ReID feature transformation, our experiments are primarily conducted on trackers that rely solely on appearance [ 35 , 29 ] . This approach minimizes the complex designs and potential interference introduced by other tracking cues [ 55 , 3 ] . In practice, we build a ReID-based tracker upon the most widely-used ReID model [ 35 ] in the MOT community [ 60 , 37 ] and validate the effectiveness of our components. Relying solely on the ReID cue, our method achieves significant performance improvements. Remarkably, in some scenarios [ 10 ] , our algorithm substantially outperforms methods that combine multiple clues [ 1 , 10 , 37 ] , establishing a new state-of-the-art result. This finding strongly indicates that the full potential of appearance information has been underestimated in past research. We also confirm the generalization capability of our proposed method by applying it to [ 29 ] with various visual encoders [ 21 , 66 , 27 , 31 ] , observing stable performance boosts across every case. Additionally, we conduct experiments on several hybrid-based methods [