Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking Journal: Journal of Visual Communication and Image Representation
Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando · 2025
Read Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking Journal: Journal of Visual Communication and Image Representation by Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19928.txt.
Introduction
Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, and Subha Fernando improve transformer-based visual tracking. One-stream trackers join template and search tokens so all can attend together, but too many background tokens then reach the target template and blur it. Earlier pruning often cut tokens near the target and lost useful context.
Their CPDATrack learns to score each search token for its chance of belonging to the target, then prunes weak background tokens while keeping nearby context. A selective attention step blocks search-to-template attention early, then lets only likely target tokens from a local region attend later. The tracker reaches top results on several benchmarks, with a strong overlap score on GOT-10k.
Catalog record
| CREATOR | Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking Journal: Journal of Visual Communication and Image Representation Janani Kugarajeevan, Thanikasalam Kokul, Amirthalingam Ramanan, Subha Fernando Abstract One-stream Transformer-based trackers have demonstrated remarkable performance by concatenating template and search region tokens, thereby enabling joint attention across all tokens. However, enabling an excessive proportion of background search tokens to attend to the target template tokens weakens the tracker’s discriminative capability. Several token pruning methods have been proposed to mitigate background interference; however, they often remove tokens near the target, leading to the loss of essential contextual information and degraded tracking performance. Moreover, the presence of distractors within the search tokens further reduces the tracker’s ability to accurately identify the target. To address these limitations, we propose CPDATrack, a novel tracking framework designed to suppress interference from background and distractor tokens while enhancing computational efficiency. First, a learnable module is integrated between two designated encoder layers to estimate the probability of each search token being associated with the target. Based on these estimates, less-informative background tokens are pruned from the search region while preserving the contextual cues surrounding the target. To further suppress background interference, a discriminative selective attention mechanism is employed that fully blocks search-to-template attention in the early layers. In the subsequent encoder layers, high-probability target tokens are selectively extracted from a localized region to attend to the template tokens, thereby reducing the influence of background and distractor tokens. The proposed CPDATrack achieves state-of-the-art performance across multiple benchmarks, particularly on GOT-10k, where it attains an average overlap of 75.1%. All code, models, and results associated with this work are publicly accessible at https://github.com/JananiKugaa/CPDATrack.git . 1 Introduction Visual Object Tracking (VOT) is a fundamental computer vision task that aims to estimate the state of a target object throughout a video sequence, starting from its initial position in the first frame. It is essential for numerous real-world applications, including autonomous driving in dynamic environments [ 38 ] , intelligent video surveillance for activity monitoring [ 2 ] , mobile robotic operations [ 39 ] , augmented reality implementations [ 37 ] , and human-computer interaction systems [ 1 ] . Despite significant advancements in tracking approaches over the years, developing a robust and generalizable VOT approach remains challenging due to real-world complexities like occlusions, abrupt motion, illumination changes, cluttered backgrounds, and distractor objects. Most of the early deep learning-based VOT approaches were Siamese-based convolutional neural network (CNN) trackers [ 3 , 26 , 65 , 46 , 34 ] . Although these approaches balance accuracy and efficiency, their reliance on CNNs limits global context modeling, and the use of linear cross-correlation fails to capture complex spatial relationships in challenging scenarios such as occlusion, deformation, and fast motion. To address these limitations, recent trackers have shifted toward Transformer-based models [ 22 ] , achieving state-of-the-art accuracy through enhanced global context modeling and improved ability to capture long-range dependencies. Initially, Transformers [ 45 ] were introduced into VOT as a correlation module to enhance feature interaction within CNN-based tracking frameworks. These hybrid CNN–Transformer trackers [ 7 , 58 , 51 , 42 , 54 , 62 , 53 , 44 , 27 , 56 ] replace the traditional cross-correlation operation with an attention mechanism, thereby enhancing robustness. However, since they still relied on CNNs for feature extraction, their ability to capture global contextual information is limited. To address this, subsequent trackers evolved into two-stream Transformer architectures [ 29 , 13 , 19 , 17 ] , eliminating CNNs entirely and processing the template and search regions through separate, identical Transformer encoders. Although two-stream Transformer trackers enable deeper global representation learning, the lack of interaction between features during extraction limits their performance. Recent trackers [ 57 , 5 , 10 , 6 , 14 , 52 , 4 , 41 , 21 , 66 , 15 , 23 ] address this limitation by adopting a single Transformer architecture for simultaneous feature extraction and relation modeling. These one-stream Transformer trackers enhance the tracker’s discriminative ability by jointly processing the template and search patches, thereby achieving state-of-the-art accuracy. Figure 1: Comparison of token pruning strategies: (a) the actual search region; (b) token elimination in OSTrack [ 57 ] using conventional background pruning, which may inadvertently discard essential contextual tokens; and (c) the proposed context-aware token pruning method, which preserves a local contextual zone around the target while removing distant background tokens. One-stream Transformer trackers process the search region and target template tokens jointly, enabling bidirectional interactions between them through the self-attention mechanism. Although such bidirectional interaction offers several advantages, it also introduces certain challenges. The search region is nearly twice as large as the target template and therefore contains substantial background information, including distractors resembling the target, which contaminate the template features and weaken the model’s discriminative capability. In addition, the large number of search tokens raises the computational burden of the tracker due to the quadratic nature of self-attention with respect to input length, which severely limits the feasibility of real-time tracking. A few trackers have recognized this issue and proposed token pruning or token fusion strategies to eliminate background tokens from the search region. These token pruning [ 57 , 24 , 23 , 31 ] and token fusion methods [ 50 , 60 ] identify less informative tokens in the search region based on their attention scores with respect to the center token of the target template, and subsequently eliminate or merge them to improve computational efficiency. Although these trackers significantly enhance tracking efficiency, their accuracy does not improve substantially and, in some cases, even declines due to several factors. First, identifying background tokens solely based on attention scores often generalizes poorly across varying target and background distributions, as these scores may not reliably reflect the true informativeness of the tokens. In addition, as illustrated in Fig. 1 (b), background token elimination methods may also remove nearby target tokens, leading to the loss of essential contextual information required for accurate tracking. Furthermore, fusion-based approaches may inadvertently introduce irrelevant background features, which can blur important spatial details and adversely affect tracking accuracy. Another group of trackers [ 14 , 15 , 32 , 23 ] proposed various attention mechanisms to reduce the influence of background tokens in the search region on the target tokens, thereby preserving discriminative target features. Some trackers [ 23 ] employed the attention scores of search region tokens relative to the centre token of the target template to identify background tokens, and subsequently suppressed their attention on target tokens in deeper encoder layers to preserve discriminative target feature representations. While this approach enhances accuracy, its sole reliance on attention scores may occasionally result in misclassification of target tokens as background tokens. Another group of trackers [ 14 , 15 , 32 ] integrated a learnable module that classifies search region tokens into target and background categories at each encoder layer, followed by suppressing the attention of background tokens on the target template. Although this strategy appears promising, the module’s dynamic classification of tokens in each layer often results in the same tokens being identified as target in some layers and as background in others, owing to the varying levels of semantic