Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

Mingqian Ji, Shanshan Zhang, Jian Yang, PCA Lab, School of Computer Science, Engineering, Nanjing University of Science, Technology, PCA Lab, School of Intelligence Science, Technology, Nanjing University { mingqianji,shanshan.zhang,csjyang}@njust.edu.cn · 2026

Read Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors by Mingqian Ji, Shanshan Zhang, Jian Yang, PCA Lab, School of Computer Science, Engineering, Nanjing University of Science, Technology, PCA Lab, School of Intelligence Science, Technology, Nanjing University { mingqianji,shanshan.zhang,csjyang}@njust.edu.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14563.txt.

Introduction

Ji, Zhang, and Yang revisit token compression for sparse multi-view 3D detectors built on vision transformers. Pruning, merging, and patch enlargement often drop background cues, break consistency, and lose fine detail, hurting 3D detection. Their SEPatch3D adjusts patch sizes live, keeping small patches for nearby objects and large patches for background scenes. Informative-patch selection and cross-granularity enhancement restore coarse-patch detail. On nuScenes and Argoverse 2, it runs up to 57 percent faster than StreamPETR and 20 percent more efficiently than ToC3D-faster, with comparable accuracy.

Catalog record

CREATORMingqian Ji, Shanshan Zhang, Jian Yang, PCA Lab, School of Computer Science, Engineering, Nanjing University of Science, Technology, PCA Lab, School of Intelligence Science, Technology, Nanjing University { mingqianji,shanshan.zhang,csjyang}@njust.edu.cn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52652
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors Mingqian Ji, Shanshan Zhang, Jian Yang, PCA Lab, School of Computer Science, Engineering, Nanjing University of Science, Technology, PCA Lab, School of Intelligence Science, Technology, Nanjing University { mingqianji,shanshan.zhang,csjyang}@njust.edu.cn Abstract Vision Transformer (ViT)-based sparse multi-view 3D object detectors have achieved remarkable accuracy but still suffer from high inference latency due to heavy token processing. To accelerate these models, token compression has been widely explored. However, our revisit of existing strategies, such as token pruning, merging, and patch size enlargement, reveals that they often discard informative background cues, disrupt contextual consistency, and lose fine-grained semantics, negatively affecting 3D detection. To overcome these limitations, we propose SEPatch3D, a novel framework that dynamically adjusts patch sizes while preserving critical semantic information within coarse patches. Specifically, we design Spatiotemporal-aware Patch Size Selection (SPSS) that assigns small patches to scenes containing nearby objects to preserve fine details and large patches to background-dominated scenes to reduce computation cost. To further mitigate potential detail loss, Informative Patch Selection (IPS) selects the informative patches for feature refinement, and Cross-Granularity Feature Enhancement (CGFE) injects fine-grained details into selected coarse patches, enriching semantic features. Experiments on the nuScenes and Argoverse 2 validation sets show that SEPatch3D achieves up to 57% faster inference than the StreamPETR baseline and 20% higher efficiency than the state-of-the-art ToC3D-faster, while preserving comparable detection accuracy. Code is available at https://github.com/Mingqj/SEPatch3D. 1 Introduction Multi-view 3D object detection has received increasing attention in recent years. This task aims to predict object categories and 3D information from multi-view RGB images, and serves as a critical component in autonomous driving. Among various detectors, sparse query-based detectors Wang et al. 2022 ; Hou et al. 2024 ; Liu et al. 2024a ; Lin et al. 2022 have recently shown strong performance on public benchmarks. These methods bypass dense BEV construction Li et al. 2022 ; Wu et al. 2025 ; Huang and Huang 2022 ; Wang et al. 2025 ; Li et al. 2023b and directly associate learnable object queries with image features, allowing them to focus on object-level information. This design enables sparse detectors to benefit more from high-quality image features, making them particularly well-suited to be paired with Vision Transformers (ViTs) Dosovitskiy 2020 , which offer strong representation capabilities. However, this performance gain comes at the cost of significant inference latency, due to the computational overhead of dense token processing Zeng et al. 2024 ; Zeng et al. 2025 ; Kim et al. 2024 ; Lin et al. 2024 , which has been largely overlooked in previous works. This motivates us to investigate efficient strategies to accelerate ViT-based sparse multi-view 3D detectors. (a) Token pruning (b) Token merging (c) Patch size enlargement (d) Ours Figure 1 : Visualization of different token compression strategies. (a) Token pruning may remove background cues crucial for hard negatives. (b) Token merging introduces irregular aggregations that disrupt semantic consistency. (c) Enlarging patch size loses fine details (red box), harming accuracy. (d) Our method enhances informative large patches (green box) to retain critical semantics. Since the computational cost of ViTs largely scales with the number of tokens, token compression has become a straightforward and effective acceleration strategy. Common techniques include token pruning, token merging, and patch size enlargement Zhang et al. 2025 . Among them, token pruning degrades low-importance tokens based on certain heuristics or learned scores. However, this process may lead to the loss of informative background regions (see Fig. 8 ), which are essential for learning hard negative samples in detection tasks. Token merging reduces the token count by combining similar or redundant tokens into more compact representations (see Fig. 1(b) ), but irregular merging disrupts contextual consistency, resulting in semantic information loss. In contrast, patch size enlargement reduces the number of tokens during the patch embedding stage by enlarging the receptive field per token (see Fig. 1(c) ). This strategy preserves the overall semantic contents, making it inherently more compatible with the demands of 3D object detection. However, as shown in Fig. 2 , we observe that simply enlarging the patch size leads to performance degradation when exceeding a certain threshold (e.g., patch size > 18). This is because coarse-grained patches may overlook fine-grained semantic cues essential for accurate object recognition and localization. The above observations motivate us to design a novel compression strategy, which not only preserves fine-grained semantic information (see Fig. 1(d) ) crucial for detection, but also is of high efficiency. Building on this insight, our key idea is to reduce the number of patches by enlarging patch sizes while enhancing semantically important regions to mitigate potential information loss. Figure 2 : Accuracy-latency under an input resolution of 640 \times 1600 on the nuScenes set. The baseline StreamPETR uses a patch size of 16. As the patch size increases, inference latency consistently decreases, but excessive enlargement leads to accuracy degradation. In contrast, our SEPatch3D- faster reduces inference latency by 57% while maintaining competitive detection accuracy. In this paper, we propose Dynamic Patch Size Selection and Informative Patch Enhancement ( SEPatch3D ) for efficient ViT-based sparse multi-view 3D object detection. Specifically, we first design the Spatiotemporal-aware Patch Size Selection (SPSS) module, which builds an indicator to estimate the object spatiotemporal distribution. Based on this, we assign small patches to scenes with nearby objects in order to preserve fine-grained details, while large patches are allocated to background-dominated scenes to reduce computational overhead. To further compensate for potential detail loss in coarse patches, we assign entropy-based importance scores to all patches by jointly considering object motions and patch contents, and select the most informative ones via the Informative Patch Selection (IPS) module, and then enhance these selected coarse patches by fusing them with fine-grained patches through a cross-attention via the Cross-Granularity Feature Enhancement (CGFE) module. We evaluate SEPatch3D on the nuScenes Caesar et al. 2020 and Argoverse 2 Wilson et al. 2021 validation datasets. On nuScenes, our method achieves up to 57% inference speedup over the StreamPETR baseline Wang et al. 2023 with less than one point performance degradation. Moreover, SEPatch3D outperforms the state-of-the-art ToC3D- faster variant Zhang et al. 2024 , achieving an additional 20% improvement in inference efficiency. Our contributions are summarized as follows: • We revisit token compression strategies when applied to 3D object detection: token pruning, merging, and patch size enlargement, and analyze their limitations, including the loss of informative backgrounds, disruption of contextual consistency, and degradation of fine-grained semantics. These motivate an efficient compression strategy that better balances accuracy and efficiency. • We propose a novel compression strategy that adaptively enlarges patch sizes according to the spatiotemporal distribution of object queries derived from motion cues. To mitigate detail loss, we introduce an entropy-based patch selection module to identify informative regions, followed by a feature enhancement module that injects fine-grained details into the selected coarse patches. • Experiments on the nuScenes and Argoverse 2 validation sets show that our method achieves faster inference while maintaining competitive accuracy against the baseline and state-of-the-art approaches. 2 Related Work Since our work focuses on the task of sparse multi-view 3D object detection and token compression for accelerating ViT-based detectors, we first provide a brief overview of token compression techniques for

Read the complete text