CoCAViT: Compact Vision Transformer with Robust Global Coordination

Xuyang Wang, Lingjuan Miao, Zhiqiang Zhou · 2025

Read CoCAViT: Compact Vision Transformer with Robust Global Coordination by Xuyang Wang, Lingjuan Miao, Zhiqiang Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05307.txt.

Introduction

Xuyang Wang, Lingjuan Miao, and Zhiqiang Zhou present CoCAViT in a 2025 paper on compact vision transformers. Large visual backbones learn strong general features, and small models can match them on familiar data. Yet small models lose much more accuracy on unfamiliar data, showing weak generalization.

The authors trace that weakness to design choices that shrink the global field of window attention. Their mechanism adds dynamic global tokens that coordinate patches across the image and adapt to new domains at low cost. The resulting backbone reached strong accuracy on a standard benchmark and clear gains on out of distribution tests.

Catalog record

CREATORXuyang Wang, Lingjuan Miao, Zhiqiang Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05307

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoCAViT: Compact Vision Transformer with Robust Global Coordination Xuyang Wang, Lingjuan Miao, Zhiqiang Zhou Abstract In recent years, large-scale visual backbones have demonstrated remarkable capabilities in learning general-purpose features from images via extensive pre-training. Concurrently, many efficient architectures have emerged that have performance comparable to that of larger models on in-domain benchmarks. However, we observe that for smaller models, the performance drop on out-of-distribution (OOD) data is disproportionately larger, indicating a deficiency in the generalization performance of existing efficient models. To address this, we identify key architectural bottlenecks and inappropriate design choices that contribute to this issue, retaining robustness for smaller models. To restore the global field of pure window attention, we further introduce a Coordinator-patch Cross Attention (CoCA) mechanism, featuring dynamic, domain-aware global tokens that enhance local-global feature modeling and adaptively capture robust patterns across domains with minimal computational overhead. Integrating these advancements, we present CoCAViT, a novel visual backbone designed for robust real-time visual representation. Extensive experiments empirically validate our design. At a resolution of 224×224, CoCAViT-28M achieves 84.0% top-1 accuracy on ImageNet-1K, with significant gains on multiple OOD benchmarks, compared to competing models. It also attains 52.2 mAP on COCO object detection and 51.3 mIOU on ADE20K semantic segmentation, while maintaining low latency. Introduction Visual backbones have been playing a key role in a wide array of visual tasks, serving as the foundational encoders that extract general-purpose features from visual data ( He et al. 2016 ; Xie et al. 2017 ; Howard et al. 2017 ) . As vision models scale up, powerful backbones with massive parameters and extensive pretraining have emerged across vision tasks ( Radford et al. 2021 ; Touvron et al. 2021 ; Bao et al. 2021 ; He et al. 2022 ) . Transformers, in particular, have surpassed traditional convolutional neural networks (CNNs) in representational capacity and flexibility, excelling in large-scale scenarios through mechanisms like dense self-attention ( Vaswani et al. 2017 ) . Models such as V-MoE ( Riquelme et al. 2021 ) , Swin Transformer v2 ( Liu et al. 2022a ) , and EVA ( Fang et al. 2023 ) highlight this shift, achieving state-of-the-art results on benchmarks like ImageNet-1K ( Russakovsky et al. 2015 ) . Figure 1: Relative accuracy drop under domain shift vs model capacity. We evaluate various vision architectures on OOD test set ImageNet-R, measuring the relative performance degradation compared to ImageNet-1K validation accuracy. Different markers represent different architectural families (triangles for Transformers, circles for CNNs, etc.). Our CoCAViT demonstrates superior robustness across model scales, particularly excelling in smaller parameter regimes where existing state-of-the-art methods suffer disproportionate performance drops. However, the quadratic complexity of self-attention in Transformers poses severe challenges for real-time deployment on resource-constrained devices. This has spurred the development of efficient visual backbones, such as MobileViT ( Mehta and Rastegari 2021 ) , LeViT ( Graham et al. 2021 ) , and EfficientViT ( Liu et al. 2023 ) , which prioritize reduced complexity and inference speed. While these compact designs enable practical applications, they often compromise on representational depth, leading to vulnerabilities in handling complex feature distributions and distribution shifts. A pressing question thus arises: How can we design a compact visual backbone that is both sufficiently expressive for general-purpose representation while ensuring efficiency for real-time use? We observe that efficient models, despite competitive in-domain performance on benchmarks like ImageNet-1K, exhibit pronounced fragility under distribution shifts. To quantify this, we evaluate various architectures pretrained on ImageNet-1K, measuring relative accuracy drops on out-of-distribution test sets. As illustrated in Figure 1, compact models suffer steeper performance degradation—often over 45% relative to in-distribution (ID) accuracy—compared to larger counterparts. This vulnerability arises from inherent architectural trade-offs: (1) aggressive scaling, such as diminished head dimensions and shallower depths, reduces attention diversity and hierarchical abstraction, fostering overfitting to training distributions; (2) reliance on local attention (e.g., window-based multi-head self attention in Swin Transformer ( Liu et al. 2021 ) or sparse variants in EfficientViT) limits global receptive fields, with partial remedies like window shifting introducing inefficiencies and incomplete context integration. These limitations amplify sensitivity to shifts, hindering reliable deployment in dynamic real-world scenarios. This robustness-efficiency gap highlights the urgent need for architectural innovations that can maintain strong generalization capabilities while preserving computational efficiency. In this work, we address these issues by exploring real-time visual backbones optimized for robust global coordination capabilities. • We begin by identifying key architectural bottlenecks in efficient transformers through empirical analysis and propose principled design choices that enhance robustness while preserving efficiency. Specifically, we adopt a hybrid architecture with wider CNN stages for robust low-level feature extraction and deeper transformer stages with optimized head dimensions. We also introduce progressive MLP expansion ratios that allocate computational capacity more effectively across network depth. • Our core innovation is the Coordinated Cross-Attention (CoCA) mechanism, which addresses the fundamental limitation of window-based attention—the lack of global context. CoCA introduces a sparse set of learnable coordinators that serve as dynamic global anchors, establishing bidirectional information flow through Gated Global Cross-Attention (GGCA) and Global-Coordinated Window Attention (GCWA). This design enables robust global coordination with linear complexity, effectively bridging window silos while maintaining computational efficiency. • We introduce CoCAViT, a compact vision transformer with robust global coordination capabilities. In addition to the aforementioned key innovations, the model incorporates a Global Semantic Token Generator that produces domain-robust coordinators through parallel channel and spatial attention modules, enhanced by anchor loss regularization to ensure semantic consistency and diversity across coordinators. To enable hierarchical global coordination without prohibitive computational costs, we develop an attention-based Token Merging mechanism that compresses coordinators across stages while preserving their most semantically relevant information through cross-attention selection. Related Work Efficient Vision Transformers. With the deepening research in Vision Transformers ( Dosovitskiy et al. 2020 ) , efficient variants like MobileViT ( Mehta and Rastegari 2021 ) and EfficientViT ( Liu et al. 2023 ) , incorporate optimizations such as sparse attention, hierarchical structures, and reduced token counts, achieving comparable or even superior performance to larger models on in-domain tasks while drastically lowering latency. Currently, hybrid architectures that combine CNN and ViT elements are recognized as an effective way to balance the strengths of both paradigms. These models result in superior performance in efficient settings ( Li et al. 2023 ; Graham et al. 2021 ) . Our work follows this hybrid architecture style, mitigating the lack of inductive biases in pure Transformers (e.g., locality and shift-invariance) while leveraging the large receptive fields of self-attention for enhanced global modeling. Local Attention Mechanisms. Local attention mechanisms have emerged as a key strategy to mitigate the quadratic complexity of vanilla self-attention in Vision Transformers, particularly in compact models. Swin Transformer ( Liu et al. 2021 ) exemplifies this through its shifted window attention, which confines computations to non-overlapping windows and uses cyclic shifts to enable cross-window communication, achieving efficient hierarchical modeling. Similar approaches, such as BiFormer ( Zhu et al. 2023 ) with bi-level routing attention, dynamically sparsify attention maps to focus on relevant tokens, reducing overhead while approximating global context. However, these methods can still suffer from limited receptive fields and sensitivity to domain shifts due to their reliance on predefined or locally constrained sparsity patterns. Our CoCAViT draws inspiration from these

Read the complete text