EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation
Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru, Xuanlong Yu, Xi ShenIntellindust AI Lab · 2026
Read EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation by Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru, Xuanlong Yu, Xi ShenIntellindust AI Lab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18739.txt.
Introduction
Liu and co-authors introduce EdgeCrafter, a compact vision transformer family for dense prediction on edge devices. Small ViTs often trail CNN models like YOLO under tight compute and memory limits. The team blames weak task-specific feature learning, not the transformer design itself. Their ECDet detector uses a distilled backbone with an edge-friendly encoder-decoder. The same features feed light heads for instance segmentation and pose estimation. The small detector reaches 51.7 box AP with under ten million parameters. Larger variants top 59.9 box AP after extra detection pre-training.
Catalog record
| CREATOR | Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru, Xuanlong Yu, Xi ShenIntellindust AI Lab |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56510 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18739
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation Longfei Liu, Yongjie Hou, Yang Li, Qirui Wang, Youyang Sha, Yongjun Yu, Yinzhi Wang, Peizhe Ru, Xuanlong Yu, Xi ShenIntellindust AI Lab Abstract Deploying high performance dense prediction models on resource-constrained edge devices remains challenging due to strict limits on computation and memory. In practice, lightweight systems for object detection, instance segmentation, and pose estimation are still dominated by CNN-based architectures such as YOLO, while compact Vision Transformers (ViTs) often struggle to achieve similarly strong accuracy–efficiency trade-offs, even with large scale pretraining. We argue that this gap is largely due to insufficient task-specific representation learning in small-scale ViTs, rather than an inherent mismatch between ViTs and edge dense prediction. To address this issue, we introduce EdgeCrafter, a unified compact ViT framework for edge dense prediction centered on ECDet, a detection model built from a distilled compact backbone and an edge-friendly encoder–decoder design. We first adapt a large DINOv3 pretrained ViT to object detection and use it as a task-specialized teacher to distill rich representations into compact student backbones. We further improve efficiency by replacing standard patch embedding with a lightweight convolutional stem and constructing multi-scale features with simple interpolation and linear projection instead of costly feature pyramids. The resulting detection-distilled representation transfers directly to instance segmentation and human pose estimation through lightweight task-specific prediction modules. On the COCO dataset, ECDet-S achieves 51.7 AP with fewer than 10M parameters using only COCO annotations. For instance segmentation, ECInsSeg achieves performance comparable to RF-DETR while using substantially fewer parameters. For pose estimation, ECPose-X reaches 74.8 AP, significantly outperforming YOLO26Pose-X (71.6 AP). These results show that compact ViTs, when paired with task-specialized distillation and edge-aware design, can be a practical and competitive option for edge dense prediction. Code is available at: https://intellindust-ai-lab.github.io/projects/EdgeCrafter/ . 1 Introduction (a) Object Detection (b) Human Pose Estimation (c) Instance Segmentation (d) (a) Compared with SOTA across various vision tasks on COCO (e) (b) Effect of pretraining and distillation strategies for the backbone Figure 1 : Comprehensive evaluation of EdgeCrafter. (a) Comparison with state-of-the-art methods across multiple vision tasks on COCO ( Lin et al. 2014 ) . The plots show model parameters (top row) and FLOPs (bottom row) versus mAP. Methods marked with ∗ are pre-trained on the Objects365 dataset ( Shao et al. 2019 ) . From left to right, the columns correspond to object detection, human pose estimation, and instance segmentation. (b) Analysis of different pretraining strategies for the backbone, based on the ECDet-T model. ViT-T (Tiny) follows the training strategy proposed in ( Steiner et al. 2022 ) on ImageNet-21K ( Ridnik et al. 2021 ) . In our experiments, supervised ImageNet-21K pretraining is weaker than no pretraining for this compact model, consistent with observations reported by Ghiasi et al. 2021 ; Zoph et al. 2020 . Task-specialized distillation yields substantially stronger downstream performance. Object detection ( Girshick 2015 ) , instance segmentation ( He et al. 2017 ) , and human pose estimation ( Cao et al. 2019 ) are core dense prediction tasks in computer vision, with applications ranging from autonomous driving ( Arnold et al. 2019 ) and robotics ( Labbé et al. 2020 ) to augmented reality ( Chan et al. 2019 ) and video analytics ( Caetano et al. 2019 ) . Bringing these models to edge devices is attractive for reasons such as responsiveness and privacy, but it remains difficult because edge platforms operate under tight resource budgets. Beyond computation, edge deployment is also constrained by limited memory budgets, which makes parameter-efficient models particularly important in practice. These constraints are especially pronounced in battery-powered or embedded settings, where both model size and compute cost directly affect deployability. In real-time edge deployment scenarios, dense prediction tasks are still overwhelmingly dominated by models built upon convolutional neural network (CNN) backbones. The YOLO family ( Redmon et al. 2016 ; Redmon & Farhadi 2017 ; Glenn. 2023 ; Glenn. 2024 ; Li et al. 2022a ; Wang et al. 2024b ; Wang et al. 2024a ) and DETR-based models ( Lv et al. 2024 ; Peng et al. 2025 ) are representative examples, offering strong accuracy–efficiency trade-offs. While pioneering attempts have explored adopting Vision Transformers (ViTs) as backbones for dense prediction ( Chen et al. 2024 ; Li et al. 2022c ) , these approaches typically rely on scaling to large model sizes (e.g., ViT-Base or Large) or require additional massive pretraining datasets such as Objects365 ( Shao et al. 2019 ) to achieve competitive performance. Recently, large-scale self-supervised pretraining paradigms ( Oquab et al. 2023 ; Siméoni et al. 2025 ) have unlocked the extraordinary representation capabilities of ViTs. Recent works such as RF-DETR ( Robinson et al. 2026 ) directly employ DINOv2 ( Oquab et al. 2023 ) as a backbone. However, even the smallest variant contains approximately 30M parameters, imposing prohibitive computational and memory constraints for strict edge deployment. Unfortunately, naively shrinking a ViT to meet stringent edge constraints results in a precipitous drop in performance. Our empirical observations show that even initializing compact ViTs with supervised IN-21K ( Ridnik et al. 2021 ; Steiner et al. 2022 ) pretraining 1 1 1 https://huggingface.co/timm/vit_tiny_patch16_224.augreg_in21k yields unsatisfactory performance for dense prediction, sometimes even performing worse than training from scratch (Figure 1(e) ). This observation, which is consistent with findings reported by Ghiasi et al. 2021 ; Zoph et al. 2020 , suggests that generic supervised pretraining alone is often insufficient for compact ViTs in edge dense prediction. In this work, we introduce EdgeCrafter , a unified compact ViT framework for edge dense prediction centered on ECDet. Our approach is built upon two key ideas. First, we employ task-specialized knowledge distillation: a large DINOv3-pretrained ViT ( Siméoni et al. 2025 ) is first adapted for object detection and then used as a teacher to supervise compact student models. Second, we design an edge-friendly student architecture that replaces the standard patch embedding with a lightweight convolutional stem ( Xiao et al. 2021 ) and constructs multi-scale features using simple interpolation and linear projections. The resulting distilled backbone and encoder are used directly in ECDet and can be transferred to ECInsSeg and ECPose through lightweight task-specific heads, resulting in a unified framework for object detection, instance segmentation, and human pose estimation. Figure 1 summarizes the two main messages of this work. First, Figure 1(d) shows that EdgeCrafter achieves strong accuracy-efficiency trade-offs across object detection, instance segmentation, and human pose estimation, measured by parameters and FLOPs, while reusing a common detection-distilled representation. Several competitive baselines in these comparisons also rely on additional Objects365 pretraining, yet EdgeCrafter remains competitive and in some cases outperforms them using only task-specific COCO annotations. Second, Figure 1(e) shows that the representation learning strategy matters substantially: standard generic pretraining is not enough for compact ViTs, whereas task-specialized distillation leads to much stronger downstream performance. Together, these results support the view that compact ViTs can be competitive for edge dense prediction when their representations and architecture are designed for the target setting. In summary, this work makes the following contributions: • We identify insufficient task-specific representation learning as a key bottleneck for compact ViTs in edge dense prediction, and show that generic