AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution
Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michaël Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li · 2026
Read AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution by Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michaël Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15622.txt.
Introduction
Zhao and 11 colleagues present AdaDINO, a context-adaptive framework that makes on-device vision foundation models cheaper by matching execution to scene and task. A distilled DINOv2 backbone offers a family of subnets, a cloud language model refines candidate classes at low frequency, and a learned selector picks the cheapest subnet predicted to hold target accuracy. Learned selection alone cuts compute 37 percent at equal segmentation accuracy.
Catalog record
| CREATOR | Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michaël Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47378 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaVFM: Adaptive Vision Foundation Models for Edge Intelligence via LLM-Guided Execution Yiwei Zhao, Yi Zheng, Huapeng Su, Jieyu Lin, Stefano Ambrogio, Cijo Jose, Michaël Ramamonjisoa, Patrick Labatut, Barbara De Salvo, Chiao Liu, Phillip B. Gibbons, Ziyun Li Abstract Language-aligned vision foundation models (VFMs) enable versatile visual understanding for always-on contextual AI, but their deployment on edge devices is hindered by strict latency and power constraints. We present AdaVFM, an adaptive framework for efficient on-device inference of language-aligned VFMs that dynamically adjusts computation based on scene context and task complexity. Our key insight is that the effect of model size reduction on performance is task-dependent in vision applications, motivating a runtime-adaptive execution strategy. AdaVFM integrates neural architecture search (NAS) into the language-aligned VFM backbone to enable lightweight subnet execution during runtime. A multimodal large language model (LLM) deployed on the cloud enables runtime control with a context-aware agent. This synergy allows efficient model adaptation under diverse conditions while maintaining strong accuracy. Extensive experiments on zero-shot classification and open-vocabulary segmentation demonstrate that AdaVFM achieves state-of-the-art accuracy-efficiency trade-offs, surpassing prior baselines by up to 7.9 % 7.9\% in acc@1 on IN1K and 5.2 % 5.2\% mIoU on ADE20K over the best models of comparable VFM sizes. For models with similar accuracy, AdaVFM further reduces average FLOPs by up to 77.9 % 77.9\% . 1 Introduction Recent advances in smartphones [ 20 , 46 ] , wearable devices [ 34 , 1 ] , augmented and virtual reality [ 2 , 8 ] , robotics [ 22 , 64 ] , and autonomous driving [ 39 , 67 ] have accelerated the push toward enabling AI on edge devices. Multimodal AI models power a variety of vision and language applications, making efficient on-device deployment essential for real-time interaction and high-quality user experiences. Figure 1 : Left (a) : Always-on smart glasses with on-device VFM. Right (b) : End-to-end mIoU on open-vocabulary ADE20K segmentation [ 77 ] . Our design significantly improves mIoU by up to 5.2\% , and reduces FLOPs by up to 77.9\% over prior models. These trends have led to growing interest in always-on contextual AI —continuous, multimodal, context-aware systems that run persistently under tight energy and latency constraints [ 33 , 36 , 47 , 6 , 24 ] . As illustrated in Fig. 1 , smart-glass scenarios exemplify a collaborative edge–cloud paradigm: the edge device executes a lightweight vision foundation model for real-time perception, while tasks requiring heavy computation or broad world knowledge are offloaded to a cloud-based multimodal large language model (MM-LLM). Deploying vision foundation models (VFMs) on edge devices, however, remains challenging, and enabling language-aligned VFMs is even more difficult. These models are large, computation-intensive, and designed for high-capacity vision tasks, while edge devices operate under strict compute, memory, and power limits. Many interactive applications also impose stringent latency requirements; for example, contextual AI systems must respond within sub-second timescales (typically 200–1000 ms) to maintain perceived interactivity and usability [ 51 , 59 , 2 , 31 ] . Directly executing language-aligned VFMs under such constraints is infeasible, and prior attempts that simply shrink model capacity [ 57 , 63 ] often suffer significant performance drops (see § 2.3 ). Thus, continuously invoking VFMs at edge remains prohibitively expensive despite their strong utility. Figure 2 : Open-vocabulary segmentation on ADE [ 77 ] using 50M and 300M-parameter VFMs (300M: DINO.txt [ 28 ] ; 50M: similarly distilled and fine-tuned). Results are shown for both the original 150-class and the grouped 9-class setting (grouping based on WordNet [ 17 ] , described in § 6.2 ). Left : Both models perform similarly on the simpler 9-class task, but in the complex 150-class setting, VFM-300M yields clear object boundaries while VFM-50M gives blurred ones. Right : Simpler tasks incur small accuracy drop (0.95) with smaller models, whereas complex tasks show a larger drop (3.84). We observe that not all edge tasks require invoking a full language-aligned VFM. Simpler tasks can be handled effectively by smaller models, whereas only more complex tasks necessitate large-scale backbones. As shown in Table 1 , DINOv2 [ 49 ] —a representative vision foundation model—exhibits minimal performance degradation on simpler datasets such as Pets and Cal101 when using smaller models, while more challenging datasets incur substantially larger drops. A similar pattern emerges in open-vocabulary segmentation. In Fig. 2 , two VFMs with 50M/300M parameters are evaluated on ADE20K [ 77 ] under two settings: the original 150-class segmentation task and a grouped 9-class task using coarse super-classes (§ 6.2 ). While both models achieve comparable accuracy on the simpler 9-class task, their performance diverges significantly on the full 150-class task. This motivates an adaptive model selection on the edge based on task difficulty , rather than relying on a single, fixed model configuration. a #params FLOPs IN1K [ 12 ] food101 [ 5 ] cal101 [ 16 ] pets [ 50 ] ViT-S 22M 4.6G 79.0 (-4.4) 89.1 (-5.6) 97.0 (-0.6) 95.1 (-1.6) ViT-B 86M 17.6G 82.1 (-1.3) 92.8 (-1.9) 96.1 (-1.5) 96.2 (-0.5) ViT-L 304M 61G 83.5 (+0.1) 94.3 (-0.4) 97.5 (-0.1) 96.6 (-0.1) ViT-g 1.8B 700G 83.4 94.7 97.6 96.7 Table 1 : DINOv2 [ 49 ] results. Numbers in parentheses show drops relative to ViT-g. Simple tasks (Cal101, Pets) exhibit small drops ( \leq 1.6 ) as model sizes decrease, while moderate/complex tasks (IN1K, Food101) show larger drops (up to 5.6). In this paper, we leverage the sensitivity of open-vocabulary tasks to model capacity to enable efficient deployment of language-aligned vision foundation models (VFMs) on edge devices. We introduce runtime adaptive selection , which dynamically chooses the most suitable execution scheme for language-aligned VFMs based on task requirements and resource availability. To support this, we integrate Neural Architecture Search (NAS) into the VFM architecture and develop a two-stage training pipeline for NAS-based VFM distillation and vision-text alignment, enabling practical open-vocabulary use cases on the edge. To further exploit this adaptive mechanism, we design a runtime management agent powered by a large language model (LLM). Although invoked only at low frequency, the agent effectively infers scene context, analyzes task properties, and synthesizes meta-information from user and environmental inputs. Using this information, it recommends an optimized execution scheme for the edge model, balancing accuracy and efficiency in dynamic real-world scenarios. In summary, we present AdaVFM , a unified runtime execution framework with a high-quality training pipeline featuring: ( i ) NAS-integrated adaptive VFM deployment for efficient edge execution; ( ii ) LLM-based semantic understanding for improved text embeddings; and ( iii ) LLM-guided runtime subnet selection to fully exploit VFM adaptiveness. End-to-end evaluations (Fig. 1 and Tab. 4 ) show that AdaVFM achieves a significantly better accuracy-efficiency trade-off than prior work [ 21 , 28 , 52 , 15 , 26 , 66 , 68 , 69 , 72 ] , with up to 7.9\% IN1K and 5.2\% ADE20K accuracy gains and up to 77.9\% FLOPs reduction. 2 Background and Related Work 2.1 Always-on Contextual AI on Edge Devices The emergence of wearable edge platforms is motivating always-on contextual AI that can operate continuously under strict power, thermal, and memory constraints. As illustrated in Fig. 1 , the system relies on a lightweight on-device vision