EdgeFM: Efficient Edge Inference for Vision-Language Models
Mengling Deng 1‡ , Yuanpeng Chen 1 ‡ ♣ {}^{1\ddagger\text{\scalebox{0.7}{\faIconFromMacro{faEnvelopeO}}}\clubsuit} , Sheng Yang 2‡ , Wei Tao 3 , Wenhai Zhang 1 , Hui Song 1 , Linyuanhao Qin 2 , Kai Zhao 1 , Xiaojun Ye 1 , Shanhui Mo 4 , Jingli Fan 1 , Shuang Zhang 1 , Bei Liu 1 , Tiankun Zhao 1 , Xiangjing An 1 ‡ \ddagger Equal Contribution Corresponding Author ♣ \clubsuit Project Leader 1 Go Further. AI 2 School of Data Science, Fudan University 3 RUYi Dynamics Co., Ltd 4 Independent Researcher Email: [email protected] Project Page: https://github.com/windog-labs/edge-fm-x · 2026
Read EdgeFM: Efficient Edge Inference for Vision-Language Models by Mengling Deng 1‡ , Yuanpeng Chen 1 ‡ ♣ {}^{1\ddagger\text{\scalebox{0.7}{\faIconFromMacro{faEnvelopeO}}}\clubsuit} , Sheng Yang 2‡ , Wei Tao 3 , Wenhai Zhang 1 , Hui Song 1 , Linyuanhao Qin 2 , Kai Zhao 1 , Xiaojun Ye 1 , Shanhui Mo 4 , Jingli Fan 1 , Shuang Zhang 1 , Bei Liu 1 , Tiankun Zhao 1 , Xiangjing An 1 ‡ \ddagger Equal Contribution Corresponding Author ♣ \clubsuit Project Leader 1 Go Further. AI 2 School of Data Science, Fudan University 3 RUYi Dynamics Co., Ltd 4 Independent Researcher Email: [email protected] Project Page: https://github.com/windog-labs/edge-fm-x (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27476.txt.
Introduction
Deng and fourteen colleagues present EdgeFM, an agent-driven inference framework for vision-language models for edge use. Existing frameworks bloat general designs or lock developers into closed ecosystems. EdgeFM strips non-essential features to cut request latency, and packages agent-tuned kernel optimizations as a library of reusable skills that can be invoked directly. It supports x86 and Orin platforms, claims the first full VLA rollout on Horizon Journey, and in most cases beats vendor toolchains, with speedups up to 1.49 times over TensorRT-Edge-LLM on Orin.
Catalog record
| CREATOR | Mengling Deng 1‡ , Yuanpeng Chen 1 ‡ ♣ {}^{1\ddagger\text{\scalebox{0.7}{\faIconFromMacro{faEnvelopeO}}}\clubsuit} , Sheng Yang 2‡ , Wei Tao 3 , Wenhai Zhang 1 , Hui Song 1 , Linyuanhao Qin 2 , Kai Zhao 1 , Xiaojun Ye 1 , Shanhui Mo 4 , Jingli Fan 1 , Shuang Zhang 1 , Bei Liu 1 , Tiankun Zhao 1 , Xiangjing An 1 ‡ \ddagger Equal Contribution Corresponding Author ♣ \clubsuit Project Leader 1 Go Further. AI 2 School of Data Science, Fudan University 3 RUYi Dynamics Co., Ltd 4 Independent Researcher Email: [email protected] Project Page: https://github.com/windog-labs/edge-fm-x |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42816 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EdgeFM: Efficient Edge Inference for Vision-Language Models Mengling Deng 1‡ , Yuanpeng Chen 1 ‡ ♣ {}^{1\ddagger\text{\scalebox{0.7}{\faIconFromMacro{faEnvelopeO}}}\clubsuit} , Sheng Yang 2‡ , Wei Tao 3 , Wenhai Zhang 1 , Hui Song 1 , Linyuanhao Qin 2 , Kai Zhao 1 , Xiaojun Ye 1 , Shanhui Mo 4 , Jingli Fan 1 , Shuang Zhang 1 , Bei Liu 1 , Tiankun Zhao 1 , Xiangjing An 1 ‡ \ddagger Equal Contribution Corresponding Author ♣ \clubsuit Project Leader 1 Go Further. AI 2 School of Data Science, Fudan University 3 RUYi Dynamics Co., Ltd 4 Independent Researcher Email: [email protected] Project Page: https://github.com/windog-labs/edge-fm-x Abstract Vision-language models (VLMs) have demonstrated strong applicability in edge industrial applications, yet their deployment remains severely constrained by requirements for deterministic low latency and stable execution under resource limitations. Existing frameworks either rely on bloated general-purpose designs or force developers into opaque, hardware-specific closed-source ecosystems, leading to hardware lock-in limitation and poor cross-platform adaptability. Observing that modern AI agents can efficiently search and tune configurations to generate highly optimized low-level kernels for standard LLM operators, we propose EdgeFM , a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment. EdgeFM removes non-essential features to reduce single-request latency, and encapsulates agent-tuned kernel optimizations as a modular library of reusable skills . By allowing direct invocation of these skills rather than waiting for closed-source implementations, it effectively closes the performance gap long dominated by proprietary toolchains. The framework natively supports mainstream platforms including x86 and NVIDIA Orin SoCs, and represents the first end-to-end VLA deployment on the domestic Horizon Journey platform , enhancing cross-platform portability. In most cases, it yields clearly better inference performance than conventional vendor-specific toolchains, achieving up to 1.49 × 1.49\times speedup over TensorRT-Edge-LLM on the NVIDIA Orin platform. Experimental results show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios. \reportnumber 001 Figure 1 : Overview of cross-platform performance. (a, b) On the x86 platform: (a) End-to-end latency decomposition; (b) Latency gap under diverse prefill/decode shapes, where red/green indicates EdgeFM/TRT-Edge-LLM advantage; (c) On the Horizon BPU platform: Deployment performance demonstrating deterministic prefill and ultra-low action decode latencies. 1 Introduction Large language models ( 2 ; 7 ; 32 ) (LLMs), built on the attention mechanism ( 28 ) , have made unprecedented advances in natural language understanding, generation, and reasoning in recent years. However, inference efficiency has become a core bottleneck restricting large-scale deployment across cloud, local, and edge environments. While extensive studies have optimized LLM inference for diverse scenarios, including cloud high-throughput serving and general-purpose local deployment, there remains a critical gap in meeting the stringent requirements of production-grade edge deployment for industrial applications. Industrial edge scenarios, including autonomous driving and embodied intelligence, impose strict design constraints. Unlike cloud services prioritizing high concurrency, edge workloads are typically single-request and latency-critical, demanding deterministic millisecond-level responsiveness under severely constrained on-board resources. General-purpose local inference frameworks, with their bloated codebases and redundant compatibility layers, struggle to meet these demands. Consequently, industrial edge deployments are frequently forced to rely heavily on proprietary, hardware-vendor-specific closed-source toolchains. While these opaque ecosystems provide baseline performance, they create severe hardware lock-in, act as black boxes that stifle agile algorithm iteration, and force developers into a passive wait for vendor-provided updates whenever novel model architectures emerge. Fortunately, the remarkable proficiency of modern LLM-based coding agents offers a transformative paradigm shift. Given that the underlying operator semantics of LLM inference are relatively standardized and computationally well-defined, these autonomous agents can assist the optimization workflow of high-performance operators by proposing candidate implementations, tuning kernel parameters, and generating hardware- or shape-specialized artifacts. Crucially, these agent-assisted kernel optimizations are encapsulated into a modular repository of reusable skills , while the runtime selects among them through a lightweight operator dispatch table. By directly invoking these skills, developers can attain high-performance inference across diverse hardware platforms without passively waiting for vendor-specific closed-source implementations. This agentic paradigm effectively bridges the performance gap long monopolized by heavily engineered proprietary toolchains, empowering the community to build independent, optimized deployment pipelines free from ecosystem lock-in. Motivated by this paradigm shift and to eliminate rigid ecosystem dependencies, this paper presents EdgeFM (Edge Foundation Model), a lightweight LLM inference framework natively built for industrial edge platforms. Its core design principles and key contributions are summarized as follows: • Edge-native design. Tailored for single-request edge workloads, EdgeFM enables lightweight deployment via a streamlined API and concise codebase, with a core focus on minimizing end-to-end inference latency. • Modular layered architecture. EdgeFM uses a layer/operator-decoupled design with a model-, hardware-, and stage-aware operator table. Its clean stack of interfaces, lightweight runtime, and unified abstractions dispatches optimal kernels at runtime, ensuring low complexity and independent component evolution. • Edge-optimized core acceleration. EdgeFM integrates mainstream low-latency acceleration techniques. We benchmark against TensorRT-Edge-LLM, NVIDIA’s official and industry-standard framework for high-performance LLM/VLM deployment, and show that our design achieves favorable and consistent latency gains over this strong baseline. • Hardware extensibility. EdgeFM is an open-source, lightweight, lock-in-free framework that supports x86, NVIDIA Orin, and the Horizon Journey 6 series with dedicated adaptations. Moreover, it provides open-source skills with reusable agent-tuned kernels for low-cost extension to diverse edge chips and industrial hardware. 2 Related Works 2.1 Algorithm-Level Optimizations for LLM Inference Algorithm-level improvements form the foundation of efficient LLM inference, focusing on reducing the computational overhead and memory footprint of the autoregressive pipeline. These optimizations apply universally to cloud and edge scenarios. As the core module of LLMs, the attention mechanism can be optimized along two complementary lines: system execution and memory access efficiency, and computational logic and paradigm innovation. For system execution and cache efficiency optimization, a series of works have broken through the memory wall of attention computation and improved runtime execution efficiency. The Flash Attention series ( 4 ; 5 ; 24 ; 36 ) optimizes the memory access pattern of attention computation via kernel fusion and on-chip memory reuse, and has become the de facto standard for high-performance attention implementation. Ring Attention ( 20 ) extends this optimization to distributed scenarios, enabling memory-efficient long-sequence inference across multiple devices. Paged Attention, first proposed in vLLM ( 15 ) and built on the insights of 25 , introduces OS-inspired paged memory management for KV cache, which greatly improves memory utilization in concurrent inference scenarios. For computational logic and paradigm optimization, works focus on reducing the inherent computational complexity of the attention mechanism or eliminating redundant computation in practical deployment. Radix Attention, proposed in SGLang ( 39 ) , eliminates redundant prefix computation via radix tree-based KV cache reuse, accelerating the execution of structured language model programs. Sparse attention designs, represented by DeepSeek sparse attention ( 35 ; 8 ) , cut down computational overhead by skipping redundant attention token pairs based on activation sparsity. Linear attention variants ( 14 ) , which are also adopted in industrial systems such as Kimi Linear ( 38 ) , replace the quadratic complexity of standard attention with linear complexity via kernel transformation, enabling efficient inference for ultra-long sequences. Weight quantization is another core direction for LLM inference acceleration, reducing memory and computational overhead via low-bit weight compression. As the pioneering works