EdgeFlex-Transformer: Transformer Inference for Edge Devices
Shoaib Mohammad, Guanqun Song, Ting Zhu · 2025
Read EdgeFlex-Transformer: Transformer Inference for Edge Devices by Shoaib Mohammad, Guanqun Song, Ting Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19741.txt.
Introduction
Shoaib Mohammad, Guanqun Song, and Ting Zhu squeeze a huge vision model onto small hardware. Their start point is a ViT-Huge model with 632 million parameters. That is far past the memory and speed budgets of edge devices. They do not retrain the model. They shrink it in stages, guided by how it acts when it runs.
First, hooks on the model flag channels that matter least. Pruning then cuts the wide layers to fit a memory budget. Some parts switch to half precision. The rest is quantized to eight-bit integers with a method that watches activations. On CIFAR-10, the shrunk model used far less peak memory. It also ran over six times faster. Accuracy held up, and in fact rose a bit.
Catalog record
| CREATOR | Shoaib Mohammad, Guanqun Song, Ting Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EdgeFlex-Transformer: Transformer Inference for Edge Devices Shoaib Mohammad, Guanqun Song, Ting Zhu Abstract Deploying large-scale transformer models on edge devices presents significant challenges due to strict constraints on memory, compute, and latency. In this work, we propose a lightweight yet effective multi-stage optimization pipeline designed to compress and accelerate Vision Transformers (ViTs) for deployment in resource-constrained environments. Our methodology combines activation profiling, memory-aware pruning, selective mixed-precision execution, and activation-aware quantization (AWQ) to reduce the model’s memory footprint without requiring costly retraining or task-specific fine-tuning. Starting from a ViT-Huge backbone with 632 million parameters, we first identify low-importance channels using activation statistics collected via forward hooks, followed by structured pruning to shrink the MLP layers under a target memory budget. We further apply FP16 conversion to selected components and leverage AWQ to quantize the remaining model weights and activations to INT8 with minimal accuracy degradation. Our experiments on CIFAR-10 demonstrate that the fully optimized model achieves a 76% reduction in peak memory usage and over 6× lower latency, while retaining or even improving accuracy compared to the original FP32 baseline. This framework offers a practical path toward efficient transformer inference on edge platforms, and opens future avenues for integrating dynamic sparsity and Mixture-of-Experts (MoE) architectures to further scale performance across diverse tasks. Code is available at: https://github.com/Shoaib-git20/EdgeFlex.git . I Introduction Transformer-based architectures, particularly Vision Transformers (ViTs), have achieved state-of-the-art performance across a wide range of computer vision tasks. However, their widespread deployment on edge devices—such as mobile phones, drones, and embedded sensors—remains severely constrained by their substantial memory and compute requirements. Edge platforms often operate under tight latency budgets and memory ceilings (typically under 100 MB), which transformer models easily exceed without careful optimization. These constraints are further exacerbated by the limited energy budgets of IoT devices. In recent years, IoT has accelerated its development in the ultra-low-power domain, where such devices typically rely on ultra-low-power communication protocols such as backscatter communication [ 17 , 6 ] or resource-efficient cross-technology communication [ 16 , 31 ] to maximize operational lifespan. Thus, IoT is likely to converge with ultra-lightweight transformers for ultra-low-power edge computing. Beyond immediate battery life, the broader push for carbon-neutral computing and sustainable I/O device usage [ 35 , 1 ] further necessitates algorithmic efficiency to minimize the environmental footprint of large-scale IoT deployments. However, running computationally intensive models on such hardware without optimization may lead to rapid battery depletion and system instability [ 19 ] . Bridging this gap between model accuracy and deployment efficiency is critical for enabling real-time AI applications in bandwidth- and power-constrained environments. Moreover, the push for on-device inference is heavily driven by security and privacy concerns. Offloading raw sensor data to the cloud exposes users to potential eavesdropping and physical layer attacks, such as optical side-channel leakage [ 20 ] or wireless jamming [ 9 , 2 ] . Processing data locally mitigates these transmission risks [ 30 ] and protects sensitive user information [ 33 ] , provided the models are efficient enough to run in real-time. Prior research has explored a variety of approaches to compress and accelerate transformer models. Unstructured pruning, for instance, removes individual weights based on magnitude or learned importance, but results in irregular sparsity patterns that are not hardware-friendly on most edge accelerators [ 24 ] . Similarly, low-bit quantization techniques—especially uniform or naïve methods—often degrade performance unless specifically guided by activation-aware strategies [ 18 ] . Hardware-aware Neural Architecture Search (NAS) offers tailored solutions by co-designing model structure and execution for a given device, but this process is computationally intensive and impractical for dynamic or large-scale deployment across heterogeneous edge environments. Meanwhile, adaptive inference systems have struggled to accommodate the highly variable per-layer memory and latency profiles typical of transformer architectures, leading to inefficient scheduling [ 15 ] . Knowledge distillation provides another path by transferring performance from large teacher models to smaller student models, yet requires complex training pipelines that may not generalize across domains or tasks [ 12 ] . Together, these limitations highlight the need for a lightweight, generalizable framework that can compress and adapt transformers with minimal overhead while respecting the stringent constraints of edge deployment. To address these challenges, we propose a multi-stage optimization pipeline that incrementally compresses and accelerates transformer models using structured, hardware-conscious techniques—without requiring model retraining or architectural search. Our approach begins with activation profiling, where we collect activation statistics from calibration data via forward hooks on MLP layers, enabling precise identification of redundant channels. Next, we apply memory-aware structured pruning, which removes low-activation channels in a manner compatible with hardware execution units, significantly reducing intermediate memory pressure. We further enhance efficiency through mixed precision execution, selectively enabling FP16 computation for compatible submodules. Finally, we apply activation-aware quantization (AWQ) to quantize weights and activations to INT8, using channel-wise scaling informed by activation sensitivity to preserve accuracy. Unlike prior methods, our approach is training-free, lightweight, and modular—allowing flexible adaptation to a wide range of ViT backbones and edge platforms. Experimental results demonstrate that our pipeline yields substantial reductions in memory and latency while maintaining competitive accuracy, making it a practical solution for real-world edge deployment. II Related Works A growing body of work has sought to make transformer models feasible on edge devices through model pruning, quantization, knowledge distillation, neural architecture search, and specialized inference frameworks. Additionally, understanding the underlying sensing, communication, and security infrastructure of edge environments is essential for contextualizing these optimization efforts. II-A IoT Infrastructure and Security Context The deployment of deep learning on edge devices is closely tied to advancements in the broader IoT infrastructure. Diverse sensing modalities, ranging from passive radar [ 22 , 23 ] and acoustic monitoring [ 34 ] to multimodal magnetic tracking systems [ 27 ] , generate complex data streams that require efficient local processing. To support these applications, communication layers have been optimized for high throughput and low latency using concurrent cross-technology frameworks [ 3 , 5 , 4 ] . Furthermore, efficient system-level data management, such as lightweight virtual file system implementations [ 29 ] , is often required to support the high data ingestion rates of these sensors on resource-constrained platforms. Furthermore, ensuring the trustworthiness of these edge systems is paramount. Vulnerabilities in the physical layer [ 32 ] and the need for robust authentication [ 14 , 28 ] underscore the preference for local inference, which minimizes the attack surface associated with raw data transmission. II-B Model Pruning Pruning and quantization directly reduce model size and arithmetic precision, while distillation transfers knowledge to smaller “student” models. NAS methods tailor architectures for resource constraints, and systems like FlexNN manage memory adaptively. Despite significant gains—often reducing model size by 4–10× or enabling INT8/4-bit inference—each approach faces limitations: pruning can yield irregular sparsity that hardware struggles with, quantization risks accuracy loss without careful activation‐aware schemes, distillation demands extensive training efforts, NAS incurs high search costs, and current inference frameworks frequently fail to address transformer‐specific memory patterns. Transformer pruning removes redundant weights or channels to shrink model size and speed up inference. Recent work on automatic BERT pruning demonstrates up to 50–70% parameter reduction with minimal accuracy loss by targeting low‐importance weights based on activation statistics [ 10 ]