Lightweight Transformer Architectures for Edge Devices in Real-Time Applications

Hema Hariharan Samson · 2026

Read Lightweight Transformer Architectures for Edge Devices in Real-Time Applications by Hema Hariharan Samson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03290.txt.

Introduction

Hema Hariharan Samson surveys lightweight transformer architectures designed to run on edge devices. These are the phones, embedded boards, and appliances where real-time AI must live within tight power and memory budgets. The survey reviews how model compression, quantization, pruning, and knowledge distillation shrink big transformers for such hardware. It examines prominent small variants, including MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, and MobileViT.

Performance benchmarks span standard datasets for language and vision. The analysis covers industry adoption across hardware platforms, such as NVIDIA Jetson and Apple Neural Engine. It also covers deployment frameworks, such as TensorFlow Lite and ONNX Runtime. The results show modern lightweight transformers can keep roughly seventy-five to ninety-six percent of full-model accuracy. They cut model size several-fold, and latency substantially.

Catalog record

CREATORHema Hariharan Samson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lightweight Transformer Architectures for Edge Devices in Real-Time Applications Hema Hariharan Samson Abstract The deployment of transformer-based models on resource-constrained edge devices represents a critical challenge in enabling real-time artificial intelligence applications. This comprehensive survey examines lightweight transformer architectures specifically designed for edge deployment, analyzing recent advances in model compression, quantization, pruning, and knowledge distillation techniques. We systematically review prominent lightweight variants including MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, and MobileViT, providing detailed performance benchmarks on standard datasets such as GLUE, SQuAD, ImageNet-1K, and COCO. Our analysis encompasses current industry adoption patterns across major hardware platforms (NVIDIA Jetson, Qualcomm Snapdragon, Apple Neural Engine, ARM architectures), deployment frameworks (TensorFlow Lite, ONNX Runtime, PyTorch Mobile, CoreML), and optimization strategies. Experimental results demonstrate that modern lightweight transformers can achieve 75-96% of full-model accuracy while reducing model size by 4-10 × \times and inference latency by 3-9 × \times , enabling deployment on devices with as little as 2-5W power consumption. We identify sparse attention mechanisms, mixed-precision quantization (INT8/FP16), and hardware-aware neural architecture search as the most effective optimization strategies. I Introduction I-A Motivation and Context Transformer architectures have revolutionized natural language processing and computer vision, achieving state-of-the-art results across diverse tasks [ 1 , 2 ] . However, the computational demands of standard transformers—characterized by quadratic attention complexity O( n^{2} ) and hundreds of millions of parameters—present fundamental barriers to deployment on edge devices. Real-world applications in autonomous vehicles, mobile health monitoring, augmented reality, and industrial IoT require inference latencies under 30-100ms, model sizes under 100MB, and power consumption below 5-10W—constraints that standard transformers routinely violate by orders of magnitude. I-B Edge Device Constraints Resource-constrained edge devices face several interconnected limitations: Memory Constraints: Mobile devices typically provide 4-8GB RAM with 1-2GB available for model inference, while embedded systems offer 512MB-2GB total memory. Computational Limitations: Edge devices deliver 5-200 TOPS (trillion operations per second) versus 300-2000 TOPS on datacenter GPUs. Power Budget: Battery-powered devices require < 5W continuous operation; industrial edge systems operate at 10-30W compared to 250-700W for server GPUs. Latency Requirements: Real-time applications demand < 30ms per-frame inference for 30 FPS video processing, < 100ms for interactive user experiences. Bandwidth Limitations: Edge deployment eliminates cloud dependency, requiring complete on-device processing without network round-trips. I-C Specific Challenges for Transformers Deploying transformers on edge devices introduces unique technical challenges: • Attention Complexity: Self-attention scales quadratically with sequence length, consuming excessive memory and compute for long sequences. • Parameter Redundancy: Large embedding matrices and feed-forward layers contain significant redundancy suitable for compression. • Precision Sensitivity: Quantization from FP32 to INT8 can degrade accuracy by 1-5% without careful calibration. • Framework Compatibility: Different edge platforms support different operators, requiring framework-specific optimizations. • Hardware Heterogeneity: Optimal deployment strategies vary across CPU, GPU, DSP, and specialized AI accelerators. II Literature Review: Lightweight Transformer Architectures II-A Core Transformer Limitations The original Transformer architecture [ 1 ] introduces computational bottlenecks unsuitable for edge deployment. BERT-base contains 110M parameters and requires 11GB memory for training [ 3 ] . Vision Transformers (ViT) demand 300+ GFLOPS per image at 224 \times 224 resolution. The quadratic attention complexity O( n^{2} ) becomes prohibitive for sequences exceeding 512 tokens or high-resolution images. II-B Knowledge Distillation Approaches II-B 1 DistilBERT DistilBERT [ 4 ] pioneered knowledge distillation for transformers, achieving 97% of BERT-base performance with 40% fewer parameters (66M vs 110M) and 60% faster inference. The architecture retains 6 transformer layers versus BERT’s 12, using teacher-student training where the student learns to match the teacher’s output distributions. On GLUE benchmark, DistilBERT achieves 77.0 average score (vs 79.5 for BERT-base) while reducing model size to 207MB. II-B 2 TinyBERT TinyBERT [ 5 ] advances distillation through a two-stage framework: general distillation on unlabeled data followed by task-specific distillation. TinyBERT-4 (4 layers, 14.5M parameters) achieves 96.8% of BERT-base performance on GLUE while being 7.5 \times smaller and 9.4 \times faster. The architecture introduces Transformer distillation that transfers knowledge from embedding layer, hidden states, attention matrices, and prediction layer. On SQuAD v1.1, TinyBERT-6 (67M parameters) reaches F1 score of 87.5 versus 88.5 for BERT-base. II-B 3 MobileBERT MobileBERT [ 6 ] introduces inverted-bottleneck architecture enabling task-agnostic compression. With 25.3M parameters, MobileBERT achieves 77.7 GLUE score and 62ms inference on Pixel 4 phone—4 \times smaller than BERT-base with competitive accuracy. On SQuAD v1.1, MobileBERT reaches F1 score of 90.3, outperforming DistilBERT (79.8) and matching TinyBERT-6. II-C Efficient Vision Transformers II-C 1 EfficientFormer EfficientFormer [ 7 ] achieves MobileNet-level speed while maintaining transformer performance through dimension-consistent design and latency-driven optimization. EfficientFormer-L1 reaches 79.2% top-1 accuracy on ImageNet-1K with only 1.6ms inference on iPhone 12 (CoreML), matching MobileNetV2 \times 1.4 speed (1.6ms, 74.7% accuracy). EfficientFormer-L7 (83.3% accuracy, 7.0ms latency) outperforms MobileViT-XS (74.8%, 7.2ms) by 8.5% accuracy at similar speed. II-C 2 EdgeFormer EdgeFormer [ 8 ] combines convolutional and transformer strengths through global circular convolution (GCC) with position embeddings. On ImageNet-1K classification, EdgeFormer achieves 78.6% top-1 accuracy with 5.0M parameters, providing 11% parameter reduction, 13% computation savings, and 23% faster inference versus MobileViT on ARM-based Rockchip RK3288. II-C 3 MobileViT MobileViT [ 9 ] treats transformers as convolutions for global information processing. MobileViT achieves 78.4% ImageNet-1K accuracy with 6M parameters—3.2% better than MobileNetV3 and 6.2% better than DeiT at similar parameter counts. On MS-COCO object detection, MobileViT provides 5.7% accuracy improvement over MobileNetV3 with comparable parameters. II-D Efficient Attention Mechanisms II-D 1 Sparse Attention Sparse attention mechanisms reduce quadratic complexity by limiting attention scope. Local attention restricts each token to attending only nearby tokens within a fixed window (e.g., \pm 128 positions), reducing complexity to O( n\times w ) where w is window size. The Performer architecture [ 10 ] approximates full attention using kernel methods, achieving linear O( n ) complexity while maintaining accuracy within 1-2% of full attention on language modeling tasks. II-D 2 Linear Attention Linformer [ 11 ] projects key and value sequences to lower dimensions, reducing attention complexity from O( n^{2} ) to O( n\times k ) where k\ll n is projection dimension. For sequences of length n=512 with k=256 projection, Linformer achieves 2-3 \times speedup with < 1% accuracy loss on BERT tasks. II-D 3 Dynamic Token Pruning Recent work on EdgeViT++ [ 12 ] introduces adaptive token pruning that dynamically reduces token count during inference. Using attention-based gating mechanisms, the model prunes 30-50% of tokens in intermediate layers while maintaining accuracy. Combined with hybrid quantization (INT8/FP16 mixed precision), EdgeViT++ achieves 65% memory reduction and 40% latency improvement over baseline ViT models on edge hardware. III Optimization Techniques III-A Quantization Strategies Quantization reduces numerical precision of model weights and activations, dramatically decreasing memory footprint and enabling faster integer arithmetic on specialized hardware. III-A 1 INT8 Quantization INT8 quantization represents weights and activations as 8-bit integers, reducing model size by 4 \times versus FP32. Joint pruning, quantization, and distillation (JPQD) on BERT-base reaches 5.24 \times compression with 4.19 \times performance gain and minimal accuracy loss on Intel Xeon processors [ 13 ] . III-A 2 FP16 Quantization FP16 (half-precision floating point) provides larger dynamic range than INT8, reducing memory by 2 \times while maintaining higher accuracy. Modern GPUs (NVIDIA Tensor Cores, Apple Neural Engine) support native FP16 arithmetic with significant speedups. Mixed FP16/INT8 quantization balances speed and

Read the complete text