MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding Conference: The 24th ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems; May 11–14, 2025; Saint-Malo, France The 24th ACM/IEEE International Conference on Embedded Artificial
Runxi Huang 1 , Mingxuan Yu 1 , Mingyu Tsoi 1 , Xiaomin Ouyang 1 · 2025
Read MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding Conference: The 24th ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems; May 11–14, 2025; Saint-Malo, France The 24th ACM/IEEE International Conference on Embedded Artificial by Runxi Huang 1 , Mingxuan Yu 1 , Mingyu Tsoi 1 , Xiaomin Ouyang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25327.txt.
Introduction
Runxi Huang, Mingxuan Yu, Mingyu Tsoi, and Xiaomin Ouyang propose MMEdge. Their 2025 paper speeds multimodal inference on edge devices. Edge devices have limited computing resources ready. Uses include driving, mobile health, and human effect. Prior work often ignored sensing and execution coupling. Inter-modality dependencies also complicate real-time inference greatly.
MMEdge pipelines sensing and encoding into fine-grained units. Computation starts before full sensor inputs arrive. Data are processed incrementally as they appear. A time aggregation module preserves quality across units. It captures rich changes between pipelined stages. An changing optimizer selects sensing and model settings. It works under latency limits and changing resources. A speculative skipping step bypasses slow modality units. Early confident predictions trigger that skipping safely. Tests used two public multimodal data sets. A drone based testbed showed lower latency overall.
Catalog record
| CREATOR | Runxi Huang 1 , Mingxuan Yu 1 , Mingyu Tsoi 1 , Xiaomin Ouyang 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25327
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding Conference: The 24th ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems; May 11–14, 2025; Saint-Malo, France The 24th ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems (ACM SenSys ’26), May 11–14, 2026, Saint-Malo, France DOI: 10.1145/xxxxxxx.xxxxx ISBN: nnn-n-nnnn-nnnn-n/nn/nn CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Machine learning CCS: Computer systems organization Embedded and cyber-physical systems Runxi Huang 1 , Mingxuan Yu 1 , Mingyu Tsoi 1 , Xiaomin Ouyang 1 Abstract . Real-time multimodal inference on resource-constrained edge devices is essential for applications such as autonomous driving, human-computer interaction, and mobile health. However, prior work often overlooks the tight coupling between sensing dynamics and model execution, as well as the complex inter-modality dependencies. In this paper, we propose MMEdge, a new on-device multimodal inference framework based on pipelined sensing and encoding . Instead of waiting for complete sensor inputs, MMEdge decomposes the entire inference process into a sequence of fine-grained sensing and encoding units, allowing computation to proceed incrementally as data arrive. MMEdge also introduces a lightweight but effective temporal aggregation module that captures rich temporal dynamics across different pipelined units to maintain accuracy performance. Such pipelined design also opens up opportunities for fine-grained cross-modal optimization and early decision-making during inference. To further enhance system performance under resource variability and input data complexity, MMEdge incorporates an adaptive multimodal configuration optimizer that dynamically selects optimal sensing and model configurations for each modality under latency constraints, and a cross-modal speculative skipping mechanism that bypasses future units of slower modalities when early predictions reach sufficient confidence. We evaluate MMEdge using two public multimodal datasets and deploy it on a real-world unmanned aerial vehicle (UAV)-based multimodal testbed. The results show that MMEdge significantly reduces end-to-end latency while maintaining high task accuracy across various system and data dynamics. A video demonstration of MMEdge’s performance in real world is available at https://youtu.be/n36M9ho2z9o . 1 1 1 Code available at: https://github.com/HKUST-MINSys-Lab/MMEdge . Accepted by SenSys 2026. 1. Introduction Multimodal sensing and learning systems are increasingly adopted in real-time applications such as autonomous driving ( Xie et al. 2023 ; Wu et al. 2024 ) , fall detection ( Chen et al. 2020 ) , and wearable interaction ( Mayberry et al. 2015 ; Pang et al. 2023 ) . Such systems continuously collect, process and fuse data from heterogeneous sensors (e.g., cameras, microphones, IMUs) to provide improved performance for complex tasks in dynamic environments. For example, RCBEVDet ( Lin et al. 2024 ) shows that integrating radar with camera can significantly enhance object detection accuracy in autonomous driving. While offloading computation to the cloud has been widely adopted in distributed multimodal sensing systems, it raises growing concerns about user’s data privacy. Moreover, such systems often experience significant latency due to unpredictable network dynamics ( Li et al. 2021 ; Huang et al. 2023 ) and the high bandwidth demands of data-intensive sensors like LiDAR ( Wu et al. 2024 ) . As a result, there is an increasing demand for efficient on-device multimodal systems, which enables end-to-end processing of sensor data locally without transmitting sensor data. However, deploying end-to-end multimodal perception systems on devices presents several major challenges, particularly under limited and dynamic resource conditions. First, there is an inherent dependency between the sensing and model inference phases when they share computational resources on the same device, making isolated optimization strategies ineffective. For instance, increasing model complexity or enhancing sensing granularity can both improve prediction accuracy, but also introduce higher latency, necessitating careful coordination between the two. Second, multimodal systems also exhibit strong inter-modality dependencies, where both fusion accuracy and latency depend on the joint optimization of all modalities. For example, Shuai et al. ( Shuai et al. 2021 ) show that human detection accuracy in radar-camera systems depends on synchronized multimodal inputs, especially under varying lighting conditions, rather than relying on any single modality alone. Moreover, there will be asynchronous processing delays across modalities, which often leads to idle waiting time and inefficient resource utilization. In such cases, faster modalities are forced to wait for slower ones, increasing end-to-end latency. These challenges motivate a unified system design that dynamically coordinates the sensing and inference configurations across different modalities at run time to optimize both accuracy and efficiency. Unfortunately, most prior work on on-device multimodal systems has focused on optimizing either the sensing or the model inference stage in isolation, without considering their interdependencies. Some methods focus on mitigating delays caused by asynchronous sensor inputs. For example, ( Li et al. 2021 ) introduced modality imputation to synthesize missing data from slow modalities using fast ones to reduce waiting time, while modality gating methods ( Hou et al. 2023 ; Mayberry et al. 2015 ) selectively skip uninformative sensor modalities before inference to improve efficiency. On the other hand, several methods aim at optimizing the model inference process itself through early exit strategies ( Cao et al. 2022 ) or expert model selection ( Liu et al. 2024 ) . However, our results indicate that optimizing sensing or inference in isolation performs poorly under real-world system and data dynamics. Although some distributed systems consider sensing-inference interactions ( Li et al. 2021 ; Wu et al. 2024 ) , they are designed to mitigate the impact of network dynamics during sensor data transmission ( Rastikerdar et al. 2024 ; Huang et al. 2023 ) , which cannot address the shared computational resource constraints between sensing and inference that are intrinsic to on-device multimodal systems. In this paper, we propose MMEdge, a new and efficient on-device multimodal inference system based on pipelined sensing and encoding. Traditional end-to-end multimodal systems typically operate in a sequential manner, where model inference is blocked until all sensor data within a time window are fully acquired. Such sequential execution not only leads to accumulated and imbalanced delays across different modalities, but also incurs significant memory overhead at runtime. MMEdge addresses these limitations by decomposing the entire computation task into a sequence of fine-grained sensing and encoding units, each unit corresponding to the smallest data segment (e.g., a video frame or audio chunk). These units are processed immediately upon arrival, enabling feature encoding to occur during the sensing interval. As a result, such design eliminates idle periods between sensing intervals, and allow fully pipelined execution of data acquisition and feature encoding without waiting for complete time-window data. To mitigate potential performance degradation from this decoupled processing, MMEdge introduces a lightweight temporal aggregation module that selectively preserve dependencies and contextual continuity across units. Specifically, this module exploits alternating temporal shift operations across units and extracts multi-scale temporal difference features to capture short- and long-term temporal correlations. This design ensures that critical temporal and semantic relationships are retained, even when data is processed in a fine-grained, pipelined manner. Moreover, to adapt to runtime resource variability and input data complexity, MMEdge incorporates an adaptive multimodal configuration optimizer that dynamically selects optimal sensing and model configurations for each modality to satisfy latency constraints under varying system conditions; and a cross-modal speculative skipping mechanism that reduces waiting time by bypassing future units of slower modalities when early predictions achieve sufficient confidence. We deploy MMEdge on