Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference

Seohyeon Cha 1 , Kevin Chan 2 , Gustavo de Veciana 1 , Haris Vikalo 1 · 2025

Read Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference by Seohyeon Cha 1 , Kevin Chan 2 , Gustavo de Veciana 1 , Haris Vikalo 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13380.txt.

Introduction

Seohyeon Cha, Kevin Chan, Gustavo de Veciana, and Haris Vikalo schedule AI models across phones, edge servers, and the cloud in a 2025 study. Real services run many tasks at once, such as detection, segmentation, and depth estimation for driving or augmented reality, while older frameworks assumed one task and one model. Loading a model onto a tier costs time, and batching requests changes the math again.

Their framework decides jointly which models to load or unload at each level, with batching built into the accounting. Smart placement keeps weak devices responsive without wasting cloud capacity.

Catalog record

CREATORSeohyeon Cha 1 , Kevin Chan 2 , Gustavo de Veciana 1 , Haris Vikalo 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13380

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference Seohyeon Cha 1 , Kevin Chan 2 , Gustavo de Veciana 1 , Haris Vikalo 1 Abstract The growing demand for intelligent services on resource-constrained edge devices has spurred the development of collaborative inference systems that distribute workloads across end devices, edge servers, and the cloud. While most existing frameworks focus on single-task, single-model scenarios, many real-world applications (e.g., autonomous driving and augmented reality) require concurrent execution of diverse tasks including detection, segmentation, and depth estimation. In this work, we propose a unified framework to jointly decide which multi‑task models to deploy (“onload”) at clients and edge servers, and how to route queries across the hierarchy (“offload”) to maximize overall inference accuracy under memory, compute, and communication constraints. We formulate this as a mixed‑integer program and introduce J3O (Joint Optimization of Onloading and Offloading), an alternating algorithm that (i) greedily selects models to onload via Lagrangian-relaxed submodular optimization and (ii) determines optimal offloading via constrained linear programming. We further extend J3O to account for batching at the edge, maintaining scalability under heterogeneous task loads. Experiments show J3O consistently achieves over 97% of the optimal accuracy while incurring less than 15% of the runtime required by the optimal solver across multi-task benchmarks. I Introduction The rapid proliferation of edge devices including smartphones, surveillance cameras, and wearables, with possible latency and privacy requirements, has sparked interest in executing Machine Learning (ML)-based inference at the edge [ 1 ] . However, as state-of-the-art ML models continue to grow in size and complexity to achieve higher accuracy, their memory and compute requirements often exceed the capabilities of resource-constrained edge hardware [ 2 , 3 ] . While model compression techniques and lightweight alternatives can help reduce resource usage, they often incur an accuracy drop, particularly in multi-task settings where a compact model must simultaneously support diverse inference tasks [ 4 ] . Collaborative inference systems offer a promising alternative by distributing inference workloads across hierarchical computing tiers – end devices, edge servers, and cloud platforms [ 5 , 6 , 7 ] . These systems typically use lightweight models on device to handle routine or latency-sensitive queries, while selectively offloading more demanding instances to upstream servers with larger models. However, most prior frameworks have focused on single-task, single-model settings (e.g., image classification), which limits their applicability. In contrast, real-world applications such as autonomous driving, augmented reality, and smart surveillance require concurrent execution of multiple tasks (e.g., detection, segmentation, and depth estimation) within the same inference pipeline [ 8 , 9 , 10 ] . Extending collaborative inference to multi-task scenarios presents new challenges. A single model trained on all tasks may suffer from degraded performance due to task interference and conflicting gradients [ 11 ] . On the other hand, maintaining a large pool of single-task models can quickly overwhelm the memory and compute capacity of edge devices. A more scalable strategy is to maintain a small library of specialized multi-task models, each trained on a subset of related tasks, and dynamically select the best model for each inference request [ 11 , 12 , 13 ] . While this approach has been explored in centralized or cloud-based settings, model selection and deployment under tight system constraints in hierarchical, distributed environments remains largely unaddressed. The joint model placement and inference offloading has previously been studied primarily in single-task settings, and has considered either a single model or a collection of models. Early efforts in edge intelligence [ 14 , 15 , 16 , 17 ] explored collaborative execution of a single Deep Neural Network (DNN) across edge and cloud, using techniques such as DNN partitioning and early exiting to reduce end-to-end latency by balancing communication and computation. More recent work has extended this to multi-model scenarios, where systems must select, cache, and place multiple candidate models across heterogeneous edge resources [ 18 , 19 , 20 , 21 , 22 ] . These approaches aim to optimize inference accuracy, latency, or energy under resource constraints, typically by coordinating model selection and query routing between edge and cloud. However, they remain largely restricted to single-task inference and shallow hierarchies, without jointly optimizing model placement and task routing across clients, edge and cloud. Table I summarizes the distinctions between our approach and these prior methods. (a) Accuracy (b) Edge Onloading Decision (c) Client Onloading Decision (d) Batch Size Fig. 1 : We compare (top) homogeneous and (bottom) heterogeneous task load distributions across clients. As task A becomes dominant, the optimizer adapts model placement and offloading strategies to maximize batching efficiency and system accuracy. TABLE I : Comparison of related works by objective, model granularity, task scope, and hierarchy of models for inference. Objective Model Task Hierarchy [ 18 ] Accuracy & Latency Multi Single Single-Tier [ 21 ] Latency & Energy Multi Single Single-Tier [ 22 ] Latency & Energy Multi Single Two-Tier [ 19 , 20 ] Accuracy Multi Single Two-Tier [ 23 ] Latency Multi Multi Single-Tier Ours Accuracy Multi Multi Multi-Tier A recent study [ 23 ] explores joint multi-task model deployment and task offloading in vehicular edge computing, where vehicles send inference queries to roadside units (RSUs) equipped with shared multi-task models. However, their “one-model-fits-all” approach limits task specialization and may underperform compared to frameworks that rely on tailored models. The framework also lacks hierarchical offloading beyond RSUs, leaving edge device and cloud resources unused. The broader problem of jointly optimizing multi-task model placement and hierarchical task routing under system constraints remains largely unexplored. To address these challenges, we propose a unified framework for joint model onloading and hierarchical offloading in distributed multi-task inference systems. Each client hosts a tailored subset of multi-task models and selectively offloads queries to higher-tier edge servers. In turn, edge servers may forward difficult requests to the cloud, which stores full-capacity models. The framework jointly determines (i) which models to onload at the client and edge levels and (ii) how to route queries across system tiers to maximize overall accuracy under memory, compute, and communication constraints. To efficiently solve this combinatorial problem, we introduce the J oint O ptimization of O nloading and O ffloading ( J3O ) algorithm, an alternating method that combines greedy submodular maximization for model onloading with constrained linear programming for task offloading. We further extend our formulation to incorporate batching, allowing edge servers to aggregate queries into homogeneous batches to improve GPU utilization. This design aligns with modern accelerator architectures, where batching amortizes fixed inference overhead. In order to bound latency, we impose a time-window constraint on batch formation. To solve the extended problem, we develop BAJ3O ( B atching- A ware J oint O ptimization of O nloading, O ffloading), which augments J3O with a batching-related latency constraint and integrates it seamlessly into the alternating optimization loop. The main contributions of this work are: • We formulate a new joint model onloading and offloading problem for hierarchical multi-task inference, aiming to maximize system-wide accuracy under memory, compute, and communication constraints. This formulation captures task-aware model specialization, multi-tier coordination, and shared resource budgets. • We propose an alternating optimization algorithm with provable performance guarantees, J3O ( J oint O ptimization of O

Read the complete text