Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson - Extended
Abhinaba Chakraborty 1 Wouter Tavernier 1 Akis Kourtis 2 Mario Pickavet 1 Andreas Oikonomakis 2 Didier Colle 1 · 2025
Read Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson - Extended by Abhinaba Chakraborty 1 Wouter Tavernier 1 Akis Kourtis 2 Mario Pickavet 1 Andreas Oikonomakis 2 Didier Colle 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08430.txt.
Introduction
Chakraborty and co-authors profile in 2025 how small edge AI boards handle many vision tasks at once. Edge devices must process camera data fast, near the source, to cut delay and bandwidth strain. Graphics units on these boards often sit underused, even on heavy work.
The team merged traces from several profiling tools on NVIDIA Jetson devices. They tracked unit use, memory, core use, and tensor core use under shared vision loads. Top level use could hit full marks with tuning, yet core resources often ran at only 15 to 30 percent. CPU side events, such as thread scheduling, also emerged as frequent bottlenecks.
Catalog record
| CREATOR | Abhinaba Chakraborty 1 Wouter Tavernier 1 Akis Kourtis 2 Mario Pickavet 1 Andreas Oikonomakis 2 Didier Colle 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08430
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson - Extended Abhinaba Chakraborty 1 Wouter Tavernier 1 Akis Kourtis 2 Mario Pickavet 1 Andreas Oikonomakis 2 Didier Colle 1 Abstract The proliferation of IoT devices and advancements in network technologies have intensified the demand for real-time data processing at the network edge. To address these demands, low-power AI accelerators, particularly GPUs, are increasingly deployed for inference tasks, enabling efficient computation while mitigating the latency and bandwidth limitations of cloud-based systems. Despite their growing deployment, GPUs remain underutilised even in computationally intensive workloads. This underutilisation stems from the limited understanding of GPU resource sharing, particularly in edge computing scenarios. In this work, we conduct a detailed analysis of both high- and low-level metrics, including GPU utilisation, memory usage, streaming multiprocessor (SM) utilisation, and tensor core usage, to identify bottlenecks and guide hardware-aware optimisations. By integrating traces from multiple profiling tools, we provide a comprehensive view of resource behaviour on NVIDIA Jetson edge devices under concurrent vision inference workloads. Our findings indicate that while GPU utilisation can reach 100 % 100\% with specific optimisations, critical low-level resources, such as SMs and tensor cores, often operate at only 15 % 15\% to 30 % 30\% utilisation. Moreover, we observe that certain CPU-side events, such as thread scheduling and context switching, frequently become bottlenecks, further constraining overall GPU performance. We provide several key observations for users of vision inference workloads on NVIDIA edge devices. I Introduction The expanding IoT ecosystem generates a vast amount of data requiring real-time processing for applications like autonomous vehicles [ 1 , 2 ] and smart cities [ 3 , 4 ] . Traditional cloud services [ 5 , 6 , 7 ] often face challenges like latency and privacy, making edge computing with GPUs [ 8 ] a compelling alternative. Despite advances in GPU architectures(such as the introduction of tensor cores [ 9 ] ), optimal resource utilisation remains elusive, particularly during complex interactions with deep learning (DL) [ 10 , 11 , 12 , 13 ] models. To optimise performance in edge computing inference systems, it is imperative to strike a balance between architectural resource utilisation and workload selection of optimised workloads [ 14 , 15 ] . Research in this domain primarily focuses on two directions: developing systems tailored to accommodate workloads on edge devices and designing high-performance, energy-efficient accelerators. For the former, a pivotal decision involves determining whether tasks should be executed locally at the edge or offloaded to the cloud [ 16 ] . For instance, in cloud environments equipped with NVIDIA A40 GPUs, a single YoloV8n [ 17 ] model can process over 1000 images per second at fp16 precision. However, network-related delays, encompassing both transmission and processing overheads, reduce the effective throughput. Rather than relying on an iterative trial-and-error method to meet quality-of-service (QoS) requirements, decisions can be guided by offline performance analysis. Tasks may be offloaded to the cloud or distributed across additional AI accelerators to achieve effective load balancing. DL compilers [ 18 ] and SDKs like TensorRT [ 19 ] optimise models for specific platforms, balancing accuracy, energy consumption, and other factors that are crucial for edge computing [ 20 ] . These optimisations reduce workload overhead, enabling concurrent model processing on the same edge GPU. However, the limited capacity of edge devices often requires manual trial and error for designing an edge system architecture. Orchestration frameworks like Kubernetes [ 21 ] and YARN [ 22 ] often prohibit GPU sharing, dedicating one GPU per DL process, which leads to underutilization [ 23 ] . For example, deploying a ResNet50 [ 24 ] model with fp16 mixed-precision 1 1 1 Weight precision refers to the accuracy of representing a DL model’s weights, usually in terms of number of bits. on an NVIDIA Jetson Orin Nano device [ 25 ] with TensorRT optimisations shows over 98\% GPU utilisation while GPU memory usage remains below 3\% . Throughput 2 2 2 Throughput is the number of images processed per unit time. increases with batch size 3 3 3 Batch size is the number of images processed per inference loop. but levels off at higher values, as illustrated in Fig- 1 . Fig. 1 : GPU Memory Usage and Throughput vs. Batch Size for ResNet50 fp16 precision model Power consumption is also a key consideration for edge devices during runtime. These complexities necessitate a thorough study of various metrics and a comparison of different tools, making deployment and DL model design optimisation decisions intricate and labour-intensive. To this end, this paper provides an in-depth analysis of computing resource usage of two NVIDIA Jetson devices (Jetson Orin Nano [ 25 ] , Jetson Nano [ 26 ] ), focusing on the impact of different weight precisions, batch sizes and multiple concurrent DL processes on metrics such as throughput, energy consumption, and other low-level GPU metrics. Key contributions include: 1. Analysis of GPU-accelerated vision inference workloads on NVIDIA Jetson devices and identification of relevant metrics using existing profiling tools. 2. Evaluation of the impact of different numeric precision levels, batch sizes, and numbers of concurrent processes across the SoC, GPU, and kernel levels in modern edge GPUs. 3. Identification of runtime bottlenecks arising from varying batch sizes and concurrent execution levels, with insights into their implications for system performance and scalability. In the following sections, we discuss background (Section II ). We then provide an overview of our experimental setup and methodology (Sections III and IV ). In Section V we provide detailed collected metrics and provide importance in different contexts. Section- VI presents a detailed workload analysis of our extensive experiments. Section- VII provides an overview of kernel-level analysis and potential bottlenecks during runtime execution. II Background Deep Neural Networks (DNNs) [ 13 ] are increasingly employed in edge AI applications such as classification [ 12 ] , object detection [ 27 ] etc., where we extract features and classify inputs. Traditionally, cloud offloading solutions [ 28 ] rely on high bandwidth and efficient resource allocation. Recently, inference-on-edge [ 29 , 30 ] has emerged as a promising alternative, offering reductions in latency, bandwidth consumption, and power usage. Techniques such as model redesign and custom architectures are driving advancements in edge-based inference. However, the diversity of hardware and software environments complicates the development of universally optimised frameworks. On general-purpose hardware, linear algebra libraries like BLAS [ 31 ] enable efficient deep-learning computations, but manual optimisation for each hardware configuration remains a laborious process. To mitigate this, domain-specific deep learning compilers and frameworks such as TVM [ 32 ] , Glow [ 33 ] , nGraph [ 34 ] , XLA [ 35 ] , and TensorRT [ 19 ] have gained prominence. These compilers optimise model-specific implementations for diverse hardware, employing techniques like layer and operator fusion to enable efficient code generation. Among edge platforms, GPUs are a prevalent architecture for inference due to their broad support. Existing research stresses the importance of concurrent execution within GPUs to maximise their capabilities [ 36 , 37 ] . GPUs managing multiple concurrent applications must support some form of virtualisation. Application-level concurrency is a relatively recent development, primarily supported on cloud-based GPUs. The two prevalent approaches