AEG: A Baremetal Framework for AI Acceleration via Direct Hardware Access in Heterogeneous Accelerators Conference: ; ;

Hua Jiang, Sayan Mandal, Brandon Kirincich, Govind Varadarajan · 2026

Read AEG: A Baremetal Framework for AI Acceleration via Direct Hardware Access in Heterogeneous Accelerators Conference: ; ; by Hua Jiang, Sayan Mandal, Brandon Kirincich, Govind Varadarajan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09565.txt.

Introduction

Jiang and three colleagues present AEG, a bare-metal runtime that runs machine-learning inference on mixed accelerators with no operating system underneath. Control logic is flattened into linear Runtime Control Blocks, letting a generic engine run high-level models through a thin hardware layer, with platform management and an in-memory file system alongside. On a ResNet-18 image task it reaches 9.2 times the compute efficiency of a Linux-based Vitis AI setup, using only 28 accelerator tiles against 304, while holding about 69 percent Top-1 accuracy on ImageNet.

Catalog record

CREATORHua Jiang, Sayan Mandal, Brandon Kirincich, Govind Varadarajan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40938
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09565

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AEG: A Baremetal Framework for AI Acceleration via Direct Hardware Access in Heterogeneous Accelerators Conference: ; ; Hua Jiang, Sayan Mandal, Brandon Kirincich, Govind Varadarajan Abstract . This paper introduces a unified, hardware-independent baremetal runtime architecture designed to enable high-performance machine learning (ML) inference on heterogeneous accelerators, such as AI Engine (AIE) arrays, without the overhead of an underlying real-time or general-purpose operating system. Existing edge-deployment frameworks, such as TinyML, often rely on real-time operating systems (RTOS), which introduce unnecessary complexity and performance bottlenecks. To address this, our solution fundamentally decouples the runtime from hardware specifics by flattening complex control logic into linear, executable Runtime Control Blocks (RCBs). This "Control as Data" paradigm allows high-level models, including Adaptive Data Flow (ADF) graphs, to be executed by a generic engine through a minimal Runtime Hardware Abstraction Layer (RHAL). We further integrate Runtime Platform Management (RTPM) to handle system-level orchestration (including a lightweight network stack) and a Runtime In-Memory File System (RIMFS) to manage data in OS-free environments. We demonstrate the framework’s efficacy with a ResNet-18 image classification implementation. Experimental results show 9.2 × \times higher compute efficiency (throughput per AIE tile) compared to Linux-based Vitis AI deployment, 3–7 × \times reduction in data movement overhead, and near-zero latency variance (CV = 0.03 % =0.03\% ). The system achieves 68.78% Top-1 accuracy on ImageNet using only 28 AIE tiles compared to Vitis AI’s 304 tiles, validating both the efficiency and correctness of this unified bare-metal architecture. 1. Introduction The growing deployment of edge systems has intensified demand for low-latency, energy-efficient AI inference on heterogeneous accelerators. Platforms such as the AMD Versal Adaptive Compute Acceleration Platform (ACAP) integrate AIE vector processor arrays that provide high throughput for signal-processing and machine-learning workloads ( 28 ) . However, the end-to-end utilization of such accelerators is frequently bottlenecked by software stacks that assume an OS-mediated execution environment ( 8 ; 25 ) . While OS-based abstractions improve portability via compiler stacks such as TVM and embedded ML runtimes, they introduce non-trivial overheads from kernel crossings, scheduler latency, and memory subsystem effects ( 9 ; 10 ; 21 ) . In our experiments, Linux kernel transitions can lead to a 7 \times increase in latency for small (1 KB) transfers, a critical penalty for inference pipelines that require frequent tensor movement between memory and compute units ( 21 ; 18 ) . Current support for end-to-end baremetal (OS-less) software stacks that can orchestrate heterogeneous AI accelerators without relying on an operating system remains limited. Prior work on baremetal ML deployment motivates this direction, yet often results in rigid, hardware-specific solutions that lack scalability ( 16 ) . This paper addresses this gap by presenting a unified, hardware-independent baremetal runtime architecture. Unlike traditional approaches that hard-code control logic for specific devices, our framework fundamentally decouples the execution model from hardware specifics through a "Control-as-Data" paradigm. We introduce runtime control blocks (RCBs), which flatten complex graph execution semantics into linear, executable data sequences. This allows a generic runtime engine to orchestrate operations across diverse hardware targets from AMD AIEs to emerging spatial architectures without recompilation or OS intervention. The main contributions of this work are: • Control-as-Data for Baremetal Resource Decoupling: We address software resource limitations inherent to baremetal environments by adopting a control-as-data mechanism. By encoding execution semantics into structured, hardware-agnostic command streams (runtime control blocks) that resemble the task instruction approach in VTA ( 22 ) , we decouple the runtime from OS-level software dependencies and eliminate the need for device-specific control logic within the runtime core. • Layered Abstraction Architecture: We introduce a Runtime Hardware Abstraction Layer (RHAL) to isolate hardware heterogeneity via a minimal primitive interface, and a Runtime In-Memory File System (RIMFS) to provide unified, file-like data management in OS-free environments. • System-Level Orchestration: We present Runtime Platform Management (RTPM), a module that assumes the role of a system executive, handling global cache coherency, interrupt dispatching, and secure network connectivity for remote provisioning. • Performance & Portability: We demonstrate that this architecture not only facilitates the seamless integration of high-level frameworks (such as ADF and PyTorch) but also eliminates user-kernel switching overheads, delivering superior latency determinism and throughput compared to OS-based baselines. 2. Background 2.1. AI Engine Architecture The AIE in AMD Versal devices represents a spatial accelerator paradigm: a two-dimensional array of tiles, each integrating a very long instruction word (VLIW) processor with single instruction multiple data (SIMD) capabilities, local SRAM, and a configurable AXI4-Stream switch ( 1 ; 3 ; 26 ) . Unlike GPU architectures that rely on hardware-managed caches, AIE requires software to explicitly choreograph data movement via distributed direct memory access (DMA) engines using global memory I/O (GMIO), making it inherently suited to dataflow execution but demanding precise orchestration of transfers and buffer management ( 1 ) . 2.2. Existing AIE Deployment Approaches Production runtime. Vitis AI is the current state-of-the-art framework for deploying neural networks on AMD AIE arrays ( 5 ) . It provides quantization, compilation, and runtime (VART) through a Linux-based driver stack. While Vitis AI offers mature tile mapping and scheduling, it operates through kernel-mediated control: applications request driver cooperation, allocate memory via kernel interfaces, and issue DMA commands via ioctl calls. Compiler infrastructures. MLIR-AIE ( 11 ) and MLIR-AIR ( 27 ) target the AIE compute fabric through compiler-managed scheduling, while XTA ( 23 ) pipelines graphs across system-on-chip (SoC) compute units. These solutions optimize compute placement but remain constrained by OS-level scheduling overhead. Edge ML frameworks. General-purpose stacks such as TVM ( 9 ) and TinyML runtimes (TensorFlow Lite Micro, CMSIS-NN) ( 10 ; 17 ) bridge high-level models to target code but typically assume OS-managed execution contexts. 2.3. Baremetal Computing OS-mediated runtimes introduce control-path overheads from system calls, context switches, and scheduler latency ( 21 ; 7 ) . For streaming inference with frequent small transfers, these fixed costs can dominate end-to-end latency. Baremetal execution eliminates user-kernel crossings, improving both latency and predictability (reduced jitter). However, baremetal deployment on spatial accelerators like AIE presents unique challenges: • Missing platform services: No file system for weights, no TCP/IP stack, no dynamic memory manager, all must be reimplemented in lightweight form. • Dependency isolation: Standard runtimes depend on libraries requiring system calls; a baremetal solution must be self-contained. • Explicit orchestration: The spatially distributed compute array requires manual coordination of DMA transfers, buffer lifetimes, and tile synchronization ( 2 ) . This motivates our central question: Can we preserve high-level programmability while eliminating OS overhead? The following section presents a unified baremetal architecture that adopts a "Control as Data" philosophy, encoding execution semantics into hardware-agnostic runtime control blocks (RCBs), while internalizing platform services through modular components (RHAL, RIMFS, RTPM). 3. Methodology 3.1. System Overview We propose a unified baremetal software stack that drastically reduces the integration complexity required to deploy existing ML models in OS-free environments. The overarching design goal is to establish a streamlined deployment path that allows standard ML models to run directly on baremetal, effectively replacing heavy OS-provided services such as memory allocation, device configuration, and peripheral I/O with a compact, application-level platform layer. To demonstrate the efficacy of this approach, we utilize the AIE array and the ADF framework as

Read the complete text