GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion
Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, Andi Quinn · 2026
Read GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion by Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, Andi Quinn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17861.txt.
Introduction
Yiwei Yang and five colleagues present GPUOS, a graphics processor runtime that cuts launch overhead for workloads made of many small tensor operations. Instead of launching a fresh kernel for each task, one long-lived kernel drains a host-managed work queue, while operators are compiled just in time and injected through function pointer tables without restarting. The design supports varied shapes, strides, types, and broadcasting, and reaches up to 15.3 times the speed of standard PyTorch on small-operation workloads.
Catalog record
| CREATOR | Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, Andi Quinn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44055 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GPUOS : A GPU Operating System Primitive for Transparent Operation Fusion Yiwei Yang, Xiangyu Gao, Yuan Zhou, Yuhang Gan, Yusheng Zheng, Andi Quinn Abstract . Modern deep learning workloads increasingly involve numerous small tensor operations—particularly in inference scenarios, attention mechanisms, and micro-batched training—where producing non-negligible kernel launch overhead. Traditional GPU computing models launch separate kernels for each operation, incurring significant CPU-GPU synchronization costs that can even exceed the actual computation time by orders of magnitude. We present GPUOS, a GPU runtime JIT system that reduces kernel launch overhead through a persistent kernel architecture combined with runtime operator injection. GPUOS deploys a single long-lived GPU kernel that continuously processes tasks from a host-managed work queue, avoiding repeated kernel launches entirely. To obtain operational flexibility by effectively supporting multiple types of operations, we leverage NVIDIA’s NVRTC to just-in-time compile new operators at runtime and dynamically inject them into the running kernel via device function pointer tables. This approach enables hot-swapping of GPU operators without kernel restarts or system recompilation. Our system introduces several key innovations: (1) a persistent worker kernel with atomic-synchronized task queues that eliminates per-operation launch overhead, (2) a runtime operator injection mechanism using NVRTC and relocatable device code that maintains an updatable jump table of device function pointers, (3) a dual-slot aliasing scheme enabling safe operator updates without suspending concurrent tasks, and (4) transparent PyTorch integration via TorchDispatch that automatically aggregates micro-operations into batched submissions. The system can support arbitrary tensor shapes, strides, data types, and broadcasting semantics through a generic tensor abstraction layer. Experimental results demonstrate that GPUOS achieves 15.3x speedup over standard PyTorch for workloads dominated by small operations, with particularly strong performance on micro-batched inference (up to 8.7x utilization) and attention computation patterns. Our transparent scheduler integrates well with the existing PyTorch code base while maintaining full compatibility with the PyTorch ecosystem. GPUOS provides a practical pattern for building GPU runtime systems that bridge the gap between operator flexibility and execution efficiency. 1. Introduction 1.1. From Big Batches to Tiny, Fast Ones Over the past decade, extensive efforts have been made to make GPUs faster in computation and larger in memory capacity. Vendor libraries saturate tensor cores with near-perfect efficiency. Compiler stacks like TVM and the PyTorch 2.0 toolchain fuse long chains of operations into heavyweight launches that minimize data movement. Memory hierarchies are carefully orchestrated to keep reuse patterns close to compute units. These optimizations remain crucial for offline training and dense inference, where batch sizes are large and execution patterns are regular. Yet, they do not fit well for current scenarios where Production machine learning systems increasingly serve latency-sensitive, micro-batch workloads where the very properties that once made the host-to-device handshake negligible now make it dominant. When a single user interaction requires hundreds of microsecond-scale kernel executions—each performing a small computation on limited data—a five-microsecond submission path is no longer an ignorable constant. It becomes a tax, applied one hundred times per request, that can dominate the total response time budget ( 18 ; 19 ) . Consider the timeline traces from modern profilers. The GPU excels at mathematical operations, computing matrix multiplications and activations in blazing bursts. But between these bursts lies whitespace: the device sits idle while the host marshals the next kernel launch. The host, in turn, spends a surprising fraction of its time navigating the submission path—crossing into kernel space, updating driver structures, synchronizing state. For instance, if each of one hundred operations per token incurs five microseconds of overhead, it means half a millisecond of pure coordination cost before we’ve even considered the actual computation. Code is open sourced at https://github.com/Multi-V-VM/GPUOS/ 1.2. Our Approach To remedy the problem, we claim that the right unit to optimize in these scenarios is not an individual kernel but the boundary that incurs the launch itself . Therefore, instead of investing ever more ingenuity in making each launch cheaper—or trying to statically fuse away the multiplicity of small operations, which dynamism often defeats—GPUOS sidesteps the issue by launching the kernel exactly once and doing everything else inside. Concretely, we regard the persistent kernel as the medium; the message is a fundamental change in where scheduling resides. If the GPU is to stay busy, the source of work should not be a series of host calls separated by system calls and driver crossings. Instead, it should be a ring buffer visible to device threads that never sleep. These threads poll for work, grab task descriptors, dispatch to the appropriate operator through a function pointer table, and immediately return for the next task. Because models evolve and applications grow new features, this persistent runtime should be extensible without downtime. GPUOS accomplishes this through runtime compilation with NVRTC ( 17 ) and a device function-pointer table that can be updated safely under load via the CUDA Driver API ( 14 ) . When a new operator is needed (e.g., a custom attention variant or a novel activation function), the operations team does not have to roll the cluster or coordinate maintenance windows. They compile a small template to PTX (Parallel Thread Execution), load the module, resolve the function symbol, and publish a pointer into an inactive slot of the operator table. A version counter flip with store-release semantics makes the operator instantly callable by all device threads. 1.3. The Mechanical Consequence The consequence of this relocation is promising. Launch overhead disappears as a factor within the steady-state execution loop. Device function calls—measured in nanoseconds—replace host submissions measured in microseconds. The whitespace between kernels in timeline traces shrinks until it is indistinguishable from instruction scheduling gaps within the persistent kernel itself. Tail latency improves because queuing and wakeup paths are deterministic and short. And because there is no need to predict which sequence of operations will occur next, GPUOS retains the semantics of eager execution while reaping the performance gains of persistent scheduling. We set out not to replace compilers or graphs, but to give them a complementary option that keeps the GPU fed when dynamism prevents capture, when shapes refuse to sit still, and when the system just needs to say: don’t launch—call . Our measured results— 15.3\times on micro-batched element-wise operations, 8.7\times on attention decoding, 23.1\times on mixed pipelines, with 20–22% energy savings—aren’t products of exotic compiler stunts. They’re the arithmetic of removing a fixed cost applied too frequently. The system is complementary, integrates well with existing frameworks, and coexists with CUDA Graphs, MPS, and MIG, each mechanism pulling its weight where it excels. 2. Motivation 2.1. One Painpoint in Production We illustrate our motivation through a representative real-world case. Consider a real-time text generation service serving thousands of concurrent sessions. Each session decodes token-by-token, weaving together a pattern of attention kernels, vector additions, activation functions, small reductions, and key-value cache updates. The workload is inherently dynamic: prompt lengths vary wildly, control flow branches based on generated content, and new model features are deployed continuously to stay competitive. The engineering team upgraded to recent CUDA toolkits, enabled mixed precision everywhere possible, fused operations aggressively where the framework permitted, and experimented extensively with CUDA Graphs. In synthetic benchmarks—where prompts were short, predictable, and shape-stable—graphs delivered impressive results. But production reality proved unruly. Users paste entire paragraphs