Hybrid JIT–CUDA Graph Optimization for Low-Latency Large Language Model Inference
Divakar Kumar Yadav, Tian Zhao · 2026
Read Hybrid JIT–CUDA Graph Optimization for Low-Latency Large Language Model Inference by Divakar Kumar Yadav, Tian Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23467.txt.
Introduction
Yadav and Zhao speed up language-model inference by splitting the work between two execution styles: fixed parts replay through CUDA graphs while changing parts run through just-in-time compiled kernels. The hybrid keeps runtime flexibility during token-by-token generation, cutting the wait for the first token by up to about two-thirds against a strong baseline. Tests on a 7-billion-parameter model cover short prompts, where launch overhead matters most.
Catalog record
| CREATOR | Divakar Kumar Yadav, Tian Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28776 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hybrid JIT–CUDA Graph Optimization for Low-Latency Large Language Model Inference Divakar Kumar Yadav, Tian Zhao Abstract Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains constrained by inference latency and kernel launch overhead, particularly in interactive, short-sequence settings. This paper presents a hybrid runtime framework that combines Just-In-Time (JIT) compilation with CUDA Graph execution to reduce launch overhead while preserving runtime flexibility during autoregressive decoding. The framework partitions transformer inference into static components executed via CUDA Graph replay and dynamic components handled through JIT-compiled kernels, enabling asynchronous graph capture and reuse across decoding steps. We evaluate the proposed approach on LLaMA-2 7B using single-GPU, batch-size-one inference across prompt lengths from 10 to 500 tokens. Experimental results show that the hybrid runtime reduces Time-to-First-Token (TTFT) by up to 66.0% and achieves lower P99 latency compared with TensorRT–LLM in this regime. These results indicate that hybrid JIT–CUDA Graph execution can effectively reduce inference latency and variance for short-sequence LLM workloads, making it a practical optimization strategy for latency-sensitive AI applications. I Introduction Large Language Models (LLMs) such as GPT-4 [ 4 ] , Claude [ 3 ] , Gemini [ 13 ] , and Grok [ 34 ] have rapidly evolved into core inference engines for conversational, coding, and multimodal AI systems. Despite their strong model capabilities, the real-time deployment of LLMs remains constrained by inference latency, particularly in interactive settings where users generate and consume responses incrementally. Prior work has shown that autoregressive decoding incurs substantial overhead from repeated GPU kernel launches, synchronization events, and host–device coordination [ 10 , 28 , 1 , 18 ] . These costs accumulate across decoding steps and become especially visible in latency-sensitive inference scenarios. To mitigate inference overhead, modern LLM runtimes employ a combination of compiler- and kernel-level optimizations, including operator fusion [ 29 , 31 , 26 ] , quantization [ 9 , 12 ] , and specialized attention kernels such as FlashAttention [ 7 , 8 ] . Production-oriented frameworks such as TensorRT–LLM [ 21 ] and FasterTransformer [ 20 ] combine these techniques to improve throughput and reduce memory overhead. Nevertheless, even highly optimized pipelines continue to incur non-trivial latency from repeated kernel dispatch and dynamic tensor management during autoregressive decoding, particularly for small batch sizes and interactive use cases. Fig. 1: High-level architecture of the Hybrid JIT–CUDA Graph Runtime. Process 1 (JIT Context Generator) executes dynamic operations such as preprocessing and sampling, while Process 2 (CUDA Graph Generator) executes static compute kernels through captured CUDA Graphs, coordinated via inter-process communication (IPC). I-A Inference Length Interactive LLM applications, including conversational agents, code assistants, and decision-support systems, are typically dominated by short-to-medium generation workloads in which responses are produced incrementally and latency sensitivity is high. Public model documentation and large-scale observational studies of deployed systems [ 3 , 5 ] consistently indicate that many real-world interactions favor relatively limited response lengths rather than long-form generation. From a systems perspective, this regime is particularly important because inference latency and variance are most perceptible to end users during early decoding steps and moderate-length generations. Optimizing execution for response lengths on the order of a few hundred tokens therefore addresses a common and operationally relevant class of workloads, even when longer-context capabilities are available. In this work, we focus on autoregressive inference up to 500 generated tokens as a representative window for latency-sensitive evaluation. This choice reflects practical deployment considerations rather than an assumption of universal workload distributions, and allows us to study kernel launch overhead, replay behavior, and tail latency under realistic interactive conditions. I-B Problem Context A primary bottleneck in LLM inference arises from the CPU-bound dispatch of numerous fine-grained GPU kernels executed sequentially at each decoding step. Even frameworks that employ dynamic graph tracing and compilation (e.g. torch.compile ) cannot fully eliminate host-side coordination and Python dispatch overhead [ 25 ] . CUDA Graphs [ 22 , 27 ] provide a mechanism to pre-capture and replay GPU workloads with minimal host intervention, substantially reducing launch overhead. However, CUDA Graphs require static tensor shapes and deterministic control flow, making them incompatible with dynamic operations such as variable-length attention, cache updates, and stochastic sampling commonly found in LLM inference. Conversely, Just-In-Time (JIT) compilation [ 24 , 2 ] can accommodate dynamic control flow and runtime-dependent tensor shapes, but JIT-executed kernels still incur per-launch overhead and variability. As a result, existing approaches face a fundamental trade-off between static execution efficiency and dynamic flexibility during autoregressive decoding. I-C Contributions This work proposes a hybrid JIT–CUDA Graph runtime, illustrated in Fig. 1 , that bridges this trade-off by combining static CUDA Graph replay with dynamic JIT execution. Specifically, we make the following contributions: • We partition the transformer inference pipeline into static components that are safe for CUDA Graph capture and dynamic components that require JIT execution. • We introduce an asynchronous CUDA Graph generation mechanism that captures static subgraphs at multiple sequence lengths without blocking inference execution. • We overlap JIT-executed dynamic operations with deterministic CUDA Graph replay to reduce kernel launch overhead and latency variance during autoregressive decoding. • We demonstrate measurable reductions in Time-to-First-Token (TTFT) and tail latency for short-to-medium length LLM inference workloads on a single GPU. The remainder of the paper is structured as follows: Section II surveys related work; Section III introduces the hybrid framework and architecture; Section IV describes implementation details and fairness methodology; Section V presents empirical evaluation; Section VI discusses limitations; and Section VII concludes with future directions. II Related Work II-A Transformer Optimization Transformer-based architectures [ 33 , 4 , 32 , 17 ] form the foundation of modern generative AI systems. Scaling these models for efficient inference has motivated a rich ecosystem of GPU-optimized runtimes, including Megatron-LM [ 30 ] , DeepSpeed-Inference [ 1 , 28 ] , FasterTransformer [ 20 ] , and TensorRT–LLM [ 21 ] . These systems employ techniques such as mixed-precision execution, fused attention kernels, and pipeline or tensor parallelism to improve throughput and reduce memory footprint. While highly effective for batch-oriented and throughput-driven workloads, these runtimes are typically designed around static or semi-static execution graphs. As a result, accommodating fine-grained dynamic behavior – such as variable sequence lengths, token-by-token autoregressive decoding, and stochastic sampling – often requires falling back to host-side coordination, which can introduce additional latency and variance in interactive inference settings. II-B Kernel Fusion Compiler-based approaches aim to reduce kernel launch overhead by fusing operator boundaries at compile time or runtime. For example, nvFuser [ 29 ] , TorchDynamo [ 25 ] , and the torch.compile stack [ 2 , 26 ] transform Python-level operator graphs into optimized CUDA kernels through a combination of ahead-of-time and just-in-time compilation. Domain-specific systems such as Triton [ 31 ] and TVM [ 6 ] further enable operator specialization and aggressive fusion for deep learning workloads. Despite these advances, fused kernels are typically invoked through host-side scheduling and remain sensitive to dynamic shapes and control flow. In autoregressive decoding, where kernel invocation patterns and tensor shapes evolve at each step, residual launch overhead and compilation boundaries can still dominate end-to-end latency. II-C CUDA Graph Replay CUDA Graphs [ 22 , 27 ,