Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference

Wenxin Dong, Mingqing Hu, Guanghui Yu, Qiang Fu, Peng Xu, Hui Xu, Yue Xing, Xuewu Jiao † , Shuanglong Li, Lin Liu · 2026

Read Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference by Wenxin Dong, Mingqing Hu, Guanghui Yu, Qiang Fu, Peng Xu, Hui Xu, Yue Xing, Xuewu Jiao † , Shuanglong Li, Lin Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11581.txt.

Introduction

Dong and colleagues present Ada-MK, adaptive MegaKernel optimization for LLM inference. Each decode token triggers thousands of launches, with launch overhead alone at 14 point 6 percent of inference time. MegaKernel fuses operators into one kernel, but hand-tuned versions lack portability, auto-compiled versions add costly branching. Ada-MK moves decisions to compile time with a shared-memory model, MLIR-based DAG search and a hybrid TensorRT-LLM engine. Single-batch throughput rises up to 23 point 6 percent over TensorRT-LLM and 50 point 2 percent over vLLM.

Catalog record

CREATORWenxin Dong, Mingqing Hu, Guanghui Yu, Qiang Fu, Peng Xu, Hui Xu, Yue Xing, Xuewu Jiao † , Shuanglong Li, Lin Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41899
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference Wenxin Dong, Mingqing Hu, Guanghui Yu, Qiang Fu, Peng Xu, Hui Xu, Yue Xing, Xuewu Jiao † , Shuanglong Li, Lin Liu Abstract . When large language models (LLMs) serve real-time inference in commercial online advertising systems, end-to-end latency must be strictly bounded to the millisecond range. Yet every token generated during the decode phase triggers thousands of kernel launches, and kernel launch overhead alone can account for 14.6% of end-to-end inference time. MegaKernel eliminates launch overhead and inter-operator HBM round-trips by fusing multiple operators into a single persistent kernel. However, existing MegaKernel implementations face a fundamental tension between portability and efficiency on resource-constrained GPUs such as NVIDIA Ada: hand-tuned solutions are tightly coupled to specific architectures and lack portability, while auto-compiled approaches introduce runtime dynamic scheduling whose branch penalties are unacceptable in latency-critical settings. We observe that under a fixed deployment configuration, the optimal execution path of a MegaKernel is uniquely determined, and runtime dynamic decision-making can be entirely hoisted to compile time. Building on this insight, we propose Ada-MK : (1) a three-dimensional shared-memory constraint model combined with K-dimension splitting that reduces peak shared memory usage by 50%; (2) MLIR-based fine-grained DAG offline search that solidifies the optimal execution path, completely eliminating runtime branching; and (3) a heterogeneous hybrid inference engine that embeds MegaKernel as a plugin into TensorRT-LLM, combining high-throughput Prefill with low-latency Decode. On an NVIDIA L20, Ada-MK improves single-batch throughput by up to 23.6% over vanilla TensorRT-LLM and 50.2% over vLLM, achieving positive gains across all tested scenarios—the first industrial deployment of MegaKernel in a commercial online advertising system. 1. Introduction Large language models (LLMs) are increasingly deployed in search, recommendation, and conversational applications, with parameter counts growing from billions to hundreds of billions. Inference latency has thus become a key bottleneck for industrial adoption, especially in latency-sensitive online services where efficient inference under limited hardware resources is critical. For extremely latency-sensitive scenarios, frequent global memory accesses and kernel launch overhead on GPUs have become the primary bottlenecks for end-to-end performance ( Kwon et al. 2023 ; NVIDIA 2024 ) . Profiling Qwen2.5-1.5B with Nsight Systems under TensorRT-LLM reveals that kernel launch overhead accounts for approximately 14.6% of end-to-end inference time (1,655,550 launches consuming \sim 3.3 s). In conventional pipelines, adjacent kernels must exchange intermediate results through HBM; MegaKernel instead leverages shared memory and registers to achieve seamless operator chaining and deep parallelism, fundamentally eliminating operator switching overhead and HBM round-trip latency. MegaKernel achieves single-launch persistent computation through Persistent Kernels, constructs producer–consumer pipelines via Warp Specialization, and overlaps computation with memory access through TMA and asynchronous I/O ( HazyResearch 2025 ) . Our target deployment is a commercial online advertising system running on NVIDIA Ada (L20) GPUs ( NVIDIA 2022a ) . Deploying MegaKernel on Ada faces severe dual constraints. First, the business demands deterministic end-to-end latency strictly within 1–5 ms. Second, the Ada architecture natively lacks TMA hardware support, requiring PTX assembly and hand-crafted software pipelines to emulate asynchronous data movement; its on-chip shared memory is only half that of the H100 (128 KB vs. 227 KB) ( NVIDIA 2022b ) , severely compressing the optimization space for pipeline stages and tile sizes. In contrast, Hopper/Blackwell architectures provide TMA hardware support and larger shared memory (227 KB), making MegaKernel deployment far more straightforward ( HazyResearch 2025 ; NVIDIA 2022b ) . In practice, the shared memory limitation on Ada reduces the achievable pipeline stages from the theoretical optimum of 4 to only 2, incurring a pipeline duty-cycle loss exceeding 30%. Existing solutions fail to meet these requirements. Stanford’s MegaKernel ( HazyResearch 2025 ) delivers strong performance, but its codebase is deeply tied to Hopper/Blackwell-specific assembly optimizations and supports only a few model architectures (e.g., Llama-1B), lacking support for Qwen and other widely-used models ( Qwen Team 2024 ; Qwen Team 2025 ) ; it also lacks support for long-sequence and large-batch Prefill phases. Mirage MPK ( Cheng et al. 2025 ; Wu et al. 2024 ) improves usability through auto-tuning, but its Managed Pointer mechanism introduces runtime if-else branching based on shared memory page states, degrading instruction issue efficiency in ultra-low-latency scenarios and falling short of hand-tuned operator performance. To address these challenges, we propose Ada-MK with the following core contributions: • Adaptive shared memory management. We model shared memory allocation from three dimensions—hardware specifications, model architecture, and dynamic workload—and reduce peak shared memory demand by 50% through K-dimension fine-grained splitting, while enabling cross-operator page reuse to reconstruct efficient pipelines on Ada’s constrained resources. • Fine-grained DAG-based automatic search. We leverage MLIR Lowering ( Lattner et al. 2021 ) to construct PTX-level dependency DAGs and solidify the optimal execution trace through offline profiling, completely eliminating runtime dynamic decision overhead. Unlike Ansor ( Zheng et al. 2020 ) and other traditional auto-tuning frameworks, Ada-MK’s DAG-level search captures finer-grained parallelism opportunities. • Heterogeneous hybrid inference engine. We embed MegaKernel as a plugin into TensorRT-LLM ( NVIDIA 2024 ) , reusing TensorRT-LLM’s native operators for Prefill and switching to the MegaKernel engine for Decode, achieving both high throughput and low latency—the first industrial deployment of MegaKernel. Experiments show that Ada-MK reduces end-to-end latency by 10%–50% over vLLM ( Kwon et al. 2023 ) , SGLang ( Zheng et al. 2025b ) , and vanilla TensorRT-LLM ( NVIDIA 2024 ) across Qwen model series, and has been deployed in production within Baidu’s commercial online advertising system. The remainder of this paper is organized as follows: § 2 presents background and related work; § 3 describes the Ada-MK overall architecture; § 4 details the three core optimizations; § 5 presents experimental evaluation; § 6 concludes with future directions. 2. Background and Related Work Figure 1. Ada-MK overall architecture. Phase I (Offline MegaKernel Synthesis): the Transformer Decoder and LM Head are parsed into a fine-grained DAG, pruned by hardware resource constraints, and profiled to select the optimal execution trace, which is then serialized. Phase II (Online Warp-Specialized Execution): the serialized MegaKernel is embedded as a TensorRT-LLM plugin, with warp-specialized roles (Loader, Consumer, Storer, Controller, Launcher) collaborating through shared memory to form an efficient software pipeline. 2.1. LLM GPU Kernel Techniques: From Local to Global Fusion GPU operator optimization has evolved from local to global fusion, forming several distinct research directions: Traditional Kernel Fusion merges adjacent kernels to reduce launch counts, but still relies on global memory for intermediate data exchange, making it difficult to circumvent memory bandwidth and data dependency bottlenecks at extreme model scales. CUDAGraph reduces launch overhead by capturing and replaying kernel sequences, but cannot achieve deep inter-operator fusion. Compiler-level operator fusion. Triton ( Tillet et al. 2019 ) achieves compile-time operator fusion through block-level programming abstractions, while Hidet ( Ding et al. 2023 ) provides finer-grained scheduling control via a task-mapping paradigm. These approaches improve fusion depth but still operate on a “kernel-per-operator” execution model, unable to eliminate operator switching overhead. Additionally, MCFuser ( Zhang et al. 2024 ) achieves high-performance fusion for memory-bound compute-intensive operator chains, Neptune ( Zhao et al. 2025 ) enables advanced fusion of reduction operators by breaking loop dependencies, and Magneto ( Di et al. 2025 ) focuses on coordinated optimization

Read the complete text