VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination

Muyan Hu Ahan Gupta Jiachen Yuan Vima Gupta Taeksang Kim Xin Xu, Janardhan Kulkarni Ofer Dekel Vikram Adve Charith Mendis, [.3em] University of Illinois Urbana-Champaign Georgia Institute of Technology Microsoft · 2026

Read VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination by Muyan Hu Ahan Gupta Jiachen Yuan Vima Gupta Taeksang Kim Xin Xu, Janardhan Kulkarni Ofer Dekel Vikram Adve Charith Mendis, [.3em] University of Illinois Urbana-Champaign Georgia Institute of Technology Microsoft (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09558.txt.

Introduction

Hu and nine colleagues introduce VTC, a compiler framework that removes needless data movement from neural-network workloads. Its virtual tensors track movement between compute steps through index maps, instead of shuttling data to and from global memory. A new search routine picks the profitable virtual-tensor plan automatically, and the design works with existing compute kernels across arbitrary operator mixes. On graphics processors it beats current compilers by up to about 2 times, averaging about 1.3 times, while cutting inference memory by up to 60 percent.

Catalog record

CREATORMuyan Hu Ahan Gupta Jiachen Yuan Vima Gupta Taeksang Kim Xin Xu, Janardhan Kulkarni Ofer Dekel Vikram Adve Charith Mendis, [.3em] University of Illinois Urbana-Champaign Georgia Institute of Technology Microsoft
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61890
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination Muyan Hu Ahan Gupta Jiachen Yuan Vima Gupta Taeksang Kim Xin Xu, Janardhan Kulkarni Ofer Dekel Vikram Adve Charith Mendis, [.3em] University of Illinois Urbana-Champaign Georgia Institute of Technology Microsoft Abstract With the widening gap between compute and memory operation latencies, data movement optimizations have become increasingly important for DNN compilation. Current optimizations such as layout transformations and operator fusion only target a subset of tensor operators and consequently miss important opportunities for reducing data movement in contemporary DNN workloads, including large language models. We introduce VTC, a novel tensor compilation framework that for the first time eliminates all unnecessary data movement by targeting the full spectrum of data movement operators. VTC proposes the concept of virtual tensors to track data movement between compute operators via index mappings rather than expensive physical data transfers to and from global memory, which can seamlessly interoperate with existing computation kernels and handle arbitrary tensor operator compositions. We also introduce a novel data movement elimination algorithm to automatically identify a profitable virtual tensor creation strategy. Evaluation on a variety of DNNs shows that VTC can outperform existing ML compilers by up to 1.93 × 1.93\times ( 1.28 × 1.28\times on average) on NVIDIA GPUs with up to 60% (17.5% on average) inference memory savings. 1 Introduction Deep neural network (DNN) workloads have gained popularity in recent years. Usually, DNN models are expressed as tensor (i.e. n- dimensional array) based computations. Consequently, DNN developers express these computations using tensor programming languages such as TensorFlow [ 3 ] , JAX [ 7 ] , and PyTorch [ 5 ] , and then utilize tensor compilers like XLA [ 1 ] , TorchInductor [ 5 ] , and TVM [ 9 ] to generate highly performant executables targeting various hardware devices. These compilers use a multi-stage pipeline similar to general-purpose compilers [ 6 ] . During the compiler frontend , computations expressed in tensor programming languages are transformed into a compiler intermediate representation (IR), known as computation graph . Nodes of these graphs represent tensor operators (e.g. matrix multiplications, convolutions), and each edge represents a tensor flowing from the output of a producer node to the input of a consumer node. The compiler middle-end performs graph-level transformations (e.g. operator fusion, tiling) to optimize the computation graph. Finally, the compiler backend maps the computation graph onto a set of kernels, each of which is a program fragment written in single-program-multiple-data (SPMD) fashion on modern hardware accelerators. To keep up with the computational demands of DNN models, specialized hardware accelerators, such as NVIDIA Tensor Cores, have been introduced. Each new generation of these accelerators has consistently improved the compute capacity to provide blazing speeds for compute instructions such as matrix multiplications (e.g., nearly 1 PFLOPS half precision on NVIDIA H100). Most tensor compilers already aim to maximize utilization on such hardware using dedicated compiler backends (e.g. Triton [ 30 ] for GPUs), achieving low latency and high throughput for compute operations. However, memory technology in modern hardware has not kept pace with these advances in compute capabilities. Figure 1 illustrates the widening compute-to-memory ratio as newer accelerators are introduced, making memory instructions significantly more expensive than compute instructions. Moreover, an increasing number of DNN models are becoming memory bound, with performance dominated by memory access requests to and from the accelerator’s global memory, which exacerbates the impact of slower memory. For example, in large language models (LLMs), the incremental decoding stage is memory-bound and can be the bottleneck of end-to-end inference [ 27 ] . Memory access requests are determined by the types and the composition of data movement operations in computation graphs. These operations only transfer data between global memory and the accelerator without performing any computation on tensor data with compute units. Critically, these inserted data movements between computational operators could introduce substantial extra latency. For example, Figure 2 demonstrates TensorRT’s latency breakdown on a Llama 3 8B [ 14 ] decoder layer. Between the computation of QKV projection and FlashDecoding [ 11 ] , the data movement operators take even more time than other computational operators. These findings underscore the critical importance of optimizing data movement operations in DNN compilers. Figure 1: Trend of compute/memory ratio for NVIDIA GPUs over time. The ratio is calculated by dividing the computation power (half-precision performance in TFLOPS for Tesla GPUs and INT8 performance in TOPS for Jetson Edge GPUs) by the peak GPU memory bandwidth (in TB/s). Figure 2: Motivating example: computation graph and latency breakdown for part of a Llama-3 decoder layer. The model size of Llama-3 is 8B, with batch size 16, context length 4096, query length 1 (decoding stage), bfloat16 precision. Frame 3 denotes a single kernel fusing all the operators with FlashDecoding. VTC can eliminate all the data movement operators in frame 2 and achieve a 4\times speed-up over TensorRT on this subgraph. Prior works that propose optimizations for data movement operations in tensor compilers broadly fall under two categories: layout optimizations and operator fusion , both of which happen in the compiler middle end. These techniques are usually incomplete, targeting only a subset of the data movement operators and miss profitable data movement elimination opportunities that could lead to significant speedups, as detailed below. Layout optimizations. The layout of a tensor determines in which order its dimensions are linearized into memory. When a producer operator generates an output tensor with a layout that differs from the expected input layout of the consumer operator, data layout conversion operators need to be inserted, resulting in excessive and avoidable data movement overhead. To find the optimal data layouts that minimize such overheads, previous work first identifies layout-sensitive tensor operators and optimally selects the adjacent layout operators (primarily Reshape and Transpose operators) to balance data movement overhead against computation efficiency [ 2 , 16 , 24 ] . However, the tensor operators considered in these works represent only a subset of data movement operations used in DNN models, overlooking optimization opportunities in other data movement operators (e.g., ScatterND ) that are key for improved performance (see Section 3.1 ). Operator fusion. During DNN execution, operators usually serve as boundaries of global memory data movement – all input and output tensors are stored in global memory. For each operator, its corresponding kernel first loads inputs from global memory into on-chip buffers, performs computations, and then writes results back from on-chip buffers to global memory. To reduce this costly data movement, existing deep learning frameworks [ 2 , 9 , 5 , 3 ] employ operator fusion, which enables intermediate results to be reused directly from fast on-chip buffers rather than being written to and read from slower global memory. However, these systems perform fusion using hand-crafted rules that target specific data access patterns (e.g., [ 23 ] ), making this optimization only available for operators that can be fused. As a result, similar to layout optimizations, operator fusion reduces data movement for only a subset of data movement operators, missing important optimization opportunities. Our Solution. In contrast with these two popular techniques, in this paper, we introduce VTC, a compiler that eliminates unnecessary data movement across all data movement

Read the complete text