TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning

Chaoyao Shen, Linfeng Jiang, Yixian Shen, Tao Xu, Guoqing Li, Anuj Pathania, Andy D. Pimentel, Meng Zhang · 2026

Read TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning by Chaoyao Shen, Linfeng Jiang, Yixian Shen, Tao Xu, Guoqing Li, Anuj Pathania, Andy D. Pimentel, Meng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12891.txt.

Introduction

Chaoyao Shen and seven colleagues speed tensor program tuning across hardware with continual learning. Compilers now need large offline sets and transfer poorly between platforms. TCL samples only 10 percent of programs, chosen for representativeness, diversity, and uncertainty. A Mamba-based cost model captures long-range schedule links at low cost. Continuous knowledge distillation moves knowledge across platforms without parameter explosion. Against Tenset-MLP, tuning runs 16.8 times faster on CPU and 12.48 times faster on GPU on average.

Catalog record

CREATORChaoyao Shen, Linfeng Jiang, Yixian Shen, Tao Xu, Guoqing Li, Anuj Pathania, Andy D. Pimentel, Meng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68576
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TCL: Enabling Fast and Efficient Cross-Hardware Tensor Program Optimization via Continual Learning Thanks: New Paper, Not an extension of a Conference Paper CCS: Computing methodologies Machine learning CCS: Software and its engineering Compilers Chaoyao Shen, Linfeng Jiang, Yixian Shen, Tao Xu, Guoqing Li, Anuj Pathania, Andy D. Pimentel, Meng Zhang Abstract . Deep learning (DL) compilers rely on cost models and auto-tuning to optimize tensor programs for target hardware. However, existing approaches depend on large offline datasets, incurring high collection costs and offering suboptimal transferability across platforms. In this paper, we introduce TCL , a novel efficient and transferable compiler framework for fast tensor program optimization across diverse hardware platforms to address these challenges. Specifically, TCL is built on three core enablers: (1) the RDU Sampler , a data-efficient active learning strategy that selects only 10% of tensor programs by jointly optimizing R epresentativeness, D iversity, and U ncertainty, substantially reducing data collection costs while maintaining near-original model accuracy (2) a new Mamba-based cost model that efficiently captures long-range schedule dependencies while achieving a favorable trade-off between prediction accuracy and computational cost through reduced parameterization and lightweight sequence modeling; and (3) a continuous knowledge distillation framework that effectively and progressively transfers knowledge across multiple hardware platforms while avoiding the parameter explosion and data dependency issues typically caused by traditional multi-task learning. Extensive experiments validate the effectiveness of each individual enabler and the holistic TCL framework. When optimizing a range of mainstream DL models on both CPU and GPU platforms, TCL achieves, on average, 16.8 × 16.8\times and 12.48 × 12.48\times faster tuning time, and 1.20 × 1.20\times and 1.13 × 1.13\times lower inference latency, respectively, compared to Tenset-MLP. 1. Introduction Deep neural networks (DNNs) have achieved remarkable success in domains such as computer vision ( 42 ; 63 ; 17 ; 41 ; 49 ; 25 ) , natural language processing ( 57 ; 40 ; 45 ; 41 ) , and autonomous driving ( 47 ; 8 ) , driving rapid innovation in AI hardware accelerators ( 2 ) . Specialized platforms such as NVIDIA GPUs ( 14 ) , Google TPUs ( 28 ) , and GraphCore IPUs ( 27 ) meet the growing computational demands of these models. Deep learning (DL) compilers ( 22 ) have emerged as a critical bridge between high-level DNN models and low-level hardware execution. Traditionally, tensor operators are implemented using manually optimized kernel libraries, such as NVIDIA cuDNN ( 11 ) and Intel OneDNN ( 33 ) , which are finely tuned for specific hardware. While these libraries offer excellent performance, they are often tightly coupled to particular architectures, making them difficult to adapt to new or diverse platforms. Moreover, they struggle to support emerging or non-standard operators, limiting their applicability in rapidly evolving model architectures. To address the limitations, search-based DL compilers ( 9 ; 4 ) have been developed to automatically generate and optimize tensor programs across a variety of hardware backends. Unlike static kernel libraries, these compilers explore the space of schedule transformations and select implementations tailored to the target device. A key component of this process is the cost model , which predicts the performance (e.g., latency or throughput) of candidate tensor programs and guides the search process effectively. Current search-based DL compilers often train cost models online ( 3 ; 62 ) , by iteratively sampling candidate tensor programs, evaluating their hardware performance, and updating the model during auto-tuning. While this dynamic feedback loop enables hardware-specific adaptation, it comes at a high cost: each program must be compiled and executed during search, which significantly increases tuning time and demands extensive computing resources. The release of large-scale tensor program datasets, such as Tenset ( 61 ) , has made it feasible to shift cost model training offline. These datasets provide millions of tensor programs with performance measurements across hardware backends, enabling the development of predictive models without requiring real-time hardware profiling. Motivated by this, recent work has increasingly focused on offline training of cost models ( 46 ) . In this paradigm, the cost model is trained once on pre-collected data from a specific hardware platform and can then be used to predict the performance of unseen tensor programs during auto-tuning. To further reduce hardware-specific data collection, researchers have explored transfer learning strategies ( 16 ; 38 ) , which adapt cost models pre-trained on one or more source platforms to new target devices by leveraging shared compiler optimization patterns and applying platform-specific fine-tuning. This combination of offline training and transfer learning reduces tuning time and resource usage while improving compiler adaptability across diverse hardware environments. Although DL compilers equipped with offline-trained cost models demonstrate strong performance in terms of tuning quality and efficiency, several practical and architectural challenges remain. First, collecting a sufficiently large set of tensor programs for each hardware platform is highly time-consuming. In our measurements, data collection on an Intel i7-12700F CPU requires approximately 40 days, while the same process on an NVIDIA GeForce RTX 3080Ti GPU takes over 60 days, posing a major bottleneck for scalability. Second, the architecture of the cost model fundamentally limits the accuracy and efficiency of performance prediction. Early approaches relied on XGBoost ( 9 ) and MLPs ( 61 ) , which have since evolved to more expressive models such as LSTMs ( 5 ) and Transformers ( 37 ) . However, LSTMs suffer from limited parallelism and slow convergence, while Transformers, despite their ability to capture long-range dependencies, impose high computational and memory overhead when applied to long sequences of schedule tokens. These architectural limitations prompted us to design a cost model that can efficiently capture sequence dependencies while maintaining a lightweight approach. Third, while transfer learning has emerged as a popular technique for enabling cross-platform generalization ( 61 ; 59 ) , most existing methods are restricted to one-to-one adaptation: knowledge is transferred from a single source hardware to a single target hardware. This restricts their ability to accumulate generalized scheduling knowledge across diverse platforms. Recent work such as MTL-TLP ( 55 ) attempts to overcome this limitation using multi-task learning to jointly learn from multiple source platforms. While promising, this approach introduces significant scalability issues: the number of trainable parameters and associated computational cost grow proportionally with the number of hardware targets, and the framework requires simultaneous access to data from all source platforms during training, limiting practicality. These limitations have prompted us to develop more flexible multi-platform learning frameworks to efficiently utilize knowledge from multiple source platforms without introducing excessive computational costs. To address the challenges of existing DL compilers across hardware platforms, namely the high cost of data collection, inefficiencies in cost model design, and suboptimal generalization across hardware, we propose TCL , a fast, data-efficient, and transferable tensor compiler framework. At the data level, it incorporates a principled active learning strategy, the RDU Sampler , which selects the most informative tensor programs by jointly considering representativeness, diversity, and uncertainty. This dramatically reduces the number of performance measurements required, achieving competitive or even superior tuning quality with as little as 10% of the data. To enhance prediction efficiency, TCL employs a new cost model architecture that is both lightweight and expressive, built upon

Read the complete text