GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research

Xinqi Li Yiqun Liu Shan Jiang Enrong Zheng Huaijin Zheng Wenhao Dai Haodong Deng Dianhai Yu Yanjun Ma · 2025

Read GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research by Xinqi Li Yiqun Liu Shan Jiang Enrong Zheng Huaijin Zheng Wenhao Dai Haodong Deng Dianhai Yu Yanjun Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24035.txt.

Introduction

Xinqi Li and eight co-authors release GraphNet, a large set of real deep-learning computation graphs meant to sharpen tensor compiler research. Compilers such as TorchInductor decide how a network actually runs on hardware, but judging them takes realistic workloads. GraphNet provides 2,700 graphs with rich metadata, spanning six task types across several frameworks.

To score compilers on the corpus, the authors define a Speedup Score that weighs runtime gains and correct execution together, under adjustable tolerance. An error-aware extension folds in numerical error, helping developers spot where bottlenecks hide. As a demo, they benchmark the default compilers of PaddlePaddle and PyTorch on vision and language samples. The full pipeline — graph extraction plus scoring tools — is public. The authors intend GraphNet as shared infrastructure: one realistic testbed, so compiler gains can be compared honestly instead of on cherry-picked models.

Catalog record

CREATORXinqi Li Yiqun Liu Shan Jiang Enrong Zheng Huaijin Zheng Wenhao Dai Haodong Deng Dianhai Yu Yanjun Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24035

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research Xinqi Li Yiqun Liu Shan Jiang Enrong Zheng Huaijin Zheng Wenhao Dai Haodong Deng Dianhai Yu Yanjun Ma Abstract Abstract We introduce GraphNet, a dataset of 2.7K real-world deep learning computational graphs with rich metadata, spanning six major task categories across multiple deep learning frameworks. To evaluate tensor compiler performance on these samples, we propose the benchmark metric Speedup Score S_{t} , which jointly considers runtime speedup and execution correctness under tunable tolerance levels, offering a reliable measure of general optimization capability. Furthermore, we extend S_{t} to the Error-aware Speedup Score ES_{t} , which incorporates error information and helps compiler developers identify key performance bottlenecks. In this report, we benchmark the default tensor compilers, CINN for PaddlePaddle and TorchInductor for PyTorch, on computer vision (CV) and natural language processing (NLP) samples to demonstrate the practicality of GraphNet. The full construction pipeline with graph extraction and compiler evaluation tools is available at https://github.com/PaddlePaddle/GraphNet . 1 Introduction The development of high-performance GPU kernels has become critical for computational efficiency in modern deep learning workloads. A widely adopted approach integrates deep learning frameworks, such as PaddlePaddle [ 8 ] and PyTorch [ 14 ] , with vendor-specific operator libraries (e.g., cuDNN [ 2 ] , oneDNN [ 12 ] ). However, emerging hardware featuring low-precision formats (e.g., BF16 [ 19 ] , FP8 [ 10 ] ) and advanced memory access patterns is creating a growing demand for custom operators-a demand that vendor-specific libraries often struggle to accommodate. To meet these demands, modern deep learning systems increasingly rely on tensor compilers (e.g., TVM [ 1 ] , XLA [ 6 ] , BladeDISC [ 21 ] ) to lower high-level computational graphs into efficient backend implementations for heterogeneous hardware platforms. Despite these advances, deep learning engineers still rely heavily on manual tuning, as existing tensor compilation frameworks often lack fine-grained hardware control and offer limited support for cross-platform complex algorithms. Meanwhile, researchers are increasingly exploring the use of large language models (LLMs) [ 13 ] and AI coding agents [ 18 ] to automatically generate efficient operators and kernels. In this challenging context, systematic evaluation of existing tensor compilers is essential to identify performance bottlenecks and guide the evolution of next-generation compilers. Existing benchmarks are often ad hoc, relying on a small set of hand-picked samples, with limited coverage of up-to-date, real-world models from the community and lacking sufficient support for cross-framework evaluation. To address these limitations, we introduce GraphNet , a large-scale dataset of deep learning computational graphs designed to enable systematic evaluation of tensor compilers across tasks and frameworks. We further propose a unified metric to benchmark compiler performance by jointly accounting for runtime speedup, numerical correctness, and compilation failures. Our main contributions are as follows: • We collected over 2.7k computational graphs from real-world models, covering diverse task categories and mainstream frameworks (e.g., PaddlePaddle, PyTorch). All samples are stored in a unified format and are compatible with multiple tensor compiler backends. • We evaluate the performance of CINN [ 17 ] and TorchInductor on CV and NLP samples from GraphNet, and propose the Speedup Score ( S_{t} ) to measure the general optimization capability of tensor compilers, as well as the Error-aware Speedup Score ( ES_{t} ), which accounts for all types of execution failures. • We present a detailed study of the dataset construction pipeline and sample constraints, and release the GraphNet dataset along with open-source extraction and evaluation tools. The rest of this paper is organized as follows. Section 2 introduces dataset properties and its distribution. Section 3 presents the design of our evaluation metric and discusses experimental results. Section 4 describes the dataset construction methodology. Sections 5 and 6 review related works and outline future directions. 2 Dataset Properties Authenticity: All samples are extracted from models in mainstream deep learning frameworks, focusing on PaddlePaddle libraries (e.g., PaddleNLP, PaddleX, PaddleScience) and PyTorch libraries (e.g., TorchVision, timm, mmseg). These libraries are actively maintained and widely adopted by the community, ensuring that GraphNet reflects real-world workloads rather than synthetic benchmarks. Compatibility: GraphNet samples adopt a standardized format that integrates computational graphs with inputs, weights, and custom operators. This format preserves complete computation semantics without information loss, ensuring compatibility with diverse compiler backends, including CINN (PaddlePaddle), TorchInductor (PyTorch), XLA (TensorFlow), TVM, BladeDISC, and TensorRT. This enables fair and reproducible evaluation, as well as seamless extensions to new compilers. Diversity: The dataset spans six major task categories: computer vision, natural language processing, audio, multimodal learning, scientific computing, and others. Models vary in scale from a few thousand to 10B parameters, with the most complex sample containing up to 3,686 operators. GraphNet also supports multiple data types, such as BF16, FP16, and FP32, which are essential for evaluating compiler optimizations across different numerical precisions and hardware backends. (a) Categories of Computational Graph (b) Operator Numbers of CV Models (c) Operator Numbers of NLP Models Figure 1 : Statistical properties of the GraphNet dataset. (a) The distribution of computational graphs across six major task categories, showing that Computer Vision (47.8%) and Natural Language Processing (39.5%) are the dominant domains. (b) and (c) Histograms showing the distribution of operator counts (on a log 2 scale) for CV and NLP models, respectively. Both categories show a high concentration of graphs with operator counts around 2^{9} (512). 3 Benchmark of Tensor Compilers 3.1 Compiler Evaluation We introduce an automated compiler evaluation workflow that enables unified and repeatable testing of multiple tensor compilers. The evaluation takes GraphNet samples as input and consists of the following key steps: 1. Baseline Execution: Execute the original model in Eager mode on a given framework and record its output and execution time T_{eager} as the baseline for subsequent comparisons. 2. Compiler Configuration: Compile the original model into an optimized executable by specifying the target compiler. Currently supported compilers include CINN, TorchInductor, XLA, TVM, BladeDISC, and TensorRT. 3. Compiled Execution: Execute the compiled model after a warmup phase to eliminate cold-start overhead and obtain the pure execution time T_{compiled} . Record its output to verify accuracy. 4. Performance Analysis: Compare the outputs collected from baseline and compiled execution to validate correctness. Quantify compiler performance by considering speedup ratio T_{eager}/T_{compiled} , execution correctness, and performance degradation. The details of the evaluation metrics are discussed in Section 3.2 . 3.2 Evaluation Metrics We design two distinct metrics: one for benchmarking tensor compilers and one for compiler development. The benchmarking metric, detailed in Section 3.2.1 , focuses primarily on acceleration and treats the error penalty as a fixed constant. The development metric (Section 3.2.2 ), exposes detailed error penalty information, allowing developers to configure it on demand. 3.2.1 Metrics for Compiler Benchmark The performance evaluation metric is parameterized by a tunable tolerance t , which enables the metric to capture compiler performance under varying correctness criteria, from strict numerical precision to more relaxed settings. Tolerance t determines whether a sample is considered correctly executed, based on the check assert_close(x, y, rtol(t), atol(t)) . The values of relative tolerance (rtol) and absolute tolerance (atol) corresponding to different t settings and data types are summarized in Appendix A . Based on tolerance t and its associated correctness criteria,

Read the complete text