Morphling: Fast, Fused, and Flexible GNN Training at Scale

Anubhab Rupesh Nasre [email protected] [email protected] IIT Madras , India IIT Madras , India · 2025

Read Morphling: Fast, Fused, and Flexible GNN Training at Scale by Anubhab Rupesh Nasre [email protected] [email protected] IIT Madras , India IIT Madras , India (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01678.txt.

Introduction

Anubhab Rupesh Nasre and Rupesh Nasre speed up graph neural network training at scale. Such nets fuse two unlike workloads. Graph walks are irregular and memory-bound. Dense matrix work is regular and compute-bound. Popular tools favor ease of use over that split.

Their system, Morphling, fuses and schedules both parts with care. It aims for fast training on large graphs. Large graphs sit behind social, commerce, and biology tools. Faster training lets teams iterate on bigger data without a matching rise in cloud cost. Scale here means graphs with billions of edges, the size behind fraud rings and web graphs that small test setups cannot mimic.

Catalog record

CREATORAnubhab Rupesh Nasre [email protected] [email protected] IIT Madras , India IIT Madras , India
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Morphling: Fast, Fused, and Flexible GNN Training at Scale Anubhab Rupesh Nasre [email protected] [email protected] IIT Madras , India IIT Madras , India Abstract Graph Neural Networks (GNNs) present a fundamental hardware challenge by fusing irregular, memory-bound graph traversals with regular, compute-intensive dense matrix operations. While frameworks such as PyTorch Geometric (PyG) and Deep Graph Library (DGL) prioritize high-level usability, they fail to address these divergent execution characteristics. As a result, they rely on generic kernels that suffer from poor cache locality, excessive memory movement, and substantial intermediate allocations. To address these limitations, we present Morphling, a domain-specific code synthesizer designed to bridge this gap. Morphling compiles high-level GNN specifications into portable, backend-specialized implementations targeting OpenMP, CUDA, and MPI. It achieves this by instantiating a library of optimized, architecture-aware primitives tailored to each execution environment. Morphling also incorporates a runtime sparsity-aware execution engine that dynamically selects dense or sparse execution paths using input feature statistics, reducing unnecessary computation on zero-valued entries. We evaluate Morphling on eleven real-world datasets spanning diverse graph structures, feature dimensionalities, and sparsity regimes. The results show that Morphling improves per-epoch training throughput by an average of 20 × \times on CPUs, 19 × \times on GPUs, and 6 × \times in distributed settings over PyG and DGL, with peak speedups reaching 66 × \times . Morphling’s memory-efficient layouts further reduce peak memory consumption by up to 15 × \times , enabling large-scale GNN training on commodity hardware. These findings demonstrate that specialized, architecture-aware code synthesis provides an effective and scalable path toward high-performance GNN execution across diverse parallel and distributed platforms. I Introduction GNNs have become the standard for learning on graph-structured data, powering applications ranging from recommendation systems to scientific computing. Unlike traditional neural networks that treat data points in isolation, GNNs generate topology-aware representations by leveraging the underlying structure of the data. They achieve this through iterative message passing, where nodes update their embeddings by aggregating high-dimensional features from their local neighborhoods. Their effectiveness stems from integrating irregular, memory-bound graph traversals directly with regular, compute-intensive dense matrix operations. This sparse–dense computational hybrid poses unique systems challenges that are poorly addressed by existing general-purpose frameworks. To manage this complexity, practitioners typically turn to frameworks like PyTorch Geometric (PyG) [ 1 ] and Deep Graph Library (DGL) [ 2 ] , which prioritize high-level programmability and seamless integration with PyTorch’s autograd engine. However, because these frameworks represent node features as dense tensors, they are forced to materialize substantial intermediate data during message passing and to execute redundant operations on naturally sparse inputs [ 4 ] . These choices introduce excessive memory movement, poor cache locality, and costly intermediate allocations. These challenges intensify significantly on graphs with large feature dimensions or millions of edges. Moreover, PyG and DGL lack unified abstractions for efficient execution across heterogeneous hardware. While CPUs thrive on cache-aware tiling and SIMD vectorization, GPUs require warp-synchronous kernels with coalesced memory access, and distributed clusters necessitate rigorous communication–computation overlap. Since existing frameworks abstract away these platform-specific decisions, they face fundamental challenges in ensuring performance portability across architectures. This optimization gap places practitioners in a difficult bind: they must choose between the development speed of high-level APIs and the raw performance of hand-tuned execution. Opting for performance requires manually implementing specialized kernels for CPUs, GPUs, and distributed clusters; however, maintaining these distinct codebases is error-prone and becomes increasingly unsustainable as hardware architectures diversify. While specialized systems like GNNAdvisor [ 3 ] and DGCL [ 4 ] attempt to bridge this gap, they function as single-platform systems that lack cross-platform portability and leave the overheads of dense feature processing unresolved. Alternatively, graph DSLs such as StarPlat [ 12 ] , GraphIt [ 8 ] , and Ligra [ 9 ] generate high-performance code but are limited to classical graph analytics, lacking essential support for backpropagation, optimizer updates, or feature sparsity. Similarly, tensor compilers such as Halide [ 18 ] and TVM [ 19 ] provide powerful scheduling abstractions but treat GNNs as generic tensor programs. Because they lack domain-specific primitives, they cannot perform graph-aware optimizations across the full training loop. These gaps highlight the need for a domain-specific approach that captures GNN computation at a high level while enabling backend-specialized code generation. The solution requires a sparsity-aware, memory-efficient architecture that scales transparently across diverse backends. To bridge this gap, we present Morphling, a synthesis-oriented framework that generates backend-specialized C++ training code from compact high-level GNN specifications. Through synthesis, Morphling integrates architecture-aware kernels for CPUs, GPUs, and distributed systems, chooses between dense and sparse execution paths based on input feature statistics, and embeds distributed communication and partitioning logic directly into the generated code. This design enables execution plans that align closely with each backend’s memory hierarchy and parallelism model while preserving the simplicity of high-level GNN programming. The contributions of this paper are as follows: • A GNN-native domain-specific language built on the StarPlat DSL and extended with GNN-specific operators, supporting forward propagation, neighborhood aggregation, backpropagation, and optimizer updates while abstracting away backend-specific complexity. • A multi-backend code synthesis framework that generates optimized OpenMP, CUDA, and MPI implementations from a single high-level program, including cache-tiled CPU kernels, warp-synchronous GPU aggregation kernels, and communication-overlapped distributed execution. • A runtime sparsity-aware execution engine that automatically selects efficient dense or sparse feature-processing paths based on feature statistics, exploiting feature-level sparsity without changing model semantics. • A comprehensive experimental evaluation across eleven real-world datasets demonstrating significant throughput improvements over PyG and DGL on CPUs, GPUs, and distributed clusters, along with up to 15\times lower peak memory consumption. II Related Work GNN Frameworks. PyG [ 1 ] and DGL [ 2 ] dominate GNN development by offering extensive model libraries and seamless integration with PyTorch’s autograd engine. However, their reliance on the gather-scatter paradigm [ 4 ] limits opportunities for kernel fusion and data reuse. First, they treat node features as dense tensors, forcing the materialization of large intermediate structures and precluding optimizations for intrinsically sparse features (e.g., one-hot encodings). Second, they lack a unified intermediate representation preventing optimizations from generalizing across backends. Consequently, targeting CPUs, GPUs, and clusters necessitates managing disjoint optimization strategies, impeding true performance portability. Some of these limitations have motivated several systems that focus exclusively on GPU execution. Beyond general-purpose frameworks, some systems focus exclusively on GPU execution. These systems improve GNN throughput by employing custom GPU kernels with hardware-aware optimizations such as 2D partitioning and neighbor grouping, as demonstrated by GNNAdvisor [ 3 ] and DGCL [ 4 ] . However, like PyG and DGL, they assume dense feature tensors, which limits the applicability of their optimizations to workloads with intrinsic feature sparsity. While they achieve high single-device throughput, they are tightly coupled to GPU architectures and thus lack the flexibility to generate code for CPUs or MPI-based clusters. In the distributed setting, industrial systems like AliGraph [ 5 ] , Euler [ 6 ] , and DistDGL [ 7 ] target billion-scale graph training. However, their primary optimizations focus on distributed storage, I/O minimization, and sampling strategies rather than the computational efficiency of the core GNN computation. These systems emphasize scale and data movement rather than computation efficiency, whereas Morphling targets the optimization of the core GNN kernels

Read the complete text