ShardTensor: Domain Parallelism for Scientific Machine Learning

Corey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Mike Pritchard, Sanjay Choudhry · 2026

Read ShardTensor: Domain Parallelism for Scientific Machine Learning by Corey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Mike Pritchard, Sanjay Choudhry (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11111.txt.

Introduction

Corey Adams and six colleagues introduce ShardTensor, a domain-parallelism method for scientific machine learning at extreme resolution. Scientific models face massive spatial data sets, yet no general framework has parallelized below batch size one per device without bespoke tricks that fail to scale or degrade accuracy.

By splitting spatial size from hardware limits, ShardTensor scales input data to arbitrary sizes for high-fidelity training and inference. Tests show strong scaling cutting latency and weak scaling handling larger data, across several split directions. Scientific machine-learning teams get a general route past the device-memory wall for extreme-scale inputs.

Catalog record

CREATORCorey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Mike Pritchard, Sanjay Choudhry
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49184
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ShardTensor: Domain Parallelism for Scientific Machine Learning Corey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Mike Pritchard, Sanjay Choudhry Abstract Scientific Machine Learning (SciML) faces unique challenges for extreme-resolution data, with mitigations that often fail to scale or degrade the accuracy of trained models. While some specialized methods have achieved remarkable results in training models or performing inference on massive spatial datasets with bespoke techniques, there is no generalized framework for parallelization over input data below batch size one per device. In this work we introduce ShardTensor : a novel paradigm of domain parallelism that enables flexible scaling of input data to arbitrary sizes. By decoupling the spatial dimensionality of input data from hardware constraints, ShardTensor enables scientific machine learning workloads to reach new levels of high fidelity training and inference. We demonstrate both strong and weak scaling of workloads during training and inference, showing improved latency with strong scaling and demonstrating the capacity to process higher data sizes with weak scaling. Additionally, we demonstrate multiple dimensions of parallelization, removing barriers to SciML on extreme-scale inputs. [ ] I Introduction Scientific machine learning applications have become a vehicle for accelerated simulation, scientific discovery, and industrial design. Machine learning has found applications in an incredible breadth of domains: healthcare and medicine [ 21 , 62 ] , industrial design [ 43 , 59 , 28 ] , fluid dynamics [ 34 ] and aerodynamics [ 11 ] , weather and climate forecasting [ 35 , 8 ] , fundamental sciences [ 18 , 12 , 31 ] , and many, many more [ 26 , 65 , 29 ] . It is not an overstatement to say that machine learning methods are fundamentally changing scientific research, all the way from early development to end user and industrial applications. Scientific data has several attributes that make it especially challenging to use for both training and inference, leading to reduced adoption or degraded applications of these scientific ML models. First, the data in scientific models is typically of high spatial resolution , with scientists working with a “more is better” philosophy - and rightly so. Higher resolution imaging across a breadth of scientific domains often leads to breakthrough results, from the first ever images of a black hole [ 15 ] to achieving atomic-resolution protein structures in cryo-electron microscopy [ 67 ] , and mapping human cerebral cortexes at petavoxel scales [ 56 ] . In multi-decadal Earth System projection, climate-critical cloud-forming turbulent processes require tens of meters in space and seconds in time to satisfyingly resolve, which remains far beyond the computational capacity of even the most ambitious global simulation frameworks [ 52 , 60 , 48 , 54 ] . From a scientific perspective, high resolution data is an aspiration. But from a computational perspective, high resolution data is a challenge; and from a machine learning perspective, where GPU memory resources can quickly become a bottleneck, high resolution data is a major challenge. Further, scientific data suffer from a computational curse of dimensionality: doubling the length of text for a language model will increase the number of input tokens by approximately double; doubling the resolution of N dimensional data will increase the size of scientific data by 2^{N} . Scientific machine learning models rapidly encounter challenges in GPU memory management, especially for model training. Building machine learning tools and techniques that can train and run inference on models at the native resolution of scientific data is a challenge the High Performance Computing community is well-positioned to address. Our contribution in this paper is a framework for high-resolution SciML that provides the simplicity and accessibility expected of PyTorch and its ecosystem while enabling this native-resolution paradigm. In this paper, we will describe ShardTensor, an abstraction and extension to a PyTorch tensor that allows domain parallelism , defined here as parallelism across devices for the input data, below even batch size one. As an example, an input batch of 3D Tensors of shape [B, C, H, W, D] (B=batch, C=channels, H=height, W=width, D=depth) would be partitioned across the B axis in a standard “data parallel” distribution. In domain parallelism, we extend this to partition further: when the limit of B=1 is reached, and each GPU has a single image, further subdivision is possible along the spatial dimensions. The name “domain parallelism” is taken from the analogous techniques in classical solvers in Computational Fluid Dynamics, numerical methods and other fields, in which this type of domain decomposition has existed for decades [ 32 , 22 , 63 ] . In this paper, we proceed as follows. First, we will provide a simplified overview of the origins of GPU memory usage for scientific machine learning, especially as it relates to high resolution data. The goal here is to motivate why domain parallelism is an avenue worth pursuing. Next, we will provide a short overview of some common techniques for reducing memory usage in scientific machine learning, followed by a description of related works focused on parallelism techniques in machine learning. Finally, we will introduce ShardTensor , starting with the design principles and goals, differentiation from DTensor [ 71 ] , and expected use cases. We highlight some existing applications, performance results, and expected areas where it might be of use to users. ShardTensor is already available for use, open source, through the NVIDIA PhysicsNeMo framework [ 49 ] . II What Causes High GPU Memory Usage? To motivate our discussion of domain parallelism techniques, we begin with an overview of the dominant origins of GPU memory usage in most scientific machine learning workloads. Of course, with the diversity of scientific workloads, it is impossible to provide an exhaustive and prescriptive description of every source of memory usage; unique workloads that require second order optimizers or non-reverse-mode auto-differentiation techniques will not necessarily fit this categorization. For a standard machine learning workload, we categorize the dominant memory drivers into four broad categories, two of which are specific to training only. For simplicity we only discuss 32-bit numerical precision in this section, but a discussion of reduced precision is found in Section II-B . 1. Model Parameters (weights, biases, encodings, etc.) contribute substantial GPU memory usage in Large Language Models (LLMs) but typically only modest usage in scientific models - though that is a trend that is changing. Every parameter in 32-bit precision will require 4 bytes of GPU storage, meaning 1 million parameters requires approximately 1 MB of GPU storage. Model parameters require storage in both training and inference. 2. Active Data , or the transient working memory for a single operation - including input/output buffers and any temporary workspace the kernel requires - occupies GPU memory only for the duration of the currently executing computation. Naturally, this is necessary during both training and inference. 3. Optimizer States represent the gradients, moments, or other related tensors required to apply updates like Adam [ 33 ] , RMSProp [ 61 ] , and other extensions of stochastic gradient descent that store gradients and other additional information. Typically, in 32-bit precision, this is a multiplicative factor of the storage required for the model parameters: a factor of two or three is typical. 4. Intermediate Activations hold

Read the complete text