Hardware-Aware Reformulation of CNNs for Efficient Execution on Specialized AI Hardware: A Case Study on NVIDIA Tensor Cores

Ganesh Bikshandi · 2026

Read Hardware-Aware Reformulation of CNNs for Efficient Execution on Specialized AI Hardware: A Case Study on NVIDIA Tensor Cores by Ganesh Bikshandi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11608.txt.

Introduction

Ganesh Bikshandi reshapes CNNs to fit specialized AI hardware. The 2026 preprint uses NVIDIA Tensor Cores as the case study. Convolutional neural networks sit at the center of modern AI. Their speed is often capped by hardware constraints, not math.

Tensor Cores demand set shapes for fast execution paths. Input channels must be multiples of 8, for instance. Some paths want multiples of 512 to run at full pace. CPU frameworks like oneDNN impose like rules for blocked formats. Bikshandi reformulates networks to meet those hardware rules. The reform keeps the model's task while easing its layout. Hardware-aware forms can unlock speed that naive forms waste.

Catalog record

CREATORGanesh Bikshandi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hardware-Aware Reformulation of CNNs for Efficient Execution on Specialized AI Hardware: A Case Study on NVIDIA Tensor Cores Ganesh Bikshandi Abstract Convolutional Neural Networks (CNNs) are central to modern AI, but their performance is often limited by hardware constraints. NVIDIA Tensor Cores, for instance, require input channels to be multiples of 8 and sometimes 512 for efficient execution. oneDNN framework for CPU imposes such a requirement for the blocked format. Traditional approaches address such alignment issue using zero-padding, which can be inefficient. In this work, we present a first-step, hardware-aware reformulation of CNN computations using rewrite rules, restructuring the underlying math to satisfy hardware alignment entirely post-training without modifying network weights. While our current implementation focuses on a single transformation for Tensor Cores, this approach is generalizable, laying the foundation to explore additional transformations for CPU and accelerators. This study represents an initial step toward semantic tuning , a systematic, hardware-aware optimization strategy for efficient deployment of CNN models on specialized AI hardware. 1 Introduction Convolutional Neural Networks (CNNs) are a foundational building block of modern deep learning systems, underpinning applications ranging from computer vision to speech and scientific computing. At a high level, CNNs apply learnable filters to structured input tensors in order to extract hierarchical feature representations. For a standard convolution, the input tensor has shape (N,C_{\text{in}},H[,W,D]) , the filter weights have shape (C_{\text{out}},C_{\text{in}},K_{1}[,K_{2},\ldots,K_{n}]) , and the resulting output tensor has shape (N,C_{\text{out}},H_{\text{out}}[,W_{\text{out}},D_{\text{out}])} . This formulation uses the NCHW layout; alternate layout is NHWC. While this mathematical formulation is hardware-agnostic, the efficient execution of CNNs on modern AI accelerators is strongly influenced by hardware-specific constraints. Specialized units such as NVIDIA Tensor Cores and analogous matrix-multiply engines in other accelerators impose alignment and tiling requirements on tensor dimensions, most notably that certain dimensions (e.g., channel counts) be multiples of fixed factors such as eight or a higher count like 512 ( 8 ) . Also, oneDNN framework ( 10 ) for CPU imposes such an alignment requirement for the blocked format. When these constraints are not satisfied, libraries either fall back to less efficient execution paths or require auxiliary modifications such as zero padding, which introduce redundant computation and memory overhead. It is possible to address these constraints during network design or training, for example by manually increasing the channels or retraining models with 1\times 1 convolution. In contrast, this paper explores a complementary and largely underexplored direction: post-training reformulation of CNN computations. Rather than modifying the network architecture or retraining weights, we rewrite the underlying convolutional mathematics to produce an equivalent formulation that satisfies hardware alignment and channel count requirements by construction. Specifically, we present an initial hardware-aware transformation that reshapes and reinterprets the convolutional computation—through width folding for input in NHWC format (alternatively height folding for NCHW format) and structured filter expansion — so that the resulting tensors conform to accelerator alignment constraints while preserving exact numerical equivalence to the original CNN. This transformation eliminates the need for zero padding and does not alter the learned parameters or outputs of the network. Although the paper focuses on a single transformation motivated by NVIDIA Tensor Core constraints, the broader goal is to establish a foundation for a family of such rewrite rules that can be systematically derived and applied across different accelerator architectures. Viewed through this lens, CNN execution can be treated as a compilation problem, where mathematically equivalent reformulations are selected to best match the capabilities and constraints of the target hardware. This work represents a first step toward such a hardware-aware compilation framework for CNNs, demonstrating that non-trivial accelerator constraints can be addressed purely through post-training mathematical rewrites. Table 1: First-layer channels in popular CNN architectures Network Input Channels AlexNet 3 VGG16 3 ResNet-18 3 ResNet-50 3 GoogLeNet 3 MobileNetV2 3 To motivate the need for hardware-aware filter alignment, we survey several widely used CNN architectures and their first-layer channel dimensions. As shown in Table 1 , most models processing RGB images have an input channel count of 3, which is not a multiple of 8, limiting the efficiency on NVIDIA Tensor Cores. Furthermore, many models have a first-layer filter count below 512, which is the recommended number mentioned in NVIDIA ( 8 ) . In such cases, our method can also increase the channel dimensions, ensuring better alignment with hardware requirements while preserving network functionality. The paper makes the following contributions: The paper makes the following contributions: • Introduce semantic reformulation as a hardware-aware optimization paradigm. • Present width folding , a semantics-preserving transformation that increases effective channel dimensionality without padding. • Provide a formal correctness proof. • Show how the transformation fits naturally as an MLIR compiler pass. • Demonstrate generalization to GEMM via 1\times 1 convolutions. • Report up to 3\times speedups over cuDNN/TensorRT on NVIDIA A100. Input Feature Map \mathbf{x}\in\mathbb{R}^{H\times W\times C_{\rm in}} Filter Weights \mathbf{W}\in\mathbb{R}^{K\times C_{\rm in}} Hardware-Aware Rewrite Width \rightarrow Channel Folding (No Zero Padding) Folded Input \tilde{\mathbf{x}}\in\mathbb{R}^{H\times\frac{W}{F}\times(C_{\rm in}\cdot F)} Block-Diagonal Filter \tilde{\mathbf{W}}\in\mathbb{R}^{K\times(C_{\rm in}\cdot F)} Output \tilde{y}(t)=y(t) Tensor Core Constraint: Channels must be multiple of 8 Key Idea: Rewrite CNN math, not the data Figure 1: Hardware-aware reformulation of a CNN via width-to-channel folding. The rewrite preserves exact semantics while satisfying Tensor Core alignment constraints without zero padding or retraining. Input tensor H\times W,\;C_{in}=1 Width folding ( F=8 ) Folded input H\times(W/8) C_{in}^{\prime}=8 Channels stacked in depth Original filter K\times 1 Channel expansion Expanded filter Diagonal replication C_{in}^{\prime}=8 Figure 2: Semantic-preserving CNN reformulation via width folding. The input width is partitioned into F=8 interleaved slices and stacked along the channel dimension, increasing the effective number of input channels without altering spatial height. The original K\times 1 convolution kernel is replicated along the main diagonal of the expanded filter matrix, ensuring independent convolution of each folded slice. This post-training transformation preserves exact convolution semantics while aligning channel dimensions with accelerator constraints. Algorithm 1 Width-Folding Transformation for Convolution 0: Input tensor X\in\mathbb{R}^{B\times H\times W\times C_{\text{in}}} , Filter tensor W_{f}\in\mathbb{R}^{K_{H}\times K_{W}\times C_{\text{in}}\times C_{\text{out}}} , Bias b\in\mathbb{R}^{C_{\text{out}}} , Folding factor F 0: Transformed tensors (X_{f},W_{f}^{\prime},b^{\prime}) or fallback 1: if W\bmod F\neq 0 or C_{\text{in}}\neq 1 then 2: return (X,W_{f},b) {Fallback: width not divisible or unsupported channels} 3: end if 4: Define X_{f}\in\mathbb{R}^{B\times H\times(W/F)\times F} 5: for b\_idx=0 to B-1 do 6: for h=0 to H-1 do 7: for w^{\prime}=0 to (W/F)-1 do 8: for f=0 to F-1 do 9: X_{f}[b\_idx,h,w^{\prime},f]\leftarrow X[b\_idx,h,F\cdot w^{\prime}+f,0] 10: end for 11: end for 12: end for 13: end for 14: Define W_{f}^{\prime}\in\mathbb{R}^{K_{H}\times K_{W}\times F\times(F\cdot C_{\text{out}})} , initialize to zero 15: for f=0 to F-1 do 16: W_{f}^{\prime}[:,:,f,f\cdot C_{\text{out}}:(f+1)\cdot C_{\text{out}}]\leftarrow W_{f}[:,:,0,:] 17: end for 18: Define b^{\prime}\in\mathbb{R}^{F\cdot C_{\text{out}}} 19: for f=0 to F-1 do 20: b^{\prime}[f\cdot C_{\text{out}}:(f+1)\cdot C_{\text{out}}]\leftarrow b 21: end for 22: return (X_{f},W_{f}^{\prime},b^{\prime}) 2 Width Folding Transformation This section describes the width folding transformation, a post-training, semantics-preserving reformulation that increases the effective channel dimension to satisfy hardware alignment constraints while leaving the learned model parameters unchanged. Given an input tensor X\in\mathbb{R}^{H\times W\times C_{in}},\quad C_{in}=1, and a 1-D convolution kernel W_{f}\in\mathbb{R}^{K\times 1}, we fold the width dimension by a factor F such that W is divisible by F . The transformation produces an equivalent convolution with C_{in}^{\prime}=F,\quad W^{\prime}=W/F, while preserving exact convolution semantics. 2.1 Input Width Folding The width

Read the complete text