Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches
Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger · 2025
Read Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches by Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03676.txt.
Introduction
Four authors tackle a memory problem in chips built to run neural networks. Such chips use layered, software-managed memory, and the standard trick of tiling splits each network layer into pieces that fit. Tiling shrinks memory use but forces extra trips to slow off-chip memory, which wastes energy and creates bottlenecks. Their method, Fused-Tiled Layers, instead fuses tiled pieces of neighboring layers so data moves less.
The team built the method into an open-source deployment framework tuned for a RISC-V chip with mixed processor types. On a typical stage of a vision transformer, the fused approach cut runtime by up to 60.1 percent. Data movement fell by 47.1 percent across off-chip transfers and on-chip shuffling combined. The authors present the method as a general fix for accelerators with managed caches.
Catalog record
| CREATOR | Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 5 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger Abstract \footertext RISC-V Summit Europe, Paris, 12-15th May 2025 1 Introduction W ith the increased interest in efficient execution of Deep Neural Networks (DNNs) at the edge, hardware architectures and DNN compilers have been proposed. While ARM-based architectures have a high level of maturity in both hardware and software ecosystems that allows for optimized execution of modern DNN models, the RISC-V ecosystem has proliferated in terms of novel hardware architectures, but there is room for improvement in terms of software support. Specifically, while popular DNN compilers such as CubeAI 1 1 1 https://stm32ai.st.com/stm32-cube-ai/ , TVM [ 1 ] , or IREE 2 2 2 https://github.com/iree-org/iree present streamlined support for ARM-based System-on-chips (SoCs) , important features of modern RISC-V SoCs , such as multi-level memory hierarchy, are not considered. In particular, layer fusion is a well-known technique performed by DNN compilers to avoid materializing intermediate tensors that increase memory footprint and bandwidth. In RISC-V-based platforms with multi-level memory hierarchy, it is particularly critical to avoid materializing huge intermediate activations in Last Level Cache (LLC) memory. Therefore, this paper presents a new flexible algorithm tailored to RISC-V SoCs with a multi-level memory hierarchy, Fused-Tiled Layers (FTL) , to minimize memory transactions by fusing a series of layers: 1. Fused-Tiled Layers (FTL) formulates the tiling of each DNN layer as a constraint optimization problem, where each output tensor dimension is linked to input tensor dimensions via a linear transformation, allowing us to merge several layers to generate valid layer fusion solutions for any layer combination. By doing so, we minimize transfers from L2 memory to LLC . 2. We benchmark FTL with a Vision Transformer (ViT) ’s Multi-Layer Perceptron (MLP) on a reduced version of the heterogeneous RISC-V SoC Siracusa [ 2 ] , with, and without the Neural Processing Unit (NPU) . Compared to the layer-per-layer tiling strategy, we demonstrate a runtime reduction of 28.8 % when only using the 8-cores cluster of RISC-V cores and 60.1 % when using the cluster and the NPU . Figure 1: Overview of the FTL on a GEMM and GeLU layer. 2 Methodology We integrate FTL into Deeploy 3 3 3 https://github.com/pulp-platform/Deeploy , an open-source bottom-up DNN deployment framework that generates optimized bare-metal C requiring minimal runtime support. We rely on kernels using the extended RV32IMCF-XpulpV2 Instruction Set Architecture (ISA) featuring hardware loops, post-increment load-store, and Single Instruction Multiple Data (SIMD) instructions. To move tiles of tensors across the memory hierarchy, we use Direct Memory Access (DMA) engines that rely on the flexibility of RISC-V systems to perform 3D transfers. Fig 1 provides a visual representation of the different steps (numbered from 1 to 4) of FTL . In step 1 , we attribute a variable for each tensor dimension related to the given operator. Then, we formulate the constraints for the tiling of the single operator in step 2 . There are three kinds of constraints: the geometrical constraints describe the data dependency between the dimensions of the output and input tensors. The kernel policy constraints are specific to the kernel’s dataflow; Finally, we add flexible performance constraints to boost the hardware utilization. In step 3 , we select the consecutive layers to fuse and bind the variable of their shared tensors dimension. Finally, in step 4 , we solve the constraint optimization problem representing the fused layers with Deeploy’s tiling and memory allocation solver. Figure 2: Overview of the modified Siracusa SoC . 3 Results 3.1 Evaluation Setup We perform our benchmark on a reduced version of the RISC-V Siracusa [ 2 ] SoC ; its architecture is described in Fig 2 . The 8 RISC-V cores are using the RV32IMCF-XPulpV2 ISA tailored to Digital Signal Processing (DSP) tasks, and the NPU is targeting GEMM and convolution. We use the GVSoC 4 4 4 https://github.com/gvsoc event-based simulator to measure the runtime, which provides fast and accurate simulation with an error typically below 10 %. 3.2 ViT ’s MLP benchmark To showcase the benefits of FTL , we benchmark a GEMM followed by a GeLU activation function. These layers are commonly found in the MLP stage of ViTs [ 3 ] . Fig 3 reports the MLP ’s runtime with and without FTL when using only the RISC-V cluster (left side) and using the cluster and the NPU (right side). There are two reasons to explain such runtime reduction when using FTL . First, FTL reduces the number of DMA transfers by 47.1 % by preventing the materialization of the MLP ’s intermediate tensor. Second, the L2 memory capacity is exceeded when materializing the MLP ’s intermediate tensor; hence, this tensor is stored in L3 Random-Access Memory (RAM) . With FTL , we don’t need to perform costly off-chip memory transfers to bring back the intermediate tensor from L3 to L1, leading to a reduction of the runtime. If double-buffering is used, FTL speeds up execution only if the kernel runtime is less than the DMA ’s runtime. As reported in Fig 3 , this is the case when using the cluster and the NPU . Figure 3: Runtime comparison of ViT ’s MLP using layer-per-layer tiling (baseline) and FTL on the Siracusa SoC