Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches

Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger · 2025

Read Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches by Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03676.txt.

Introduction

Four authors tackle a memory problem in chips built to run neural networks. Such chips use layered, software-managed memory, and the standard trick of tiling splits each network layer into pieces that fit. Tiling shrinks memory use but forces extra trips to slow off-chip memory, which wastes energy and creates bottlenecks. Their method, Fused-Tiled Layers, instead fuses tiled pieces of neighboring layers so data moves less.

The team built the method into an open-source deployment framework tuned for a RISC-V chip with mixed processor types. On a typical stage of a vision transformer, the fused approach cut runtime by up to 60.1 percent. Data movement fell by 47.1 percent across off-chip transfers and on-chip shuffling combined. The authors present the method as a general fix for accelerators with managed caches.

Catalog record

CREATORVictor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE5 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fused-Tiled Layers: Minimizing Data Movement on RISC-V SoCs with Software-Managed Caches
Victor J.B. Jung † \dagger, Alessio Burrello ‡ \ddagger, Francesco Conti * and Luca Benini † \dagger

Abstract




\footertext

RISC-V Summit Europe, Paris, 12-15th May 2025

1
Introduction

W
ith the increased interest in efficient execution of
Deep Neural Networks (DNNs)
at the edge, hardware architectures and
DNN
compilers have been proposed.
While ARM-based architectures have a high level of maturity in both hardware and software ecosystems that allows for optimized execution of modern
DNN
models, the RISC-V ecosystem has proliferated in terms of novel hardware architectures, but there is room for improvement in terms of software support.

Specifically, while popular
DNN
compilers such as CubeAI
1
1

1

https://stm32ai.st.com/stm32-cube-ai/
, TVM
[
1
]
, or IREE
2
2

2

https://github.com/iree-org/iree
present streamlined support for ARM-based
System-on-chips (SoCs)
, important features of modern RISC-V
SoCs
, such as multi-level memory hierarchy, are not considered.
In particular, layer fusion is a well-known technique performed by
DNN
compilers to avoid materializing intermediate tensors that increase memory footprint and bandwidth.
In RISC-V-based platforms with multi-level memory hierarchy, it is particularly critical to avoid materializing huge intermediate activations in
Last Level Cache (LLC)
memory.
Therefore, this paper presents a new flexible algorithm tailored to RISC-V
SoCs
with a multi-level memory hierarchy,
Fused-Tiled Layers (FTL)
, to minimize memory transactions by fusing a series of layers:

1.

Fused-Tiled Layers (FTL)
formulates the tiling of each
DNN
layer as a constraint optimization problem, where each output tensor dimension is linked to input tensor dimensions via a linear transformation, allowing us to merge several layers to generate valid layer fusion solutions for any layer combination. By doing so, we minimize transfers from L2 memory to
LLC
.

2.

We benchmark
FTL
with a
Vision Transformer (ViT)
’s
Multi-Layer Perceptron (MLP)
on a reduced version of the heterogeneous RISC-V
SoC
Siracusa
[
2
]
, with, and without the
Neural Processing Unit (NPU)
. Compared to the layer-per-layer tiling strategy, we demonstrate a runtime reduction of 28.8 % when only using the 8-cores cluster of RISC-V cores and 60.1 % when using the cluster and the
NPU
.

Figure 1:
Overview of the
FTL
on a
GEMM
and GeLU layer.

2
Methodology

We integrate
FTL
into Deeploy
3
3

3

https://github.com/pulp-platform/Deeploy
, an open-source bottom-up
DNN
deployment framework that generates optimized bare-metal C requiring minimal runtime support.
We rely on kernels using the extended
RV32IMCF-XpulpV2

Instruction Set Architecture (ISA)
featuring hardware loops, post-increment load-store, and
Single Instruction Multiple Data (SIMD)
instructions.
To move tiles of tensors across the memory hierarchy, we use
Direct Memory Access (DMA)
engines that rely on the flexibility of RISC-V systems to perform 3D transfers.
Fig
1
provides a visual representation of the different steps (numbered from 1 to 4) of
FTL
.
In step
1
, we attribute a variable for each tensor dimension related to the given operator.
Then, we formulate the constraints for the tiling of the single operator in step
2
. There are three kinds of constraints: the
geometrical constraints
describe the data dependency between the dimensions of the output and input tensors. The
kernel policy constraints
are specific to the kernel’s dataflow;
Finally, we add
flexible performance constraints
to boost the hardware utilization.
In step
3
, we select the consecutive layers to fuse and bind the variable of their shared tensors dimension.
Finally, in step
4
, we solve the constraint optimization problem representing the fused layers with Deeploy’s tiling and memory allocation solver.

Figure 2:
Overview of the modified Siracusa
SoC
.

3
Results

3.1
Evaluation Setup

We perform our benchmark on a reduced version of the RISC-V Siracusa
[
2
]

SoC
; its architecture is described in Fig
2
.
The 8 RISC-V cores are using the
RV32IMCF-XPulpV2

ISA
tailored to
Digital Signal Processing (DSP)
tasks, and the
NPU
is targeting
GEMM
and convolution.
We use the GVSoC
4
4

4

https://github.com/gvsoc
event-based simulator to measure the runtime, which provides fast and accurate simulation with an error typically below 10 %.

3.2
ViT
’s
MLP
benchmark

To showcase the benefits of
FTL
, we benchmark a
GEMM
followed by a GeLU activation function. These layers are commonly found in the
MLP
stage of
ViTs

[
3
]
.
Fig
3
reports the
MLP
’s runtime with and without
FTL
when using only the RISC-V cluster (left side) and using the cluster and the
NPU
(right side).
There are two reasons to explain such runtime reduction when using
FTL
. First,
FTL
reduces the number of
DMA
transfers by 47.1 % by preventing the materialization of the
MLP
’s intermediate tensor. Second, the L2 memory capacity is exceeded when materializing the
MLP
’s intermediate tensor; hence, this tensor is stored in L3
Random-Access Memory (RAM)
. With
FTL
, we don’t need to perform costly off-chip memory transfers to bring back the intermediate tensor from L3 to L1, leading to a reduction of the runtime.
If double-buffering is used,
FTL
speeds up execution only if the kernel runtime is less than the
DMA
’s runtime. As reported in Fig
3
, this is the case when using the cluster and the
NPU
.

Figure 3:
Runtime comparison of
ViT
’s
MLP
using layer-per-layer tiling (baseline) and
FTL
on the Siracusa
SoC

Read the complete text