HERMES: High-Performance RISC-V Memory Hierarchy for ML Workloads

Pranav Suryadevara · 2025

Read HERMES: High-Performance RISC-V Memory Hierarchy for ML Workloads by Pranav Suryadevara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13064.txt.

Introduction

Suryadevara designs HERMES in twenty twenty-five, a memory system for an open chip design used in machine learning. Learning models need data fed fast, and slow memory can starve them. HERMES tunes the layers of memory in RISC-V chips for image, sequence, and attention models. It adds smart pre loading, caches that know tensor shapes, and mixed memory types.

A shared top level cache links to a special learning accelerator chip. The team tested the design in two standard simulators under typical learning loads. The report lays out the main design choices and the hard parts that remain. It aims at memory that stays fast as models and chips scale up.

Catalog record

CREATORPranav Suryadevara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HERMES: High-Performance RISC-V Memory Hierarchy for ML Workloads Pranav Suryadevara Abstract The growth of machine learning (ML) workloads has underscored the importance of efficient memory hierarchies to address bandwidth, latency, and scalability challenges. HERMES focuses on optimizing memory subsystems for RISC-V architectures to meet the computational needs of ML models such as CNNs, RNNs, and Transformers. This project explores state-of-the-art techniques such as advanced prefetching, tensor-aware caching, and hybrid memory models. The cornerstone of HERMES is the integration of shared L3 caches with fine-grained coherence protocols equipped with specialized pathways to deep-learning accelerators such as Gemmini. Simulation tools like Gem5 and DRAMSim2 were used to evaluate baseline performance and scalability under representative ML workloads. The findings of this study highlight the design choices, and the anticipated challenges, paving the way for low-latency scalable memory operations for ML applications. I Introduction Machine learning (ML) models have become integral to modern computing, powering applications across fields such as Computer Vision, Natural Language Processing, and Autonomous Systems. With these advancements, models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformers, Large Language Models (LLMs), and Large Vision-Language Models (LVLMs) often require extensive computational resources and memory bandwidth. Hence as a result optimizing memory hierarchies for ML workloads is of significant importance for enhancing performance and efficiency [ 1 , 2 ] . RISC-V, an open-source Instruction Set Architecture (ISA) initially developed at the University of California, Berkeley, is gaining traction due to its flexibility and extensibility [ 3 , 4 ] . However, the performance of RISC-V processors for ML workloads is currently limited by memory bottlenecks, including latency, limited bandwidth, and lack of scalability [ 6 ] . The HERMES project aims to design a high-performance memory subsystem that is tailored to optimize ML workloads on RISC-V architectures. Using techniques such as advanced prefetching [ 8 ] , tensor-aware caching [ 9 ] , and hybrid memory models [ 7 ] , HERMES seeks to overcome these challenges and enhance the efficiency of ML computations. This report presents a foundational study of HERMES, outlining the design considerations involved, the initial simulation results obtained, and the challenges anticipated to achieve a scalable, low-latency memory hierarchy system for ML workloads. The project evaluates the performance of the system using simulation tools such as Gem5 [ 10 ] and DRAWSim2 [ 11 ] . II Literature Survey II-A Memory Bottlenecks in ML Workloads ML workloads, especially the deep-learning models, are characterized by large datasets and frequent memory access patterns. Studies have identified that the key challenges that impact the efficiency of memory subsystems are: High Latency Tensor operations, such as matrix multiplication, require frequent access to large datasets stored in memory. The latency factor involves fetching in data from DRAM which can significantly degrade performance if it isn’t mitigated by efficient caching and prefetching mechanisms [ 1 , 2 , 5 ] . Bandwidth Constraints ML models, particularly those involving convolutions and recurrent operations, demand a high memory bandwidth. The concurrent fetching of weights, activations, and gradients often saturates the available bandwidth, leading to bottlenecks in the data transfer. This issue is exacerbated in systems with limited memory available on the chip [ 6 , 7 , 12 ] . Scalability As ML models grow in size and complexity, memory subsystems must scale to accommodate the increasing data loads. Current architectures struggle to maintain performance as the size of the dataset exceeds the cache capacity, thus requiring frequent access to the slower off-chip memory [ 10 , 15 ] . Addressing these bottlenecks requires innovative memory hierarchies that can balance latency, bandwidth, and scalability for a variety of ML workloads. II-B Existing Memory Solutions for ML Various techniques have been developed to optimize memory systems for ML workloads. These solutions address specific aspects of memory bottlenecks and have been implemented in modern hardware architectures. Tensor-Aware Caching Tensor-aware caching improves data reuse by optimizing the cache layout for tensor operations. For example, Google’s Tensor Processing Unit (TPU) uses specialized caching mechanisms that align with the structure of tensor computations, reducing latency and increasing throughput [ 1 ] . This approach ensures that frequently accessed tensors remain in fast on-chip memory, minimizing DRAM accesses. Prefetching Strategies Advanced prefetching techniques predict data access patterns and load data into the cache before it is needed. Techniques such as Stride Prefetching and Machine Learning-Based Prefetching have been proven effective in reducing latency for sequential and non-sequential data access patterns [ 8 , 5 ] . For instance, Intel’s microprocessors incorporate sophisticated hardware prefetchers that dynamically adapt to the workload behavior at hand [ 9 , 12 ] . Hybrid Memory Models Combining DRAM with High-Bandwidth Memory (HBM) offers a balance between capacity and speed. HBM provides significantly higher bandwidth compared to the traditional DRAM, making it suitable for ML workloads with high data transfer requirements [ 7 , 16 ] . For example, NVIDIA’s GPUs use HBM to deliver the necessary bandwidth for training large neural networks [ 13 ] . These techniques have shown promise in improving memory performance for ML workloads, but their integration in a RISC-V architecture remains limited. II-C RISC-V-Based Accelerators The open-source nature of RISC-V has led to the development of several accelerators designed for ML workloads. These accelerators benefit from custom memory pathways and specialized hardware support for tensor computations. Gemmini Gemmini is a configurable deep-learning accelerator for RISC-V systems. It provides efficient support for matrix multiplication and convolution operations, which are fundamental to ML workloads [ 6 , 14 ] . The Gemmini design allows integration with custom memory pathways, facilitating efficient data transfer between caches and the accelerator. Simulation Tools Tools like Gem5 and DRAMSim2 are widely used to evaluate memory hierarchies and system performance. Gem5 offers a flexible platform for modeling CPU and memory interactions [ 10 , 18 ] , while DRAMSim2 provides detailed simulations of DRAM behavior [ 11 , 17 ] . These tools are essential for testing and validating new memory subsystem designs in RISC-V architectures. II-D Research Gaps While existing solutions address specific aspects of memory optimization, there is limited research on integrating these techniques into a cohesive memory subsystem for RISC-V architectures specifically targeting ML workloads. Most current implementations focus on proprietary architectures, such as TPUs and GPUs, leaving a gap in the open-source RISC-V ecosystem [ 15 , 13 ] . HERMES: A Unified Memory Solution HERMES aims to fill this gap by: 1. Combining shared L3 caches, hybrid memory models, tensor-aware caching, and advanced prefetching into a unified memory hierarchy. 2. Optimizing memory pathways for Gemmini and similar RISC-V accelerators [ 6 ] . 3. Providing a scalable and efficient solution for deep-learning models on RISC-V platforms. III Problem Statement Current RISC-V memory hierarchies face significant challenges in meeting the performance requirements of ML workloads due to several interrelated issues: 1. High Latency : Accessing large datasets stored in off-chip memory introduces delays that degrade system performance. Tensor operations, such as matrix multiplications and convolutions, require frequent memory accesses, and high latency in these operations can significantly reduce overall computation speed. 2. Limited Bandwidth : ML workloads involve the simultaneous transfer of large volumes of data, including weights, activations, and gradients.

Read the complete text