Deep-Learning-Driven Prefetching for Far Memory
Yutong Huang, Zhiyuan Guo, Yiying Zhang · 2025
Read Deep-Learning-Driven Prefetching for Far Memory by Yutong Huang, Zhiyuan Guo, Yiying Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00384.txt.
Introduction
Yutong Huang, Zhiyuan Guo, and Yiying Zhang present FarSight, a system that fetches data ahead of need for far memory, in work from 2025. Far memory moves less used data to cheaper, remote tiers. Guessing which data will be needed next can hide the delay of fetching it.
Past learning based fetchers were tested mostly in simulation or on small hardware. FarSight splits program meaning from memory layout, so a trained model can forecast access over a compact word set. Those forecasts are mapped to real addresses at run time. Across four heavy workloads, FarSight ran much faster than the prior best system.
Catalog record
| CREATOR | Yutong Huang, Zhiyuan Guo, Yiying Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep-Learning-Driven Prefetching for Far Memory Yutong Huang, Zhiyuan Guo, Yiying Zhang Abstract . Modern software systems face increasing runtime performance demands, particularly in emerging architectures like far memory, where local-memory misses incur significant latency. While machine learning (ML) has proven effective in offline systems optimization, its application to high-frequency, runtime-level problems remains limited due to strict performance, generalization, and integration constraints. We present FarSight , a Linux-based far-memory system that leverages deep learning (DL) to efficiently perform accurate data prefetching. FarSight separates application semantics from runtime memory layout, allowing offline-trained DL models to predict access patterns using a compact vocabulary of ordinal possibilities, resolved at runtime through lightweight mapping structures. By combining asynchronous inference, lookahead prediction, and a cache-resident DL model, FarSight achieves high prediction accuracy with low runtime overhead. Our evaluation of FarSight on four data-intensive workloads shows that it outperforms the state-of-the-art far-memory system by up to 3.6 times. Overall, this work demonstrates the feasibility and advantages of applying modern ML techniques to complex, performance-critical software runtime problems. 1. Introduction Machine learning (ML) has recently demonstrated strong potential in addressing a variety of systems challenges. Prominent examples include optimizing operating system and virtual machine configurations ( yang2022stun ; cheng2024dynamic ; cano2018adares ) , predicting request arrival times for cluster-level resource management ( resource-central ; ml-datacenter ; peng2019dl2 ; Paragon ) , and estimating object hotness to improve data placement in heterogeneous memory ( ML-Object-HeteroMem ; lu2024hammerefficienthotcolddata ; IDT-RL-TieredMem ) . These successes typically rely on models that are either trained and evaluated offline or invoked infrequently at runtime—such as during object allocation. However, the performance of applications is directly influenced by frequent, fine-grained events like memory accesses and instruction execution. So far, ML-based solutions for such high-frequency events are largely confined to the micro-architectural level. Examples include CPU branch prediction ( TinyBERT-BP ; tarsa2019-bp ) , instruction prefetching ( pierce1994wrongpath ; ramesh2024pdip ; protean2023 ) , and cache line prefetching ( duong2024twilight ; zhang2023dart ; voyager ) ). Unlike hardware-level ML techniques that operate on small, consistent inputs with tight latency and resource constraints, applying ML at the software systems runtime layer introduces distinct challenges and opportunities. At the systems level, application behavior can vary widely by input data, application configurations, and environments, and application performance is impacted by longer-term history. Meanwhile, system software has more flexible access to idle CPU cores and memory space, potentially allowing for more resource-demanding ML techniques. This discrepancy raises a fundamental research question: How to properly leverage ML techniques to solve complex, performance-critical runtime problems at the systems layer? To answer this question, we explore a representative and challenging problem: ML-based far memory prefetching in the Linux kernel. Far-memory systems—where most of the data resides in remote memory and only a small subset is cached locally—are gaining traction for reducing overall memory costs through the use of cheaper, network-attached memory and memory sharing across nodes. In far-memory systems, application performance is highly sensitive to the latency of fetching data on demand from remote memory, as a far-memory access over RDMA takes more than 20 times of a local DRAM access. To make far memory viable, the on-demand data fetching overhead must be mitigated. Today’s predominant approach is to improve far-memory communication bandwidth and latency with interconnect technologies like CXL ( cxl_spec2023 ; chen2023pond ; wang2023dmemos ; subramaniam2022cxl ; chen2024nextgen ) . However, these approaches require substantial hardware upgrades and ecosystem adoption, slowing their deployment. An alternative and orthogonal strategy is to reduce the frequency of on-demand fetches through software prefetching. Today’s far-memory systems employ rule-based prefetchers ( e.g. , sequential or strided access patterns ( mira ; almaruf2020leap ) ), which perform well for regular memory access patterns but fail in the face of complex access behaviors influenced by non-trivial interactions between application phases, control flow, data locality, and cache hierarchy. Many data-center workloads including graph processing ( PageRank ; han2024graph ) , tree and index structures ( guttman1984r ; gusfield1997algorithms ) , pointer chasing ( hsieh2017implementing ) , and recursive data structures ( harold2004xml ) exhibit memory access patterns that defy rule-based prefetching. If these access patterns could be learned and predicted accurately, far-memory systems could proactively fetch data and mitigate the performance penalties associated with remote access, even in the absence of new hardware. Figure 1. FarSight Achieving Three Key Goals Together. FarSight, FastSwap ( fastswap ) , and Hermit ( hermit ) are far-memory systems that run in the Linux kernel. Voyager ( voyager ) , Hashemi etal. ( pmlr-v80-hashemi18a ) , and Twilight ( duong2024twilight ) are micro-architecture CPU cache prefetchers implemented in simulation or with offline traces. Modern deep-learning (DL) techniques such as LSTM ( hochreiter1997long ) , RNN ( socher2011parsing ) , and Transformer ( vaswani2017attention ) have demonstrated the ability to model complex, long-range, and nonlinear patterns that traditional heuristics cannot capture. However, incorporating such models in performance-critical system settings introduces a unique set of challenges. To ensure good application performance, ML techniques must impose minimal overhead. This makes online training impractical, as it introduces significant memory and computational costs. Offline training, while avoiding these runtime expenses, raises a different challenge: ensuring the trained model remains effective on new or unseen inputs. At runtime, predictions must be both accurate and lightweight–—incorrect predictions lead to prefetching irrelevant data, which wastes local memory and degrades application performance. Although larger models and longer input histories can enhance prediction accuracy, they also increase the cost of access tracking, inference latency, and memory usage due to the size of model weights. Ultimately, reconciling the competing goals of high accuracy, low overhead, and strong generalization remains a fundamental and difficult challenge (Figure 1 ). To confront this challenge and demonstrate the benefits of ML techniques for performance-critical systems problems, we build FarSight , a Linux-based far-memory system that incorporates modern DL techniques for effective data prefetching. Our key insight is that memory access behavior is governed by both application semantics ( e.g. , algorithmic logic) and input-dependent runtime context (e.g., memory layout). The semantics tend to generalize across inputs and can be learned offline, but the actual memory addresses are input-specific and best handled at runtime. FarSight exploits this separation by training a DL model to learn semantic patterns and delegating address resolution to a runtime system component. Specifically, we propose to represent application semantics as relationships between memory accesses. For each access, we observe that the subsequent access usually only has a small set of possibilities. We assign each possible outcome an ordinal identity. While the actual memory addresses corresponding to these possibilities vary across different inputs, the transition pattern— i.e. , which ordinal is likely to follow given the history–—is often learnable and generalizable. For instance, in a linked list traversal, each accessed node is always followed by the next node in the list. Although the concrete addresses of these nodes differ per execution, the access behavior remains the same. Similarly, in the PageRank algorithm, a page’s rank update depends on the ranks of linking pages. The specific memory addresses of those linked pages depend on the input graph, but their identities can be precisely captured at runtime and predicted using ordinal labels. To formalize the above idea, we