INDIGO: Page Migration for Hardware Memory Disaggregation Across a Network DOI: XXXXXXX.XXXXXXX Conference: The 39th ACM International Conference on Supercomputing; June 9–11, 2025; Salt Lake City, USA ISBN: 000-0-0000-0000-0/00/00 #166
Archit Patke, Christian Pinto, Saurabh Jha, Haoran Qiu, Zbigniew Kalbarczyk, Ravishankar Iyer · 2025
Read INDIGO: Page Migration for Hardware Memory Disaggregation Across a Network DOI: XXXXXXX.XXXXXXX Conference: The 39th ACM International Conference on Supercomputing; June 9–11, 2025; Salt Lake City, USA ISBN: 000-0-0000-0000-0/00/00 #166 by Archit Patke, Christian Pinto, Saurabh Jha, Haoran Qiu, Zbigniew Kalbarczyk, Ravishankar Iyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18140.txt.
Introduction
Archit Patke and five colleagues present INDIGO, a page migration framework for hardware memory disaggregation. Disaggregation lets servers borrow remote memory over a network, creating large shared pools. The catch is contention: remote access can badly slow applications, so operating systems migrate hot pages closer to the processor.
Existing migration policies ignore that transfer costs vary with network congestion. INDIGO adds page telemetry and a learning-based method that adapts to network conditions. Implemented in the Linux kernel and tested on a real prototype, it improved application performance by fifty to seventy percent over leading policies. It also cut network traffic by up to half.
Catalog record
| CREATOR | Archit Patke, Christian Pinto, Saurabh Jha, Haoran Qiu, Zbigniew Kalbarczyk, Ravishankar Iyer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
INDIGO: Page Migration for Hardware Memory Disaggregation Across a Network DOI: XXXXXXX.XXXXXXX Conference: The 39th ACM International Conference on Supercomputing; June 9–11, 2025; Salt Lake City, USA ISBN: 000-0-0000-0000-0/00/00 #166 Archit Patke, Christian Pinto, Saurabh Jha, Haoran Qiu, Zbigniew Kalbarczyk, Ravishankar Iyer Abstract . Hardware memory disaggregation (HMD) is an emerging technology that enables access to remote memory, thereby creating expansive memory pools and reducing memory underutilization in datacenters. However, a significant challenge arises when accessing remote memory over a network: increased contention that can lead to severe application performance degradation. To reduce the performance penalty of using remote memory, the operating system uses page migration to promote frequently accessed pages closer to the processor. However, previously proposed page migration mechanisms do not achieve the best performance in HMD systems because of obliviousness to variable page transfer costs that occur due to network contention. To address these limitations, we present INDIGO: a network-aware page migration framework that uses novel page telemetry and a learning-based approach for network adaptation. We implemented INDIGO in the Linux kernel and evaluated it with common cloud and HPC applications on a real disaggregated memory system prototype. Our evaluation shows that INDIGO offers up to 50–70% improvement in application performance compared to other state-of-the-art page migration policies and reduces network traffic up to 2×. 1. Introduction Background and Motivation. Hardware memory disaggregation (HMD) is an emerging technology that enables cache-line granularity access to remote memory with protocols such as CXL or openCAPI ( CXL [n. d.] ; ope [n. d.] ; CCI [n. d.] ) , thereby creating expansive memory pools and reducing memory underutilization in datacenters. To cost effectively support the growing requirements of memory-intensive applications ( Stonebraker and Weisberg 2013 ; Vaswani et al. 2017 ; Murphy et al. 2010 ) , memory expansion has gone beyond a single compute node to multiple nodes and a remote memory pool that are connected together with a switched interconnect ( Pinto et al. 2020 ; xco [n. d.] ; h3- [n. d.] ) . As with any switched network, contention from multiple hosts results in slow down of remote memory access as shown in Figure 1 . Specifically, we find that multi-node contention in HMD systems results in 3–6× higher performance degradation compared to traditional tiered memory systems ( Gouk et al. 2023 ; Gouk et al. 2022 ; Patke et al. 2022 ; Lim et al. 2009 ; Syrivelis et al. 2017 ) . Our discussions with a leading cloud and HPC provider indicate that network contention is a significant bottleneck in the deployment of CXL memory. Page migration ( lin [n. d.] ) can be reused to address network contention in HMD systems by transferring frequently accessed pages from remote to local memory. However, the associated page transfer cost , i.e. the time required to copy pages between remote and local memory, is higher in HMD systems and increases during network contention as shown in Figure 2 (Left). Previously proposed page migration mechanisms such as ( lin [n. d.] ; Kim et al. 2021a ; Verghese et al. 1996 ; Agarwal and Wenisch 2017 ; Ren et al. 2023 ; Kannan et al. 2017 ; Raybuck et al. 2021 ; Li et al. 2023b ; Yan et al. 2019 ; Maruf et al. 2023 ; Lee et al. 2023 ) that are designed for legacy NUMA systems, or disaggregated systems with point-to-point CPU-memory connections, do not consider such variable page transfer costs resulting in sub-optimal scenarios as described below. Figure 1 . Compared to tiered memory systems, multiple nodes contend for the same remote memory pool leading to network contention in HMD systems. Figure 2 . Previously proposed page migration mechanisms do not consider the variable page transfer costs in HMD systems. (Left) Page transfer cost for 4 KB pages in tiered memory and HMD systems under varying network contention. (Right) INDIGO considers the impact of variable page transfer costs with Page Promoter and Page Telemetry resulting in decreased application runtime. Each application was run with local memory allocation = 10% of the working set size and compared with the best runtime between TPP ( Maruf et al. 2023 ) , MEMTIS ( Lee et al. 2023 ) and Nimble ( Yan et al. 2019 ) First, previously proposed mechanisms are not effective at transferring pages during network contention. Page migration is only beneficial when the improvement in access locality outweighs the page transfer cost. During network contention, the cost of transferring pages increases because of reduced available bandwidth, which results in excessive page transfers with the current mechanisms. Second, previously proposed mechanisms fail to transfer pages optimally when application access patterns are dynamic. Page migration is based on access rates estimated using moving averages ( Raybuck et al. 2021 ; Lee et al. 2023 ) or bit vectors ( Kwon et al. 2016 ) . These estimates become highly inaccurate with dynamic patterns. For example, Graph500 applications like BFS and SSSP have estimation errors greater than 1000%, resulting in the transfer of infrequently accessed pages. In HMD systems, this issue is exacerbated due to the high cost of page transfers. INDIGO Design. To resolve the limitations of previous page migration mechanisms in the context of HMD systems, we introduce INDIGO. INDIGO consists of two major components: Page Telemetry and Page Promoter . Page Telemetry improves the accuracy of page access estimation by differentiating between stationary and dynamic application access patterns. Differentiation between the two patterns is enabled by calculating the size of each interval (i.e., “burst duration”) in which the access pattern does not vary significantly. Smaller burst durations imply that the application memory access pattern is dynamic (and vice versa for larger ones). Page Promoter adopts a network-aware page migration policy that considers the trade-off between the cost of page transfer and memory access locality benefit (we develop an analytical model to consider this trade-off in Section 2.4 ) within the constraint of limited local memory allocation. The input to the Page Promoter includes measurements from Page Telemetry, application memory allocation, and network telemetry. The Page Promoter migrates pages if the access rates and burst duration exceeds a threshold value. This threshold value is decided at the application start by a contextual multi-armed bandits agent that is trained using historical application runtime data. We also provide an alternate heuristic-based network adaptive policy if no application training data is available 1 1 1 The heuristic-based approach is only network adaptive but not adaptive to the dynamicity of application access patterns . INDIGO is implemented between Linux kernel and the userspace. Page Telemetry and Page Promoter are both implemented in the Linux kernel, while the contextual multi-armed bandits control is implemented in the userspace. Both components contribute to INDIGO’s improved performance as shown in Figure 2 (Right). Results. We evaluated INDIGO with common cloud applications, including in-memory databases ( Stonebraker and Weisberg 2013 ) , in-memory key-value stores ( Carlson 2013 ) , ML inference ( Vaswani et al. 2017 ) , graph analytics kernels ( Murphy et al. 2010 ) , and scientific computational code ( Gahvari et al. 2011 ) on a real HMD prototype system: ThymesisFlow ( thy [n. d.] )