LCI: a Lightweight Communication Interface for Efficient Asynchronous Multithreaded Communication

Jiakun Yan, Marc Snir · 2025

Read LCI: a Lightweight Communication Interface for Efficient Asynchronous Multithreaded Communication by Jiakun Yan, Marc Snir (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01864.txt.

Introduction

Jiakun Yan and Marc Snir present a communication library in 2025 for supercomputer programs that juggle many threads at once. Modern machines and methods push messaging toward background, overlapping work, where old libraries waste thread effort. Their library, named LCI in the paper, offers compact point-to-point tools with several ways to signal completion.

Its runtime leans on atomic data structures, fine-grained locks, and close knowledge of the network hardware. Tests on two cluster types, with small benchmarks and two full applications, show it beating rival libraries in threaded settings. It even tops the classic one-process-per-core mode in some cases. The code is open source.

Catalog record

CREATORJiakun Yan, Marc Snir
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LCI: a Lightweight Communication Interface for Efficient Asynchronous Multithreaded Communication Jiakun Yan, Marc Snir Abstract . The evolution of architectures, programming models, and algorithms is driving communication towards greater asynchrony and concurrency, usually in multithreaded environments. We present LCI, a communication library designed for efficient asynchronous multithreaded communication. LCI provides a concise interface that supports common point-to-point primitives and diverse completion mechanisms, along with flexible controls for incrementally fine-tuning communication resources and runtime behavior. It features a threading-efficient runtime built on atomic data structures, fine-grained non-blocking locks, and low-level network insights. We evaluate LCI on both Inifiniband and Slingshot-11 clusters with microbenchmarks and two application-level benchmarks. Experiment results show that LCI significantly outperforms existing communication libraries in various multithreaded scenarios, achieving performance that exceeds the traditional multi-process execution mode and unlocking new possibilities for emerging programming models and applications. 1. Introduction High-performance computing (HPC) architectures have become increasingly heterogeneous with extensive on-node parallelism ( LLNL [n. d.] ; HPCwire 2024 ) , while applications employ complex algorithms with sparsity or adaptivity ( Hofmeyr et al. 2020 ; Marcello et al. 2021 ; Abdulah et al. 2024 ) . In addition, new asynchronous, task-oriented programming models with runtime resource management and scheduling are becoming more popular ( Bauer et al. 2 11 ; Kaiser et al. 2020 ; Bosilca et al. 3 11 ; Augonnet et al. 2024 ) . These trends are leading to a shift of application communication characteristics: multiple threads can logically initiate communications simultaneously; more asynchronous point-to-point communications are being used, as opposed to collective communication of bulk-synchronous styles; and there can be more simultaneously pending fine-grained communications and more opportunities for computation-communication overlap. These characteristics fall out of the original focus of MPI, the de facto standard HPC communication library designed over 30 years ago. Since then, new communication libraries and MPI features have been introduced to tackle the asynchrony. Multiple research efforts, mainly by the MPI community, have been taken to improve multithreaded communication support. However, they still fall short of the needs of applications due to limited flexibility and constrained design space. • Limited Flexibility : Each communication library only offers a limited selection of communication mechanisms. However, modern programming systems and/or applications can need combinations of many communication mechanisms. Clients often must implement their communication mechanisms on top of the existing library interface. This requires a significant effort and is not optimal when the library does not expose low-level functionality. • Constrained Design Space : Most communication libraries were not designed with multithreaded performance in mind from the beginning. Existing efforts to improve multithreaded communication support (mainly for MPI) are hence handicapped by legacy code base and backward compatibility concerns, resulting in a solution that is not optimal in terms of both performance and programmability. Suboptimal communication support, in turn, complicates the innovation of new programming models, forcing developers to adopt workarounds such as funneling communication through a single thread ( Swartvagher 2022 ) , hacking into inner communication layers ( Castillo et al. 2019 ) , or using proxy processes for communication progressing ( Zhu et al. 2023 ) . To address these issues, we present the Lightweight Communication Interface (LCI), a communication library designed from scratch with asynchronous multithreaded communication in mind. It provides a unified interface that supports flexible combinations of all common point-to-point communication primitives, including send-receive, active messages, and RMA put/get (with/without notification), and various built-in mechanisms to synchronize with pending communications, including synchronizers, completion queues, function handlers, and completion graphs. In addition, the interface offers both a simple starting point for users to program and a wide range of options for them to incrementally fine-tune the communication resources and runtime behaviors, minimizing potential interference between communication and computation. Finally, it is supported by a lightweight and efficient runtime optimized for threading efficiency and massive parallelism. The runtime is built with a deep understanding of low-level network activities and employs optimizations such as atomic-based data structures, thread-local storage, and fine-grained nonblocking locks. We evaluate LCI with microbenchmarks, a k-mer counting mini-app, and an astrophysics AMT-based application on Infiniband and Slingshot-11 clusters. The results show that LCI outperforms existing communication libraries, including standard MPI, MPICH with the VCI extension, and GASNet-EX, by a large margin in multithreaded performance while maintaining comparable single-threaded performance. The rest of the paper is organized as follows: Section 2 discusses related works. Section 3 introduces LCI’s communication interface and shows how it can seamlessly support dynamic programming systems. Section 4 presents the key designs in LCI runtime. Section 5 analyzes the evaluation results. Section 6 concludes the paper and discusses future works. 2. Related Work 2.1. Asynchronous Communication MPI-1 ( Snir 1998 ) was designed around coordinated communication paradigms, including two-sided send-receive and collective operations. It was developed at a time when most HPC applications followed the Bulk-Synchronous Parallel (BSP) programming model, which alternates computation and communication phases, effectively synchronizing all cores in the system. The BSP model becomes increasingly problematic as core counts increase, their compute speeds vary, and applications become more irregular. In contrast, asynchronous models allow threads to issue communication in an uncoordinated manner. New MPI features have been proposed since then to improve support for asynchronous communication. RMA operations have been included in MPI since MPI-2, though its "window" abstraction still operates in a partially collective style. The MPI continuation proposal ( Schuchart et al. 9 01 ) recently introduced a way for clients to attach callbacks to pending MPI operations, aiming for more efficient polling in the case of heavy communication overlapping. A range of communication libraries has also been proposed. GASNet ( Bonachea and Jeong 2002 ) and the later GASNet-EX ( Bonachea and Hargrove 2018 ) focus on one-sided active messages and RMA primitives. They are intended to be used by runtime developers or as a compilation target, so their interfaces are generally more complicated than MPI. At a higher level, PGAS libraries/languages, such as UPC ( El-Ghazawi and Smith 2006 ) , UPC++ ( Bachan et al. 2019 ) , and OpenSHMEM ( Chapman et al. 2010 ) , rely on RMA operations to maintain a global address abstraction. Recently, new communication libraries have been proposed. YGM ( Steil et al. 2023 ) features a batch-processing active messages interface and utilizes aggregation for better throughput. UNR ( Feng et al. 2024 ) emphasizes notifiable RMA operations, optimizing them for multi-NIC aggregation and ease of use. UCX ( Shamis et al. 2015 ) and Libfabric ( (2024) OFIWG ) provide low-level abstractions that are portable across multiple interconnects. They offer more flexible interfaces but at a much lower level. They also require manual bootstrap. Their primary usage is to support communication libraries rather than high-level programming systems/applications. While these libraries have made significant progress in supporting asynchronous communication, they often provide a limited selection of features that cannot fully fulfill the communication needs of complicated runtime systems/applications. LCI improves upon these libraries by providing a more comprehensive and flexible interface that allows for a broader range of communication patterns and optimizations. It also has an additional performance focus on multithreaded communication. ( Yan and Snir 2025 ) presents an overview and some considerations of an earlier version

Read the complete text