Do MPI Derived Datatypes Actually Help? A Single-Node Cross-Implementation Study on Shared-Memory Communication

Temitayo Adefemi · 2025

Read Do MPI Derived Datatypes Actually Help? A Single-Node Cross-Implementation Study on Shared-Memory Communication by Temitayo Adefemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13804.txt.

Introduction

Temitayo Adefemi benchmarks whether MPI derived datatypes truly speed up shared-memory communication on a single node. Derived types promise copy-free handling of scattered data, but reports usually cover one MPI library only. Three two-dimensional codes, a fluid solver, a Game of Life, and an image reconstruction, were each written with manual packing and with derived types.

Identical communication patterns were timed under four major MPI implementations with several operation styles and small rank counts, with halo data checked as bit-identical. Results flipped by code and by library: derived types sometimes won and sometimes trailed badly, with stack-specific slowdowns also appearing. The advice is to profile both designs under the exact stack and mode intended.

Catalog record

CREATORTemitayo Adefemi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13804

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do MPI Derived Datatypes Actually Help? A Single-Node Cross-Implementation Study on Shared-Memory Communication Temitayo Adefemi Abstract MPI’s derived datatypes (DDTs) promise easier, copy-free communication of non-contiguous data, yet their practical performance remains debated and often reported only for a single MPI stack. This paper provides a cross-implementation assessment using three representative 2D applications—a Jacobi CFD solver, Conway’s Game of Life, and a lattice-based image reconstruction—each written in two ways: (i) BASIC with manual packing/unpacking of non-contiguous regions and (ii) DDT using MPI_Type_vector / MPI_Type_create_subarray with correct true extent via MPI_Type_create_resized . For API parity, we benchmark identical communication semantics: non-blocking point-to-point ( Irecv / Isend + Waitall ), neighborhood collectives ( MPI_Neighbor_alltoallw ), and MPI-4 persistent operations ( *_init ). We run strong and weak scaling on 1 1 – 4 4 ranks, validating bitwise-identical halos, and evaluate on four widely used MPI implementations: MPICH , Open MPI , Intel MPI , and MVAPICH 2 (single node, ARCHER2). Results are mixed. DDTs can be fastest—e.g., non-blocking DDTs for the image reconstruction code on Intel MPI / MPICH —but can also be among the slowest on other stacks ( Open MPI / MVAPICH 2 for the same code). For the CFD solver, BASIC variants generally outperform DDTs across semantics, whereas for Game of Life, the ranking flips depending on the MPI library. We also observe stack-specific anomalies (e.g., MPICH slowdowns with DDT neighborhood/persistent modes). Overall, no strategy dominates across programs, semantics, and MPI stacks; performance portability for DDTs is not guaranteed . The practical takeaway is to profile both DDT-based and manual-packing designs under the intended MPI implementation(s) and communication mode(s). Limitations include single-node scope and no memory-overhead analysis; multi-node studies and GPU-aware paths are promising directions for future work. I Introduction The Message Passing Interface (MPI) has become a cornerstone of parallel programming since its first draft was presented at the Supercomputing ’93 conference in November 1993 [ 2 ] . Over time, the MPI standard has evolved to support a wide range of communication mechanisms between processes, including one-sided, synchronous, and asynchronous communication, as well as buffered communication [ 4 , 6 ] . Among its foundational features are derived datatypes , also known as user-defined datatypes [ 8 ] [ 3 ] . The derived datatype mechanism is a powerful and integral feature of MPI that enables communication of structured, non-contiguous, and non-homogeneous application data using any MPI communication operation. This eliminates the need for tedious, explicit, and often time and spaceconsuming manual packing and unpacking between intermediate communication buffers, thereby improving both efficiency and portability of MPI programs. Although research has explored the performance efficiency of MPI-derived datatypes across various programs, most existing studies are confined to a single MPI implementation. As a result, the observations drawn from these experiments cannot be easily generalized to other implementations. To address this gap, our study aims to answer a critical research question: ”Do MPI derived datatypes actually help across real codes and across multiple MPI stacks?” We also seek to determine the extent of their benefits and limitations. To this end, we implement two versions of three representative parallel programs to evaluate the use of MPI datatypes for memory and data management. The first version of each program employs standard MPI basic datatypes (e.g., MPI_INT , MPI_DOUBLE ) along with manual packing and unpacking to manage communication buffers. The second version leverages advanced MPI-derived datatypes to represent non-contiguous memory regions directly, allowing communication without explicit data copying. For comprehensive evaluation and diversity in implementation, both versions of all programs are developed and tested using four widely adopted MPI implementations: MPICH , Open MPI , Intel MPI , and MVAPICH 2 . We evaluate two implementation strategies across three representative parallel programs: (i) a 2D single-phase CFD solver using the Jacobi iterative method, (ii) a 2D Conway’s Game of Life cellular automaton, and (iii) a 2D lattice-based calculation for image reconstruction. Each program is provided in two versions. The BASIC version uses only MPI basic types (e.g., MPI_INT , MPI_DOUBLE , MPI_CHAR ) and communicates non-contiguous data via explicit copies into contiguous scratch buffers. The DDT version replaces manual packing with MPI-derived datatypes (e.g., MPI_Type_vector , MPI_Type_create_subarray , and MPI_Type_create_resized ) to describe halo rows, columns, and corners, enabling single-call exchanges. Our results reveal no systematic performance advantage for either MPI-derived datatypes or manual packing. In some experimental configurations, derived datatypes yielded lower execution times, whereas in others they did not. This lack of a consistent trend holds across all communication semantics evaluated and all MPI implementations examined. The relative performance of derived datatypes versus manual packing and unpacking is highly context-dependent, influenced by factors such as problem size, the characteristics of the underlying computation, the chosen MPI stack, the communication semantics, and the overall communication volume in the application. Consequently, our only robust recommendation for practitioners seeking optimal runtime performance when faced with a choice between derived datatypes and manual packing/unpacking is to empirically benchmark and profile their specific application to make an informed decision. The paper makes the following contributions: • We present a systematic, cross-implementation study of MPI-derived datatypes across three real stencil-style applications (a CFD solver, Conway’s Game of Life, and an image reconstruction kernel), comparing them with equivalent manual packing using basic MPI types. • We enforce strict API parity between the BASIC and DDT variants and evaluate three communication semantics—non-blocking point-to-point, neighborhood collectives, and MPI-4 persistent operations—to isolate the impact of the datatype representation itself from other algorithmic effects. • We perform an extensive experimental evaluation across four widely used MPI implementations (MPICH, Open MPI, Intel MPI, and MVAPICH2) on a modern HPC system, considering both strong and weak scaling scenarios and validating bitwise correctness of the exchanged halo data. • We analyze the resulting performance behavior and show that MPI-derived datatypes are neither universally beneficial nor universally harmful: their impact is strongly implementation- and semantic-dependent, highlighting the lack of performance portability and the necessity of empirical evaluation when choosing between BASIC and DDT designs. II Related Work The MPI standard introduced the concept of derived datatypes in its first version in the mid-1990s [ 1 ] , recognizing them as a powerful feature for describing non-contiguous and heterogeneous data layouts. Derived datatypes allow an application to communicate complex memory layouts with a single MPI operation, instead of manually packing data into contiguous buffers [ 11 ] . This mechanism improves programmer convenience and can enable optimized data transfer (e.g., using network scatter-gather capabilities) by the MPI library [ 8 , 7 ] . The elimination of tedious, error-prone manual packing and unpacking not only eases development but also offers potential performance benefits and improved code portability [ 11 ] . Early sources, such as the MPI-1 standard and tutorials, emphasized the use of derived types for these reasons [ 1 ] . Indeed, derived datatypes have been considered a cornerstone of MPI’s expressiveness since its inception. Despite their promise, MPI-derived datatypes have historically faced performance challenges in practice. Many MPI implementations in the late 1990s and 2000s did not efficiently support derived types, leading to suboptimal performance. Gropp et al. noted as early as 1999 that users often found it faster to manually

Read the complete text