Deciphering boundary layer dynamics in high-Rayleigh-number convection using 3360 GPUs and a high-scaling in-situ workflow
Mathis Bode, Damian Alvarez, Paul Fischer, Christos E. Frouzakis, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, Victor A. Mateevitsi, Misun Min, Michael E. Papka, Silvio Rizzi, Roshan J. Samuel, Jörg Schumacher · 2025
Read Deciphering boundary layer dynamics in high-Rayleigh-number convection using 3360 GPUs and a high-scaling in-situ workflow by Mathis Bode, Damian Alvarez, Paul Fischer, Christos E. Frouzakis, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, Victor A. Mateevitsi, Misun Min, Michael E. Papka, Silvio Rizzi, Roshan J. Samuel, Jörg Schumacher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13240.txt.
Introduction
Mathis Bode and co-authors run hot-air churn sims at top heat grades on a huge chip fleet and film them live. The work dates to 2025. Heat churn spans many scales, and its thin wall zones shift fast. Full three-part reads of those zones need top machines.
The runs used a known flow code on a top Europe machine, the big one on near three thousand plus chips. Heat grades hit a top mark far past past runs. A live film flow taps data as the sim runs, at high pace. That lets the team track fast wall shifts that disk saves would miss. The pair of gains frames the work.
Catalog record
| CREATOR | Mathis Bode, Damian Alvarez, Paul Fischer, Christos E. Frouzakis, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, Victor A. Mateevitsi, Misun Min, Michael E. Papka, Silvio Rizzi, Roshan J. Samuel, Jörg Schumacher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13240
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deciphering boundary layer dynamics in high-Rayleigh-number convection using 3360 GPUs and a high-scaling in-situ workflow Mathis Bode, Damian Alvarez, Paul Fischer, Christos E. Frouzakis, Jens Henrik Göbbert, Joseph A. Insley, Yu-Hsiang Lan, Victor A. Mateevitsi, Misun Min, Michael E. Papka, Silvio Rizzi, Roshan J. Samuel, Jörg Schumacher Abstract Turbulent heat and momentum transfer processes due to thermal convection cover many scales and are of great importance for several natural and technical flows. One consequence is that a fully resolved three-dimensional analysis of these turbulent transfers at high Rayleigh numbers, which includes the boundary layers, is possible only using supercomputers. The visualization of these dynamics poses an additional hurdle since the thermal and viscous boundary layers in thermal convection fluctuate strongly. In order to track these fluctuations continuously, data must be tapped at high frequency for visualization, which is difficult to achieve using conventional methods. This paper makes two main contributions in this context. First, it discusses the simulations of turbulent Rayleigh-Bénard convection up to Rayleigh numbers of R a = 10 12 Ra=10^{12} computed with NekRS on GPUs. The largest simulation was run on 840 nodes with 3360 GPU on the JUWELS Booster supercomputer. Secondly, an in-situ workflow using ASCENT is presented, which was successfully used to visualize the high-frequency turbulent fluctuations. 1 Introduction The availability of exascale supercomputers is both, a great opportunity to address unsolved scientific questions and advance technology, and a tremendous challenge for all scientists involved. Faster computers often result in even larger data sets, making it more difficult to process, visualize, and analyze that data. Other challenges include scalability issues of numerical algorithms, data security in increasingly heterogeneous or modular computing environments, and efficient use of complex high-performance computing (HPC) systems. Solving the Navier-Stokes equations (NSEs) or simplified forms of fluid flow problems (collectively referred to as Computational Fluid Dynamics - CFD) is one of the traditional large application areas for supercomputers [ 1 , 2 ] . Due to the coupling of scales in turbulent flows, potential multi-physics effects, and the numerical difficulty of the problem, many fluid flow applications are still computationally too expensive even for exascale systems [ 3 , 4 ] . NekRS is one of the frontrunner codes for solving low-Mach/incompressible NSEs on GPUs [ 5 , 6 ] and is applied in this paper to decipher boundary layer effects in a three-dimensional (3D) flow subject to turbulent heat transfer at high Rayleigh numbers. For this purpose, a large-scale simulation with 840 nodes/3360 GPUs, i.e., 90 % of all JUWELS Booster nodes, was performed for almost 24 hours to reach a statistically steady state of turbulence plus another 6 hours utilizing in-situ visualization on JUWELS Booster. JUWELS Booster, hosted at the Jülich Supercomputing Centre (JSC), was ranked 18th on the TOP500 list [ 7 ] when the presented simulations were performed. Turbulent heat transfer is a fundamental transport process in various fluid flows in nature and technology. The temperature difference in heat transfer problems causes density differences; denser fluid sinks due to gravity acceleration and lighter fluid rises. In the context of heat transfer in buoyancy-driven flows, also known as free (or natural) convection, the Rayleigh number is a central dimensionless parameter. The Rayleigh number Ra relates the resulting buoyancy to viscous forces and thus describes the thermal driving of the fluid flow. Below a critical Rayleigh number Ra_{\rm cr} , heat transfer in an initially quiescent system takes place exclusively by heat conduction, i.e., without fluid motion. Above this level, Ra>Ra_{\rm cr} , heat is transferred by natural convection, i.e., conduction and additional fluid motion. The higher the Rayleigh number, the more turbulent the flow becomes as measured by the Reynolds number (which also increases roughly as Re\sim Ra^{1/2} . With increasing Ra the amount of transfered heat increases and the smallest physically relevant scales in the flow decrease. Examples of turbulent heat transfer at high Rayleigh numbers are atmospheric flows, stellar convection or oceanic turbulence, where the relevant Rayleigh numbers are Ra\gtrsim 10^{18} . The paradigm to these complex heat transfer processes in nature and technology is the Rayleigh-Bénard convection (RBC) case, a plane fluid layer between two infinitely extended parallel plates which is uniformly heated from below and cooled from above [ 8 , 9 ] . An infinitely extended natural convection layer is numerically modeled by a finite domain with a periodicity length L in the horizontal directions and a layer height H . The aspect ratio of the domain follows to \Gamma=L/H in the RBC setup (cf. subsection 4.1 ). A higher turbulence level (which is in line with a higher Ra ) results in a scale ratio between the largest and smallest vortices or thermals in the flow which grows with Ra^{3/2} . With the largest length scale given as the constant domain size, this implies that as Ra increases, the smallest relevant lengths become ever smaller. Thus, the grid resolution of the direct numerical simulation (DNS) must become correspondingly finer. For the analysis of RBC turbulence, highly accurate data are essential. With growing Ra , the local resolution must be increased to resolve all relevant effects. Ensuring that all data are sufficiently numerically resolved, a planar RBC setup with a square base was considered here for different Ra and \Gamma . This was also achieved by analyzing which DNS conditions are reasonable in the context of computing costs, maximum system memory, and physical significance. One major result of a recent series of DNS of RBC for Ra\leq 10^{11} was that the boundary layers of the temperature and velocity fields in a horizontally unconstrained configuration ( \Gamma=4 and periodic boundary conditions in horizontal directions) are dominated by fluctuations [ 10 ] . This adds another layer of complexity from a HPC standpoint. For the largest case in this work ( Ra=${10}^{12}$ ), the generated data (three velocity components, pressure, temperature) at each time step is 1.2 TB in size or even 1.9 TB if the spectral element grid is additionally written out. It is therefore not possible with classical visualization, i.e., by writing out complete checkpoint files and visualization as post-processing, to store data during a simulation in order to visualize the dynamics of the field fluctuations smoothly with respect to time. To fully visualize these fluctuations, which are physically essential to understand the local mechanisms of turbulent heat transfer, an in-situ workflow using ASCENT [ 11 ] was implemented and successfully applied on 840 JUWELS Booster nodes, as well as on other supercomputers at smaller scale. It has already been shown in the past that ASCENT can be used for large-scale simulations (up to 16\,000 GPUs [ 12 ] ). However, the use case considered here is different in that it requires both, large GPU numbers and a very high visualization frequency for the physical reasons mentioned above. As a consequence, the optimization of all sub-applications, i.e., NekRS and ASCENT, is essential, as otherwise the computational overhead will be too high. This paper presents such an optimized implementation and coupling. In-situ visualization allows data visualization directly in the GPU memory. For this purpose, the simulation is periodically interrupted after a defined number of time steps and images are rendered. This approach significantly