Improving the scalability of a high-order atmospheric dynamics solver based on the deal.II library

Giuseppe Orlando, Tommaso Benacchio, Luca Bonaventura · 2025

Read Improving the scalability of a high-order atmospheric dynamics solver based on the deal.II library by Giuseppe Orlando, Tommaso Benacchio, Luca Bonaventura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00384.txt.

Introduction

Giuseppe Orlando and co-authors scale up a weather dynamics solver in a 2025 paper. Their code, built on the deal.II library, runs a high order scheme for air motion with a matrix free design that skips building one giant system matrix. Such codes must spread well across huge clusters to be of use.

Profiling first showed where time went, and the team then tuned preconditioners and solvers inside that frame. Swapping to the newest library build and apt run flags also lifted parallel speed. The gains help the solver face bigger grids and longer runs for atmosphere studies on shared supercomputers.

Catalog record

CREATORGiuseppe Orlando, Tommaso Benacchio, Luca Bonaventura
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00384

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving the scalability of a high-order atmospheric dynamics solver based on the deal.II library Giuseppe Orlando, Tommaso Benacchio, Luca Bonaventura Abstract We present recent advances on the massively parallel performance of a numerical scheme for atmosphere dynamics applications based on the deal.II library. The implicit-explicit discontinuous finite element scheme is based on a matrix-free approach, meaning that no global sparse matrix is built and only the action of the linear operators on a vector is actually implemented. Following a profiling analysis, we focus on the performance optimization of the numerical method and describe the impact of different preconditioning and solving techniques in this framework. Moreover, we show how the use of the latest version of the deal.II library and of suitable execution flags can improve the parallel performance. (1) CMAP, CNRS, École polytechnique, Institute Polytechnique de Paris Route de Saclay, 91120 Palaiseau, France [email protected] (2) Weather Research, Danish Meteorological Institute Sankt Kjelds Plads 11, 2100 Copenhagen, Denmark [email protected] (3) Dipartimento di Matematica, Politecnico di Milano Piazza Leonardo da Vinci 32, 20133 Milano, Italy [email protected] Keywords : Numerical Weather Prediction, Non-conforming meshes, Flows over orography, IMEX schemes, deal.II 1 Introduction Efficient numerical simulations of atmospheric flows are key to viable weather and climate forecasts and several related practical applications. Medium-range global numerical weather predictions (NWP) up to ten days ahead are typically required to complete within a one-hour operational threshold in the forecast cycles of meteorological centres. In addition, the computational meshes employed for these forecasts are being refined to head towards the km-scale, while experimental limited-area suites already reach the hectometric scale [ 10 ] . Hence, in the current context of increasing demand of computational resources driven by increasingly high spatial resolutions, massively parallel model scalability is a fundamental requirement. In this work, we present recent advances in the parallel performance of a Implicit-Explicit Discontinuous Galerkin (IMEX-DG) solver [ 12 ] employed for atmosphere dynamics applications [ 13 , 15 , 14 ] , following up on earlier analyses by the authors [ 16 ] . The implementation is carried out in the framework of the deal.II library [ 1 , 2 , 3 ] . First, we show how the use of suitable preconditioning techniques can significantly improve the performance of the elliptic pressure solver. Next, we assess the impact of activating execution flags and of using the most recent version of the library (9.6.2, [ 1 ] ). Finally, we present the result of employing a different strategy for the solution of a linear system. The paper is structured as follows. In Section 2 , we briefly review the model equations and the relevant details of the numerical methodology used in this work. The profiling and parallel performance analysis in numerical experiments with a three-dimensional benchmark of atmospheric flow are presented in Section 3 . Finally, some conclusions are reported in Section 4 . 2 The model equations and some details of the numerical method The compressible Euler equations of gas dynamics represent the most comprehensive mathematical model for atmospheric flows [ 20 ] . Considering for simplicity the dry, non-rotating case, the mathematical model reads as follows: \displaystyle\frac{\partial{\rho}}{\partial{t}}+\dive\left(\rho\vel\right)=0 \displaystyle\frac{\partial{\left(\rho\vel\right)}}{\partial{t}}+\dive\left(\rho\vel\otimes\vel\right)+\grad p=\rho\mathbf{g} (1) \displaystyle\frac{\partial{\left(\rho E\right)}}{\partial{t}}+\dive\left[\left(\rho h+\rho k\right)\vel\right]=\rho\mathbf{g}\cdot\vel, where t\in(0,T_{f}] is the temporal coordinate, supplied with suitable initial and boundary conditions on the computational domain. Here \otimes denotes the tensor product, T_{f} is the final time, \rho is the density, \mathbf{u} is the fluid velocity, and p is the pressure. Moreover, \mathbf{g}=-g\mathbf{k} is the acceleration of gravity, with g=$9.81\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ and \mathbf{k} being the upward pointing unit vector in the standard Cartesian frame of reference. Finally, E denotes the total energy per unit of mass, h is the specific enthalpy and k=1/2\left|u\right|^{2} is the specific kinetic energy. System ( 2 ) is complemented by the equation of state of ideal gases, given by \rho e=\frac{1}{\gamma-1}p , where e denotes the specific internal energy e=E-k and the isentropic exponent \gamma is taken equal to 1.4 . The time discretization is based on an Implicit-Explicit (IMEX) Runge-Kutta (RK) method [ 6 ] , which is a widely employed approach for ODE systems that include both stiff and non-stiff components. IMEX-RK schemes are represented compactly by the companion Butcher tableaux: \mathbf{c} \mathbf{A} \mathbf{b}^{\top} \tilde{\mathbf{c}} \tilde{\mathbf{A}} \tilde{\mathbf{b}}^{\top} with \mathbf{A}=\left\{a_{lm}\right\},\mathbf{b}=\left\{b_{l}\right\},\mathbf{c}=\left\{c_{l}\right\} , l,m=1\dots s , denoting the coefficients of the explicit method, \tilde{\mathbf{A}}=\left\{\tilde{a}_{lm}\right\},\tilde{\mathbf{b}}=\left\{\tilde{b}_{l}\right\} , and \tilde{\mathbf{c}}=\left\{\tilde{c}_{l}\right\} , l,m=1\dots s denoting the coefficients of the implicit method, and s representing the number of stages of the method. We refer, e.g., to [ 6 , 19 ] for a detailed analysis of the order and coupling conditions of the two companion schemes. A generic IMEX-RK l stage with time step \Delta t for the Euler equations reads therefore as follows: \displaystyle\rho^{(l)}=\rho^{n}-\sum_{m=1}^{l-1}a_{lm}\Delta t\dive\left(\rho^{(m)}\vel^{(m)}\right) \displaystyle\rho^{(l)}\mathbf{u}^{(l)}+\tilde{a}_{ll}\Delta t\grad p^{(l)}=\mathbf{m}^{(l)} (2) \displaystyle\rho^{(l)}E^{(l)}+\tilde{a}_{ll}\Delta t\dive\left(h^{(l)}\rho^{(l)}\vel^{(l)}\right)=\hat{e}^{(l)}, where we have set \displaystyle\mathbf{m}^{(l)} \displaystyle= \displaystyle\rho^{n}\vel^{n}-\sum_{m=1}^{l-1}a_{lm}\Delta t\dive\left(\rho^{(m)}\vel^{(m)}\otimes\vel^{(m)}\right)-\sum_{m=1}^{l-1}\tilde{a}_{lm}\Delta t\grad p^{(m)} (3) \displaystyle\hat{\mathbf{e}}^{(l)} \displaystyle= \displaystyle\rho^{n}E^{n}-\sum_{m=1}^{l-1}\tilde{a}_{lm}\Delta t\dive\left(h^{(m)}\rho^{(m)}\vel^{(m)}\right))-\sum_{m=1}^{l-1}a_{lm}\Delta t\dive\left(k^{(m)}\rho^{(m)}\vel^{(m)}\right) (4) Notice that, substituting formally \rho^{(l)}\vel^{(l)} into the energy equation and taking into account the definitions \rho E=\rho e+\rho k and h=e+p/\rho , the following nonlinear Helmholtz-type equation for the pressure is obtained: \displaystyle\rho^{(l)}\left[e(p^{(l)},\rho^{(l)})+k^{(l)}\right] \displaystyle- \displaystyle\tilde{a}_{ll}^{2}\Delta t^{2}\dive\left[\left(e(p^{(l)},\rho^{(l)})+\frac{p^{(l)}}{\rho^{(l)}}\right)\nabla p^{(l)}\right] (5) \displaystyle+ \displaystyle\tilde{a}_{ll}\Delta t\dive\left[\left(e(p^{(l)},\rho^{(l)})+\frac{p^{(l)}}{\rho^{(l)}}\right)\mathbf{m}^{(l)}\right]=\hat{e}^{(l)}, which is solved through a fixed point procedure. We refer to [ 12 , 17 , 18 ] for further references and details on the theoretical properties of the method. The spatial discretization is based on a Discontinuous Galerkin (DG) method [ 5 ] , which combines high-order accuracy and flexibility in a highly data-local framework. In particular, this method is particularly well suited for mesh adaptive approaches [ 13 , 15 ] . The shape functions correspond to the products of Lagrange interpolation polynomials for the support points of \left(r+1\right) -order Gauss-Lobatto quadrature rule in each coordinate direction, with r denoting the polynomial degree. The discrete formulation of method ( 2 ) can therefore be expressed as \displaystyle\mathbf{A}^{(l)}\mathbf{U}^{(l)}+\mathbf{B}^{(l)}\mathbf{P}^{(l)}=\mathbf{F}^{(l)} (6) \displaystyle\mathbf{C}^{(l)}\mathbf{U}^{(l)}+\mathbf{D}^{(l)}\mathbf{P}^{(l)}=\mathbf{G}^{(l)}, (7) where \mathbf{U}^{(l)} represents the vector of degrees of freedom of the velocity field, while \mathbf{P}^{(l)} is the vector of the degrees of freedom of the pressure field. We refer to [ 13 , 17 , 16 ] for the detailed expression of all the matrices and vectors. We report only the expression of the matrix A^{(l)} that will be useful for the discussion in Section 3 : A_{ij}^{(l)}=\sum_{K\in\mathcal{T}_{\mathcal{H}}}\int_{K}\rho^{(l)}\boldsymbol{\varphi}_{j}\cdot\boldsymbol{\varphi}_{i}\mathrm{d}\mathbf{x}, (8) where \mathcal{T}_{\mathcal{H}} represents the computational mesh, \boldsymbol{\varphi}_{i} denotes the basis function of the space of polynomial functions employed to discretize the velocity, and \mathrm{d}\mathbf{x} is the spatial element of integration. Formally, from ( 6 ) one can derive \mathbf{U}^{(l)}=\left(\mathbf{A}^{(l)}\right)^{-1}\left(\mathbf{F}^{(l)}-\mathbf{B}^{(l)}\mathbf{P}^{(l)}\right), (9) and obtain \mathbf{D}^{(l)}\mathbf{P}^{(l)}+\mathbf{C}^{(l)}\left(\mathbf{A}^{(l)}\right)^{-1}\left(\mathbf{F}^{(l)}-\mathbf{B}^{(l)}\mathbf{P}^{(l)}\right)=\mathbf{G}^{(l)}. (10) The above system is then solved following the fixed point procedure described in [ 12 ] . In Sections 3.1 and 3.3 we describe the impact of using different preconditioning and solving techniques for system ( 10 ). 3 Numerical results In this Section, we show results of simulations of an idealized three-dimensional test case of atmospheric flow over orography [ 11 , 15 ] using the numerical method described in the previous Section and focusing on its scalability. The simulations have been run using up to 1024 2x AMD EPYC Rome 7H12 64c 2.6GHz CPUs at MeluXina HPC facility 1 1 1 https://docs.lxp.lu/ and OpenMPI 4.1.5 has been employed. The compiler is GCC version 12.3 and

Read the complete text