Final Project Report on the RPI Contributions to the FASTMath SciDAC 5 Institute (SC0021285) Major Goals of the RPI Efforts to Develop Unstructured Mesh Methods The FASTMath unstructured mesh developments performed by RPI focused on methods, algorithms and associated software implementations of tools to carryout performant and scalable mesh-based operations on exascale computing systems. The selection of the tools developed was guided by the needs of SciDAC partnership projects. The FASTMath supported efforts focused on the design and development of core tools, while the extensions needed to meet specific SciDAC partnership application needs that demonstrate the usefulness of the tools were supported by the SciDAC partnerships. The RPI team’s developments included: • Hybrid unstructured mesh and particle technologies: A core development in this area is focused on the advancement of PUMIParticle, our mesh centric infrastructure for hybrid particle and mesh simulations. This infrastructure is the basis for new codes for microturbulence PIC calculations of plasmas, impurity particle transport, material point method calculations of ice sheets, and neutronics calculations. This infrastructure is designed to execute at scale on heterogeneous exascale computers. • Advanced mesh coupling techniques building on the Parallel Coupling for Multimodel Simulation infrastructure (PCMS): PCMS is designed to support efficient one-way and in-memory two-way coupling of physics simulation codes. This is a joint project with RAPIDS and employs ADIOS2. • Machine Learning (ML) for unstructured mesh and coupled applications: We are developing machine learning techniques for improved hexahedral dominated mesh generation, automatic shock/feature detection for automatic mesh adaptation, and new unstructured mesh driven multiscale workflows that use machine learning. • Architecture-aware task placement and load balancing: Efforts in this area are focused on GPU implementations of graph partitioners, graph-based coarsening techniques and the EnGPar procedures for dynamic load balancing of adaptive mesh and unstructured mesh- based particle simulations. • Unstructured mesh adaptation for high order methods: Efforts focused on developing a complete set of GPU conforming mesh adaptation methods and the advancement of high- order curved mesh adaptation methods. • High-order analysis methods: Efforts in this area focused on the integration of the GPU- based conforming mesh adaptation procedures into the MFEM high order finite element and methods for effective curved mesh adaptation. • Error/correction indicators and adaptive simulation workflows for extended MHD, Maxwell equation, and sheet simulations. • Software stewardship of the RPI developed and supported unstructured mesh tools indicated below. Our ability to address the needs of SciDAC applications, and to effectively transition the new methods and algorithms developed to those applications, is enhanced through direct involvement with SciDAC application partnerships. RPI is part of the following SciDAC 5 partnerships “Improved Coupled Climate Simulations in E3SM Through Enhanced Sea-Ice Mechanics” and “Framework for Antartic System Science in E3SM” BER Partnerships, and the “High Fidelity Digital Models for Fusion Pilot Plant Design”, “Center for Advanced Simulation of RF - Plasma - Material Interactions”, “HiFiStell: High-Fidelity Simulations for Stellarators” and “Computational Evaluation and Design of Actuators for Core-Edge Integration” FES Partnerships. The tools being developed in this project are open source (BSD) and maintained in GitHub repositories. Project Accomplishments The specific areas of methods and algorithm development carried out by the RPI team included: • Omega_h unstructured mesh adaptation library • Parallel unstructured mesh infrastructure for particle calculations • Curved mesh adaptation for high-order finite element methods • Core tools for GPU-based material point method for Voronoi meshes • Parallel Coupler for Multimodel Simulation (PCMS) • Error/correction indicators and adaptive simulation workflows • ML surrogate material models from multiscale simulations • ML-based region detection for hexahedral mesh generation • Coarsening and communication for hierarchical partitioning • Software stewardship for unstructured mesh components Omega_h unstructured mesh adaptation library: The Omega_h unstructured mesh adaptation library was specifically developed to support the execution of massively parallel conforming mesh adaptation processes on heterogeneous computer where the bulk of the processing power is provided by accelerators (particularly GPUs). Specific Omega_h developments in this project included: • Development of a set of array structures and APIs to support the storage of, and operations on, mesh based physics fields for effective GPU operation. • Developments supporting evolving geometric models which required refactoring key classes to more effectively employ geometric model topology classification information to answer geometric queries needed by mesh adaptation procedures. • Porting the CUDA-only memory pool implementation to Kokkos to increase performance. With this change, mesh adaptation benchmark case ran 3 times faster on a Perlmutter NVIDIA A100, 1.5 times faster on a GCD of a Frontier AMD MI250x, 1.6 times faster on a tile of Aurora Intel PVC GPU, and 1.35 times faster on NVIDIA GH200 APU. Parallel unstructured mesh infrastructure for particle calculations: The Parallel Unstructured Mesh Infrastructure for PIC was greatly expanded and generalized into the Parallel Unstructured Mesh Infrastructure for Particle operations in SciDAC 5. Key developments carried out in this project included: • Expanding the original set of three particle data structures in which particles were directly related to elements to include a fourth option, the Disorganized Particle Structure (DPS), which operates at the process level. The driver for this development was applications, such magnetically confined fusions system simulations of impurity transport, where the number of particles per element is, on average, is low and constantly changing. PUMIParticle is implemented such that all APIs operate with each of the four particle data structures and the user can easily apply the one most effective to the current application needs. • Introduction of a fast dynamic load balancing procedure to support applications where the distribution of particles per element and/or process evolves through the simulation. • The PUMIParticle APIs were extended to effectively address operations where the particles interact with material boundaries which are explicitly represented at the non- manifold topology level of the domain geometry. • The PUMIParticle APIs were extended to support the execution of tally operations which represent the most computationally expensive component in neutronics simulations. These procedures were linked into OpenMC. Test cases demonstrated factor of 19 decrease in computational times over the existing OpenMC unstructured mesh methods. Since that demonstration the OpenMC developers adopted key components of the PUMIParticle methods and have reduced, but not eliminated, the performance gap. The tally APIs are currently being used is a CEDA FES partnership effort to develop a more performant and scalable neutral particle transport code. • The PUMIParticle infrastructure was used to develop a fully featured global impurity transport code (GITRm) that is now being used in physics studies at ORNL and General Atomics. Curved mesh adaptation for high-order finite element methods: When applying high-order finite element methods, such as employed in MFEM (the code used by the 2025 Gordon Bell Prize winners), to domains with curved boundaries, it is necessary to employ curved meshes that approximate the curved model geometry to the level of the high-order finite elements used. This creates challenges in the generation and adaptation of the needed meshes. Key developments carried out in this project included: • Accepting meshes of linear, or quadratic, geometry elements produced by available mesh generators and improving the geometric model approximation of the mesh edges and mesh faces on curved boundaries to the geometric order required. (Currently up to sixth order geometry is supported. However, since Bezier geometry is used for the element geometric representation, increasing the order further is a simple task of resizing a set of the arrays employed.) • Effective identification of invalid, or nearly invalid, elements that can be created when curving element edges and faces to the boundary or when adapting the mesh as part of an adaptive analysis. • The development of effective local curved mesh modification operators to correct invalid elements due to curving to the boundary or to support adapting the mesh to match the requested mesh size field during and adaptive simulation. • (Effort still in progress) Modification of the local curved mesh modification operators to be more computationally and data storage efficient. These efforts include extending the mesh modification operators to more efficiently introduce interior curved mesh entities up to cubic (an order found satisfactory to meet the needs of SciDAC high-order finite element applications). The operator also favors maintaining as low a geometric order of mesh regions, faces and edges as possible that maintain acceptable element shapes. To take advantage of using as low an order geometry of mesh entities as possible, the curved mesh representation data structures and methods are extended to store only the level of geometric information needed for each mesh entity. Core tools for GPU-based material point method for Voronoi meshes: To support the effective HPC implementation of Material Point Methods (MPM) on polygonal meshes a modified version of PUMIParticle, referred to as polyMPO, is being developed for the efficient execution of MPM simulations on GPUs. The development of polyMPO was required since polygonal meshes have different mesh entity adjacency properties than “finite element” like unstructured meshes that PUMIParticle relies on for efficient GPU execution. The new structures have been developed and integrated with the newly DPS particle data structure. The core mesh/particle interaction operators needed by the MPM analysis code have been developed and tested on planar meshes. Parallel Coupler for Multimodel Simulation (PCMS): Considering the DOE’s investment in codes that model specific physics and the increased desire to directly couple simulations across multiple scales, there is a need to provide a tools that effectively supports the coupling of exascale enabled physics codes to each other, and to a broad range of faster simulation tools and surrogate models. Lessons learned in the ECP WDMApp project clearly demonstrated the need for an effective code coupling technology that could directly interact with physics field data to support from one-way file-based up to in-memory two-way coupling of codes that employed their own physics models and parallel physics fields data structures. The Parallel Coupler for Multimodel Simulation has been specifically designed to meet this need. Although still under active development, as we enter SciDAC 6, key developments carried out in SciDAC 5 included: • The ability to effectively support both one-way file based and in-memory two-way coupling of codes build on the capabilities of the ADIOS2 package. The development of this capability was a collaborative effort with the RAPIDS SciDAC Institute. • The development of an efficient method, using a rendezvous based parallel control scheme, to support coupling massively parallel simulation codes that employ their own parallel field data distribution mechanisms. • Multiple methods to support effective field transfer that can take full advantage of the information provided by the codes being coupled including, when available, the field’s distribution functions, and the existence of any conservation or positivity constraints. Supported methods of field transfer include radial basis functions applied when limited distribution information is available, and mesh intersection methods when full knowledge of the piecewise distribution of the fields is provided. Methods to support conservative field transfer are available. • Advanced coordinate transformation capabilities that have included curvilinear systems defined in terms of Fourier functions and up to 5 and 6 dimensional spaces. Through the support of multiple FES SciDAC 5 partnerships, PCMS has been used to couple core and edge plasma physics codes, three-dimensional fields to five dimensional plasma codes, and edge plasma codes to neutral transport codes. Error/correction indicators and adaptive simulation workflows: A generalized version of the Zienkiewicz and Zhu projection-based error estimation procedure and associated set of correction indicators to calculated mesh size field has been developed. The error indicator and correction indicator have been used to in the development of multiple adaptive simulation workflows. Adaptive simulation workflows developed in SciDAC partnerships using these procedures include: • Simulation of Maxwell’s equations as needed to understand tokamak RF heating. This workflow was developed in MFEM employing curved, high-order finite elements. • Simulation of ELM formation and pellet injection processes in tokamak plasmas using the M3D-C1 extended MHD code. • Simulation of the motion of land ice using building on the Albany finite element code. ML surrogate material models from multiscale simulations: A new multiscale heat transfer capability is being developed to model 3D chiplet stacks for improved thermal management. The multiscale homogenization strategy is applied to BEOL structures. This has also required developments of advanced modeling tools to extract representative volume elements from microelectronic design files such as GDSII. Additionally, interfaces for machine-learned surrogate material models have been incorporated to study biological tissues. ML-based region detection for hexahedral mesh generation: To enhance automation of the geometry decomposition, an artificial neural net approach to enhance the identification of hexahedral meshable regions has been developed. Core to the approach is encoding topology and geometry model information. To improve the effectiveness of the methods, the metrics used have been extended to include evaluation geometric considerations of the remaining domain after region removal. Coarsening and communication for hierarchical partitioning: Advances graph coarsening methods were developed and integrated into the PuLP/XtraPuLP GPU-based graph partitioner. Strong and weak scaling studies show PuLP/XtraPuLP is scalable, running across up to 512 NVIDIA GPUs without any measurable bottlenecks. Software stewardship for unstructured mesh software components: As part of our software stewardship efforts, we have developed new CI/CD pipelines to test the PUMI, Omega_h, and PUMIParticle software packages on Perlmutter, Frontier and Aurora GPUs using Globus Compute. Substantial progress has been made to make the RPI unstructured mesh software xSDK compliant. Wrapper functions to common MPI operations were added to the interface, and the interface was refactored from the singleton model to user managed PCU instances. Opportunities for training and professional development provided Each of the professional staff, postdocs and graduate students participate in a substantial subset of the following professional development activities: • Weekly meetings of all RPI SCOREC unstructured mesh team members • Weekly meetings on each of the specific unstructured mesh development areas • Yearly FASTMath center review meeting • DOE topic area webinars • Taking/auditing university courses • Attending and presenting at technical meetings • Taking part in training courses • Publishing papers • Engaging undergraduate students in research projects Available Open Source Software The open source RPI unstructured mesh software consist of components designed to support the development of unstructured mesh simulation capabilities that are performance-portable and execute on accelerator-based computers from laptops to exascale supercomputers. The main PUMI components include: • PUMI: A CPU-based unstructured mesh infrastructure supporting scalable conforming mesh adaptation based on generalized local mesh modification • Omega_h: A performance portable (CPUs and GPUs currently) unstructured mesh infrastructure supporting scalable conforming mesh adaptation based on local mesh modification • PUMI-Pic: A performance-portable (CPUs and GPUs currently) unstructured mesh infrastructure that includes scalable mesh and particle data structures and operators to support particle-in-cell simulation codes • PUMI-Tally: A performance-portable (CPUs and GPUs currently) unstructured mesh infrastructure that includes scalable mesh and particle data structures and operators to support Monte Carlo neutral particle transport • PUMI-Fields: A performance-portable (CPUs and GPUs currently) infrastructure to support the storage of and operations on unstructured mesh fields • PUMI-Balance: A hyper-graph based parallel unstructured mesh dynamic load balancing component • polyMPO: Supports all relevant material point method (MPM) operations on spherical centroidal Voronoi tessellations (spherical unstructured polygonal meshes) The open source, performance portable (CPUs and GPUs currently), Parallel Coupler for Multimodel Simulations (PCMS) consist of components designed to support the effective coupling of independent physics simulations codes. PCM components include: • PCMS-Comm: Rendezvous communication pattern component for coupling parallel simulation fields defined on independent partitions of mesh based data • PCMS-Fields: Mesh-to-mesh field transfer component Open source physics simulation codes: • GITRm: Fully 3D unstructured mesh-based global transport code for multi-species impurities using Monte Carlo particle (neutral atom and ion) tracking to simulate the erosion, ionization, migration, and redistribution. • MuMFiM: A multiscale framework for high performance simulations of fibrous materials. Year 5 Products Year 5 Papers [1] Nath, D.D., Younkin, T.R., Guterl, J., Shephard, M.S. and Sahni, O., 2025. A GPU‐ Accelerated 3D Unstructured Mesh Based Particle Tracking Code for Multi‐Species Impurity Transport Simulation in Fusion Tokamaks. Contributions to Plasma Physics, p.e202400073, https://doi.org/10.1002/ctpp.202400073. [2] Merson, J.S., Smith, C.W., Shephard, M.S., Hasan, F.H., Paudel, A., Castillo-Crooke, A.E., Mathew, J. and Elahi, S.M., 2025. PCMS: Parallel Coupler for Multimodel Simulations. Submitted to Computer Physics Communications, arXiv preprint arXiv:2510.18838. [3] Adams, M.F., Wang, P., Merson, J.S., Huck, K. and Knepley, M.G., 2025. A performance portable, fully implicit landau collision operator with batched linear solvers. SIAM Journal on Scientific Computing, 47(2), pp.B360-B381. [4] Huq, M.F., Srinivasaragavan, V.V., Sahni, O. and Curreli, D., 2025. A block-structured nonuniform meshing technique for reducing the degrees-of-freedom in hybrid particle-in- cell plasma simulations. Computer Physics Communications, 313, p.109612. [5] Parvez, N. and Merson, J.S., 2025. A physics preserving neural network based approach for constitutive modeling of isotropic fibrous materials. Engineering with Computers, 41(4), pp.2219-2238. [6] Slota, G.M. and Mandulak, M., 2025. Scaling distributed graph processing to hundreds of GPUs. In International Conference on Parallel Processing (ICPP). [7] Slota, G.M., Mandulak, M., Pandey, U. and Fabius, A., 2025, September. 2D Distributed Label Propagation on 400 GPUs. In 2025 IEEE High Performance Extreme Computing Conference (HPEC) (pp. 1-8). IEEE. Year 5 Presentations [1] J.S. Merson, M.S. Shephard, A. Paudel, F.H. Hasan, U. Riaz, “Stellarator Meshing for XCGS”, HifiStell SciDAC Partnership All-Hands Meeting, Atlanta, GA, October 6, 2024. [2] R.M. Churchhill, et. al., “A Vision for Simulation-based, Multi-fidelity Digital-Twin in Fusion Energy”, APS Division of Plasma Physics Meeting, Atlanta, GA, October 7, 2024. [3] J.S. Merson, M.S. Shephard, A. Paudel, F.H. Hasan, U. Riaz, “Meshing advances for Tokamak Simulations”, CEDA SciDAC Partnership All-Hands Meeting, Atlanta, GA, October 9, 2024. [4] L. Bertagna, C.W. Smith, M. Perego, M.S. Shephard, “Mesh Adaptation for Ice Sheet Modeling”, SIAM Conference on Computational Science and Engineering (CSE25), Fort Worth, TX, March 4, 2025. [5] O. Sahni, D.D. Nath, T.R. Younkin, M.S. Shephard, “Multiscale Global Impurity Transport in Fusion Devices”, SIAM Conference on Computational Science and Engineering (CSE25), Fort Worth, TX, March 5, 2025. [6] J.S. Merson, A. Paudel, F.H. Hasan, M. McCall, C.W. Smith, A.E. Castillo-Crooke, M.S. Shephard, “PCMS: A Geometry and Discretization Aware Coupling Tool for Fusion Devices”, SIAM Conference on Computational Science and Engineering (CSE25), Fort Worth, TX, March 6, 2025. [7] C. Zhang, G. Diamond, C.W. Smith, M.S. Shephard, “Developing Particle Simulation Code in Fusion Plasma Physics and Gas Dynamics Using Unstructured Mesh Infrastructure and PUMIPic”, SIAM Conference on Computational Science and Engineering (CSE25), Fort Worth, TX, March 6, 2025. [8] M.S. Shephard, J.S. Merson, O. Sahni, M.W. Beall, C.W. Smith, D.D. Nath, U. Riaz, F. H. Hasan, S. Tendulkar, “Advances in FASTMath Unstructured Mesh Components in Support of DOE Fusion Simulation Codes”, SIAM Conference on Computational Science and Engineering (CSE25), Fort Worth, TX, March 7, 2025. [9] J.S. Merson, M.S. Shephard, A. Paudel, F.H. Hasan, U. Riaz and S.M. Elahi, “Geometry, Meshing and Coupling for Stellarator Simulations”, HifiStell All-Hands Meeting, Princeton University, Princeton, NJ, March 22, 2025. [10] M.S. Shephard, J.S. Merson, O. Sahni, A.Y. Joshi, U. Riaz, D.D. Nath, M.W. Beall and S. Tendulkar “Geometry/Meshing and GITRm for Stellarator RF Simulations”, RF-PMI All- Hands Meeting, Princeton Plasma Physics Laboratory, Princeton, April 2, 2025. [11] J.S. Merson, F.H. Hasan, A.Y. Joshi, C.W. Smith, M.S. Shephard, P.K. Romano, P.C. Shriwise, G.J. Wilkie, R.M. Churchill, “GPU Acceleration of Unstructured Mesh Based Tallies (and Transport) in OpenMC”, OpenMC Application to Tokamak Neutronics Analysis Meeting, Argonne National Laboratory, May 28, 2025. [12] F.H. Hasan, P.K. Romano, P. Shriwise, A.Y. Joshi, R.M. Churchill, H. Belanger, C.W. Smith, M.S. Shephard and J.S. Merson, GPU Enabled Unstructured Mesh Tallies for Fusion Neutron Transport Simulations, 18th US Congress on Computational Mechanics, Chicago, IL, July 23, 2025. [13] D.D. Nath, A.E. Castillo-Crooke, M.S. Shephard, A. Turner, D. Sulsky and O. Sahni, Accelerator/GPU Support for Sea Ice Simulations in MPAS Framework Within Energy Exascale Earth System Model, 18th US Congress on Computational Mechanics, Chicago, IL, July 22, 2025. [14] J.S. Merson, A. Paudel, F.H. Hasan, S.M. Elahi, A.E. Castillo-Crooke, C.W. Smith, M.S. Shephard, Enabling High-Performance Coupled Multiphysics Fusion Simulations with the Parallel Coupler for Multi-Model Simulations (PCMS), SIAM Annual Meeting, Montreal, Canada, July 29, 2025. [15] M.S. Shephard, J.S. Merson, O. Sahni, C.W. Smith, D.D. Nath, U. Riaz, F.H. Hasan, A.E. Castillo-Crooke, A.Y. Joshi, M.W. Beall, O. Klaas, R. Nastasia, S. Tendulkar, Advances in FASTMath Unstructured Mesh Components in Support of DOE Fusion Simulation Codes, SIAM Annual Meeting, Montreal, Canada, July 30, 2025. [16] M.S. Shephard, M. Stowell, FASTMath Unstructured Mesh Technologies, ATPESC 2025, Chicago, Il, August 5, 2025. [17] J.S. Merson, M.S. Shephard, U. Riaz, B.J. Sturdevant, A. Paudel, F.H. Hasan, S.M. Elahi, A.E. Castillo-Crooke, C.W. Smith, GPU-Accelerated Tallies for Unstructured Meshes, StellFoundry All-Hands Meeting, Oak Ridge National Laboratory, Oak Ridge, TN, Aug. 6-7, 2025. [18] J.S. Merson, M.S. Shephard, U. Riaz, B.J. Sturdevant, A. Paudel, F.H. Hasan, S.M. Elahi, A. Castillo-Crooke, C.W. Smith, Tokamak Modeling & Meshing Software, StellFoundry All-Hands Meeting, Oak Ridge National Laboratory, Oak Ridge, TN, Aug. 6-7, 2025. [19] M.S. Shephard and J.S. Merson, “Unstructured Mesh Tools for Magnetically Confined Fusion System Simulations”, Virtual seminar for the Simons Collaboration on Hidden Symmetries and Fusion Energy, Sept. 11, 2025. Year 5 Posters [1] M.W. Beall, A.E. Castillo-Crooke, F.H. Hasan, A.Y. Joshi, O. Klaas, J.S. Merson, D.D. Nath, A. Paudel, U. Riaz, O. Sahni, E.S. Seol, M.S. Shephard, C.W. Smith, S. Tendulkar, “Support of Geometry and Meshing for Fusion Energy System Simulation Codes”, APS Division of Plasma Physics Meeting, Atlanta, GA, October 9, 2024 [2] Posters as member of listed team at the APS Division of Plasma Physics Meeting, Atlanta, GA, October 7-11, 2024: a. “Overview of Research in the SciDAC-5 Partnership for Advanced Simulation of RF - Plasma - Material Interactions” b. “Simulation of plasma turbulence in stellarator equilibria with the global gyrokinetic particle-in-cell code XGC” [3] F.H. Hasan, G. Wilke, P.K. Romano, P. Shriwise, R.M. Churchill, C.W. Smith, M.S. Shephard and J.S Merson, “Integrating OpenMC and PUMIPic to Enable GPU Acceleration of Unstructured Mesh Tallies for Monte Carlo Transport Simulations”, Open Source Software for Fusion Energy (OSSFE) Conference, virtual meeting, March 18, 2025. [4] C.W. Smith, E. Abeysinghe, J.M. TenBarge, J. Juno, L. Jayathilake, Y. Jayawardana, L. and M.S. Shephard, “Plasma and Space Weather Science Gateway for Education and Research”, 2025 NSF CSSI/CyberTraining/SCIPE PI Meeting, Denver, CO, 7/27/25-7/29/25. [5] E.G. Boman, A.E. Castillo-Crooke, J.S. Merson, O. Sahni, M.S. Shephard, C.W. Smith, B.J. Sturdevant, “FASTMath Unstructured Mesh Components, SciDAC PI meeting, Rockville, MD, 9/17/25.