Exascale In-situ visualization for Astronomy & Cosmology
Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Emiliano Tramontana · 2025
Read Exascale In-situ visualization for Astronomy & Cosmology by Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Emiliano Tramontana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24545.txt.
Introduction
Nicola Tuccari leads a team of five authors on visualizing huge simulations. Cosmology simulations now make more data than disks can hold. Writing all that data to disk has long slowed supercomputing. In-situ processing studies data while the simulation still runs. Nothing full is stored, so the storage bottleneck disappears.
Their tool, Hecuba, streams simulation data through a distributed database. It feeds viewing tools live, without files in between. They link Hecuba to ChaNGa, a high-performance cosmology simulator. N-body results, point clouds of stars and dark matter, appear on screen live. Viewers use ParaView and VisIVO, two known analysis tools. The work shows exascale runs can be watched as they unfold.
Catalog record
| CREATOR | Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Emiliano Tramontana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24545
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exascale In-situ visualization for Astronomy & Cosmology Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Emiliano Tramontana Abstract Modern simulations and observations in Astronomy & Cosmology (A&C) produce massively large data volumes, posing significant challenges for storage, access and data analysis. A long-standing bottleneck in high-performance computing, especially now in the exascale era, has been the requirement to write these large datasets to disks, which limits the performance. A promising solution to this challenge is in-situ processing, where analysis and visualization are performed concurrently with the simulation itself, bypassing the storage of the simulation data. In this work, we present new results from an approach for in-situ processing based on Hecuba, a framework that provides a highly distributed database for streaming A&C simulation data directly into the visualization pipeline to make possible on-line visualization. By integrating Hecuba with the high-performance cosmological simulator ChaNGa, we enable real-time, in-situ visualization of N-body simulation results using tools such as ParaView and VisIVO. [orcid=0009-0004-7802-2602, [email protected], ] *1 1 [orcid=0000-0002-5574-2787, [email protected], ] 1 [orcid=0000-0003-2357-7796, [email protected] ] 1 [[email protected] ] 1 [orcid=0000-0002-7169-659X, [email protected] ] 1 keywords visualization ,cosmology ,in-situ visualization ,high performance computing † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: ITADATA-WS 2025: The 4 {}^{\text{th}} Italian Conference on Big Data and Data Science – Workshops, September 9–11, 2025, Turin, Italy † † address: Università di Catania, Dipartimento di Matematica e Informatica, Catania, Italy † † address: INAF Astrophysical Observatory of Catania, Catania, Italy † † address: Barcelona Supercomputing Center, Barcelona, Spain † † These authors contributed equally. 1 Introduction Astrophysical observations and simulation codes executed on high-performance supercomputers generate massive data volumes, often reaching petabyte scales. Managing such data poses considerable challenges that must be addressed to enable scientific discoveries Hey et al. 2009 . Emerging pre-exascale infrastructures offer new possibilities for scaling applications in Astronomy and Cosmology (A&C), in particular to support high-performance visualization, which is essential for understanding simulation outcomes. Traditionally, scientific visualization has relied on post-processing, where simulation data is first stored permanently on disk and only later retrieved for analysis and visualization. However, this type of workflow can be inefficient given the massive amount of data generated. An alternative processing paradigm is in-situ visualization, which enables the analysis and rendering of data in real time, concurrently to the execution of the simulation, thus allowing users to generate the visualization as the data is generated. Processing such massive datasets is only feasible with the use of HPC resources. A major bottleneck in achieving high performance and scalability comes from the reliance on parallel file systems. Additionally, file-based workflows often introduce execution patterns that require strict synchronization and complex code, limiting the capability to adapt to new requirements or hardware changes. A practical and emerging alternative for scientific computing are Key-Value (KV) databases, as they are particularly well suited for handling time-series and spatial datasets. Moreover, they allow to analyze partial results and react, for instance, by discarding a cosmological simulation as soon as a certain event occurs. By leveraging Hecuba, we enable a specific form of in-situ visualization known as in-transit visualization Moreland et al. 2011 , where analysis and visualization are executed on dedicated I/O nodes. These I/O nodes receive the full simulation results, but they avoid to store them, but rather they generate new information from analysis or provide run-time visualization. In previous work Tuccari et al. 2025 , we presented the first steps of the integration of the Hecuba platform within the ChaNGa high-performance cosmological simulator and the in-situ visualization of its N-body results with the ParaView and VisIVO tools. In this paper, we present further developments, focusing on the ParaView plugin and reducing VisIVO dependence on file-based data storing, to enable in-situ visualization. 2 Related Works In-situ visualization has become increasingly important in the field of high-performance computing, as simulations now produce massive volumes of data that are impractical to store and analyze with just post-processing. Several surveys in the literature provide comprehensive overviews of existing techniques and tools for in-situ visualization, highlighting their central role in scientific workflows that demand real-time analysis while reducing I/O overheads Rivi et al. 2012 . Three main approaches to in-situ visualization can be identified: tighty coupled, loosely coupled and hybrid. In the tightly coupled model, visualization routines are directly embedded within the simulation code, which allows efficient memory access and minimal latency but requires synchronous execution and additional memory resources. The loosely coupled model, by contrast, separates computation and visualization into independent processes that may run on different resources and communicate through shared memory or network interfaces. Finally, hybrid approaches attempt to balance the advantages of both by performing data reduction during the simulation and forwarding the processed results to external resources for further analysis. Several frameworks have been developed to support these approaches. One example is VisIt Libsim Whitlock et al. 2011 , which implements the tightly coupled model by enabling simulation codes to function as a compute engine directly linked to VisIt. Another widely adopted framework is ParaView Catalyst Ayachit et al. 2015 , which provides flexible integration mechanisms within simulation codes and supports both tightly and loosely coupled configurations. These solutions have demonstrated the feasibility and advantages of in-situ workflows, but they also highlight persistent challenges, such as the complexity of code instrumentation and the risk of resource contention between simulation and visualization tasks. 3 Data, Tools and Methodology 3.1 ChaNGa Cosmological Data ChaNGa 1 1 1 https://github.com/N-BodyShop/changa (Charm N-body GrAvity solver) is a parallel code designed for collisionless N-body simulations in astrophysics. It is capable of performing both cosmological simulations with periodic boundary conditions in comoving coordinates and simulating isolated stellar systems. It also includes hydrodynamics modeling using the Smooth Particle Hydrodynamics (SPH) method. Gravity calculations are computed using the Barnes-Hut tree algorithm, which includes hexadecapole expansion of nodes and Ewald summation for handling periodic forces. For time integration, ChaNGa uses a leapfrog integrator that allows each particle to evolve with its own individual timestep. ChaNGa stores data using Tipsy, a binary file format used primarily in astrophysical simulations to store snapshot data of N-body systems, it can store three types of particles: gas, dark matter and star. 3.2 Hecuba Hecuba 2 2 2 https://github.com/bsc-dd/hecuba is a set of tools and interfaces that aim to facilitate the management and utilization of persistent data for Big Data applications. Figure 1: Hecuba architecture implementing an Object Mapper for Apache Cassandra support both off-line and on-line ChaNGa data analysis. Hecuba implements an Object Mapper for Apache Cassandra Lakshman and Malik 2010 (a recognized NoSQL database) that enables programmers to interact with it and its data as regular in-memory objects. Currently, Hecuba implements an interface for Python and C++ programming languages. In order to provide a mechanism to synchronize run-time visualizations, we are extending Hecuba to implement a lambda architecture integrating Apache Kafka 3 3 3 Apache Kafka, https://kafka.apache.org/ . Lambda architecture is a data-processing structure defined to speed up online analysis for Big Data applications. With this approach, Hecuba allows at the same time to persist the generated data in a key-value datastore and also to produce a