In-Situ High Performance Visualization for Astronomy & Cosmology
Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Robert Wissing, Sijing Shen, Emiliano Tramontana · 2025
Read In-Situ High Performance Visualization for Astronomy & Cosmology by Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Robert Wissing, Sijing Shen, Emiliano Tramontana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24547.txt.
Introduction
Nicola Tuccari leads a team of seven authors on in-situ visualization. Astronomy and cosmology face a flood of simulation and telescope data. Saving full petascale results to disk blocks fast analysis. Filtering data first cuts detail or volume before viewing. The team run viewing at the same time as the simulation.
Hecuba, their framework, streams data through a distributed database. The stream goes straight to online visualization, skipping storage. They join Hecuba with ChaNGa, a fast cosmology simulator. Its N-body output shows live in ParaView and VisIVO. N-body output tracks many particles that pull on each other. The demo shows scientists can explore runs while they compute. That live view helps catch errors and guide costly runs early.
Catalog record
| CREATOR | Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Robert Wissing, Sijing Shen, Emiliano Tramontana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24547
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In-Situ High Performance Visualization for Astronomy & Cosmology Nicola Tuccari, Eva Sciacca, Yolanda Becerra, Enric Sosa Cintero, Robert Wissing, Sijing Shen, Emiliano Tramontana Abstract The Astronomy & Cosmology (A&C) community is presently witnessing an unprecedented growth in the quality and quantity of data coming from simulations and observations. Writing results of numerical simulations to disk files has long been a bottleneck in high-performance computing. To access effectively and extract the scientific content of such large-scale data sets appropriate tools and techniques are needed. This is especially true for visualization tools, where petascale data size problems cannot be visualized without some data filtering, which reduces either the resolution or the amount of data volume managed by the visualization tool. A solution to this problem is to run the analysis and visualization concurrently (in-situ) with the simulation and bypass the storage of the full results. In particular we use Hecuba, a framework offering a highly distributed database to stream A&C simulation data for on-line visualization. We will demonstrate the Hecuba platform integration with the Changa high performant cosmological simulator and the in-situ visualization of its N-body results with the ParaView and VisIVO tools. 1 INAF Astrophysical Observatory of Catania, Catania, Italy; [email protected] 2 Università di Catania, Dipartimento di Matematica e Informatica, Catania, Italy 3 Barcelona Supercomputing Center, Barcelona, Spain 4 Institute of Theoretical Astrophysics, University of Oslo, Oslo, Norway 1 Introduction Astrophysical observations and simulation codes executed on high-performance supercomputers generate massive data volumes, often reaching petabyte scales. These enormous datasets present significant challenges in terms of storage, access, and analysis, which are crucial for facilitating scientific discoveries Hey et al. 2009 . Pre-exascale systems provide remarkable opportunities to scale HPC applications in Astrophysics and Cosmology (A&C), particularly for the high-performance visualization needed to interpret their results. The dominant paradigm for scientific visualization over the years has been post-processing. This approach involves saving the data generated by simulation codes to permanent storage, so visualization and analysis tools need to load this data after it is stored. An emerging processing paradigm is in-situ visualization. It involves visualizing and analyzing data as it is generated, allowing the user to create graphical output while the simulation executes. Only HPC resources have the capability to process such vast amounts of data. However, the primary source of performance and scalability challenges lies in parallel file systems. Additionally, working with files imposes rigid application workflows that require synchronization and complex code, making it difficult to adapt to new requirements or hardware changes. Key-Value (KV) databases are widely used in data analytics and represent a realistic alternative. They are well-suited for scientific applications such as time-series or spatial data. Furthermore, they allow analyzing partial results and react, for instance, by discarding a cosmological simulation as soon as a certain event occurs. In this work we exploit Hecuba that allows for a particular type of in-situ visualization: in-transit visualization Moreland et al. 2011 , i.e. analysis and visualization is run on I/O nodes that receive the full simulation results but write information from analysis or provide run-time visualization. We focus on the Hecuba platform integration with the Changa high performant cosmological simulator and the in-situ visualization of its N-body results with the ParaView and VisIVO tools. 2 Data, Tools and Methodology 2.1 ChaNGa Cosmological Data ChaNGa 1 1 1 https://github.com/N-BodyShop/changa (Charm N-body GrAvity solver) is a code that performs collisionless N-Body simulations. It supports cosmological simulations with periodic boundary conditions in comoving coordinates, as well as simulations of isolated stellar systems. Additionally, it can incorporate hydrodynamics using the Smooth Particle Hydrodynamics (SPH) method. Gravity calculations are performed using a Barnes-Hut tree algorithm, which includes hexadecapole expansion of nodes and Ewald summation for handling periodic forces. The code employs a leapfrog integrator with individual timesteps assigned to each particle for time integration. ChaNGa stores data using the Tipsy data format. 2.2 Hecuba Hecuba 2 2 2 https://github.com/bsc-dd/hecuba is a set of tools and interfaces that aims to facilitate the management and utilization of persistent data for Big Data applications. Hecuba implements an Object Mapper for Apache Cassandra Lakshman & Malik 2010 (a recognized NoSQL database) and thus allows programmers to use a common interface to access data as regular in-memory objects, regardless they are persistent (stored in disk) or they are in-memory data. Currently Hecuba implements an interface for Python and C++ programming languages. In our current work we are extending Hecuba to also support a lambda architecture Marz & Warren 2015 which is a data-processing architecture defined to speed up online analysis for Big Data applications, without losing the ability to persist the output data of the applications. Thus, Hecuba will be able to support both off-line and on-line data analysis. Notice that the programmer can use the same interface to access the data whether it is in memory, in the storage or it is coming through a stream: only needs to modify the class definition of the object to indicate which kind of object it is. Figure 1. : Hecuba architecture implementing an Object Mapper for Apache Cassandra support both off-line and on-line ChaNGa data analysis. 2.3 Paraview Paraview 3 3 3 https://www.paraview.org/ Ahrens et al. 2005 is a tool designed to support the visualization and analysis of large scientific data sets. To this end, Paraview supports, for example, hardware-accelerated rendering or parallel rendering on shared and distributed memory machines. The decoupled architecture of Paraview allows the use of remote execution of a Paraview server to execute the data acquisition together with the data generating application (saving in this way moving a large amount of information between machines) and the data visualization in a desktop machine (with a more limited amount of memory and disk). Another useful capability of Paraview is its ability to support user plugins to customize the data reading and visualization process. In this work, we have implemented a custom Paraview plugin that uses the Hecuba interface to implement online acquisition of the data to visualize. 2.4 VisIVO VisIVO is an integrated suite of tools and services specifically designed for the Virtual Observatory and for creating customized views of 3D renderings from astrophysical datasets Sciacca et al. 2015 . VisIVO Server 4 4 4 https://github.com/VisIVOLab/VisIVOServer consists of three core components: VisIVO Importer, VisIVO Filter, and VisIVO Viewer. We are currently focusing on evolving VisIVO and enabling it to take advantage of new HPC facilities. In this work we explored the possibilities given by integrating Hecuba with VisIVO Server. By interacting with a highly distributed database, VisIVO would gain the capability to perform in-situ visualization. 3 Preliminary Results and Future Works The current preliminary results for VisIVO include the development of a prototype featuring a novel Importer that retrieves simulated data from a distributed database using Hecuba’s API. This prototype will serve as the basis for implementing a complete pipeline that utilizes all VisIVO Server modules to generate visualizations from the data retrieved by the reader and passed through one of VisIVO’s filters.