STRIELAD - A Scalable Toolkit for Real-time Interactive Exploration of Large Atmospheric Datasets
Simon Schneegans 1, Lori Neary 2, Markus Flatken 1, Andreas Gerndt 1 · 2025
Read STRIELAD - A Scalable Toolkit for Real-time Interactive Exploration of Large Atmospheric Datasets by Simon Schneegans 1, Lori Neary 2, Markus Flatken 1, Andreas Gerndt 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00033.txt.
Introduction
Simon Schneegans and co-authors present STRIELAD, a toolkit for exploring huge atmospheric data sets live in a 2025 paper, based on the head byline. Better computers and models now simulate weather and climate in fine detail, but the output volume strains viewing and analysis. Their system aims to keep exploration interactive at full scale.
Parallel, distributed feature extraction runs on high performance machines, while smart level of detail rendering keeps the display fluid. Analysts can roam a large simulation in real time instead of waiting on batch jobs. The toolkit targets the grand data challenges that modern weather modeling creates, pairing heavy back end compute with a responsive front end for scientists.
Catalog record
| CREATOR | Simon Schneegans 1, Lori Neary 2, Markus Flatken 1, Andreas Gerndt 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STRIELAD - A Scalable Toolkit for Real-time Interactive Exploration of Large Atmospheric Datasets Simon Schneegans 1, Lori Neary 2, Markus Flatken 1, Andreas Gerndt 1 Abstract Technological advances in high performance computing and maturing physical models allow scientists to simulate weather and climate evolutions with an increasing accuracy. While this improved accuracy allows us to explore complex dynamical interactions within such physical systems, inconceivable a few years ago, it also results in grand challenges regarding the data visualization and analytics process. We present STRIELAD, a scalable weather analytics toolkit, which allows for interactive exploration and real-time visualization of such large scale datasets. It combines parallel and distributed feature extraction using high-performance computing resources with smart level-of-detail rendering methods to assure interactivity during the complete analysis process. \onlineid 0 \vgtccategory Research \vgtcinsertpkg \teaser Real-time volumetric visualization of ice, water, and rain bands using our sub-volume selection mechanism in the graphical user interface. In addition, large terrain data is combined with the weather simulation to analyze their correlations. \CCScatlist \CCScat C.2.4Distributed SystemsClient/ServerDistributed applications; \CCScat I.3.2Graphics SystemsDistributed/network graphicsRemote systems; \CCScat I.6.6Simulation Output Analysis; \CCScat J.2Physical sciences and engineeringEarth and atmospheric sciences Motivation The substantial progress in high-performance computing (HPC) together with an improved understanding of natural processes enable scientists to model and simulate physical phenomena with an increasing accuracy and complexity. A major drawback of these technological advances is the growing gap between the possibilities to simulate and to analyze the produced data. It is not only a challenge of data complexity; the sheer amount of data, generated e.g. by numerical simulations, require tremendous effort to enable efficient data analysis and knowledge discovery. Especially interactive and explorative approaches are promising but also challenging. Yet they are important for urgent decision making in various fields such as traffic- or catastrophe management. Only when these gaps are closed, the full potential for knowledge discovery can be exploited. STRIELAD, a scalable toolkit for real-time interactive exploration of large atmospheric datasets, will enable end-users to explore such large volumes of simulation data in real-time. It combines smart data acceleration structures, parallel and distributed processing, innovative data selection and interaction mechanisms. This is demonstrated with the 2.9 TB weather simulation dataset provided for the 2017 IEEE visualization contest. Additionally, this data is combined with various other data sources such as flight trajectories and geographical information obtained from satellite imagery using DLR’s terrain rendering software [ 7 ] . The goal of this paper is not to answer all given questions of the visualization contest. It is rather a description of STRIELAD, which in turn can be used to find those answers by real-time exploration. 1 Scalable Visualization Toolkit A lot of applications for scientific visualization of simulation data are available on market. Some of them are scalable to enable the visualization of large-scale data [ 2 ] [ 4 ] . Furthermore, these applications often provide an extensive feature set for various application domains. However, to our knowledge none of them enables spatio-temporal exploration of huge simulation data with interactive frame rates and feature extraction in range of milliseconds. With STRIELAD, end-users are able to interactively select sub-volumes of a dataset and apply boolean operations on them. For each sub-volume, several scalar thresholds or domain limits can be defined. To speed-up feature extraction, the processing is importance driven, distributed and highly parallel. Combined with level-of-detail rendering and immediate streaming of result data, this creates a very responsive and interactive system. A shader editor in the graphical user interface (GUI) allows to control the appearance of the defined sub-volumes. The shading possibilities range from informative to realistic. Figure 1 depicts the high-level architecture of the framework. A frontend application performs real-time rendering and user interaction. The GUI enables end-users to define multiple volumes of interests (depicted in \autoref fig:teaser). These sub-volumes are specified by several lower and upper scalar limits. Based on this information, the user’s position and the simulation time, requests are compiled and sent to a backend application. There, feature extraction is performed on an octree which has been generated in a pre-processing step. Furthermore, the user has the ability to modify shader code at runtime via the GUI. This allows for powerful coloring and shading of the received geometry. Figure 1: Distributed visualization setup: The frontend determines nodes for parallel processing. The backend executes the feature extraction on a HPC cluster system and streams results back. 1.1 Preprocessing Before data can be explored with STRIELAD, a preprocessing step generates the required acceleration structure. In this step, the rectlinear input dataset is partitioned into many equally sized blocks (leaf nodes), and an octree is generated by sub-sampling (see \autoref fig:octree). The additional memory requirement for the inner nodes of the octree is approximately 15 % of the size of the original data. This is about 400 GB for the visualization contest dataset. Since several terabytes of data are read and written during the preprocessing, the actual execution time strongly depends on disc speed and may take several hours. While the size of the individual octree nodes has no influence on the memory requirements of the acceleration structure, it significantly influences the visualization performance. If the size is small, many nodes are generated. As a consequence, feature extraction in the backend is highly parallel and visualization updates are received with low latency. However, since many nodes have to be rendered by the frontend, a lot of draw calls have to be issued and therefore the communication with the graphics driver may become a bottleneck. The ideal node size depends on disc speed, backend processing power, network speed, CPU and GPU performance of the frontend. We achieved very good results with a node size of 20\times 20\times 20 cells. This results in seven octree levels and a total node count of 51.393 per time step. 1.2 Parallel feature extraction The parallel and distributed feature extraction application is based on Viracocha [ 5 ] . As depicted in \autoref fig:viracocha, our application implements a master-slave parallelization approach. The master process receives user queries from a frontend application, dissipates the requests into small independent work items, and distributes these work items to n slave instances for parallel processing. Furthermore, an integrated caching approach increases the I/O bandwidth for repeated data processing. We enhanced the scheduling approach of Viracocha to update the processing order of a running computation. This allows to fully change the processing order, e.g. discard previously visible data, insert new items for processing, or rearrange the data processing based on priority values. The slave processes execute a boolean intersection algorithm for sub-volume extraction. The algorithm uses VTK [ 6 ] to load and process the individual octree nodes. It gets a description of pipelined intersection \cap operations defined by the limits per sub-volume. First, the dataset is clipped according to each limit, then the result is triangulated and normals are generated. Finally, it is transferred back to the frontend application for rendering. Since multiple sub-volumes can be defined on the frontend, union \cup operations are also possible. 1.3 Level-of-detail rendering Figure 2: The original dataset is partitioned into many small data blocks (left). Then, by combining eight blocks into one, an octree is generated (center). When rendering, a set