Experiences of Commercial Supercomputing in Radio Astronomy Data Processing Journal: Astronomy & \& Computing
I. P. Kemp, S. J. Tingay, S. D. Midgely, D. A. Mitchell · 2025
Read Experiences of Commercial Supercomputing in Radio Astronomy Data Processing Journal: Astronomy & \& Computing by I. P. Kemp, S. J. Tingay, S. D. Midgely, D. A. Mitchell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17657.txt.
Introduction
Ian Kemp, Steven J Tingay, Stuart Midgley, and Daniel Mitchell report on commercial supercomputing for radio astronomy. Their 2025 arXiv paper starts from a market shift. Commercial high-performance systems now outgun the machines Australian radio astronomy relies on. One even exceeds what the Square Kilometre Array's science processors are expected to need.
The authors surveyed scientists and providers with commercial HPC experience in astronomy and kindred data-heavy fields, through semi-structured interviews. They add lessons from two earlier case studies with very different data and processing demands. A thematic analysis pulls the key messages out of the interviews. Commercial systems offer real advantages in access and scale, the study finds, alongside new risks in cost and control. The paper gives observatories a grounded brief for deciding what to rent, what to own, and what to keep in-house.
Catalog record
| CREATOR | I. P. Kemp, S. J. Tingay, S. D. Midgely, D. A. Mitchell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Experiences of Commercial Supercomputing in Radio Astronomy Data Processing Journal: Astronomy & \& Computing I. P. Kemp, S. J. Tingay, S. D. Midgely, D. A. Mitchell Abstract The ongoing exponential growth of computational power, and the growth of the commercial High Performance Computing (HPC) industry, has led to a point where ten commercial systems currently exceed the performance of the highest-used HPC system in radio astronomy in Australia, and one of these exceeds the expected requirements of the Square Kilometre Array (SKA) Science Data Processors. In order to explore implications of this emerging change in the HPC landscape for radio astronomy, we report results from a survey conducted via semi-structured interviews with 14 Australian scientists and providers with experience of commercial HPC in astronomy and similar data intensive fields. We supplement these data with learnings from two earlier studies in which we investigated the application of commercial HPC to radio astronomy data processing, using cases with very different data and processing considerations. We use the established qualitative research approach of thematic analysis to extract key messages from our interviews. We find that commercial HPC can provide major advantages in accessibility and availability, and may contribute to increasing researchers’ career productivity. Significant barriers exist, however, including the need for access to increased expertise in systems programming and parallelisation, and a need for recognition in research funding. We comment on potential solutions to these issues. 1 Introduction 1.1 Purpose of this study This work is part of ongoing research to explore the applicability of commercial supercomputing to next generation radio astronomy. In this paper we will show that multiple commercial computing providers could today host infrastructure with processing power of the scale required for significant future applications, including the Square Kilometre Array (SKA) Science Data Processors (SDP). We argue that raw processing power is not sufficient, and that there are a range of advantages and disadvantages associated with commercial High Performance Computing (HPC) which would need to be considered if this source of computing power is to find a role in radio astronomy. This note on practice has an Australian focus, on the grounds that Australia is currently on the cutting edge of radio astronomy instrumentation, with the Australian Square Kilometre Pathfinder (ASKAP), the Murchison Widefield Array (MWA) and the forthcoming low frequency telescope of the Square Kilometre Array (SKA-low). The Australian research community has extensive experience at the intersection of HPC and large-scale radio astronomy, and has been involved in planning future development in this area via the Australian SKA Regional Data Centre (AusSRC) survey ( Quartermaine and Kitaeff 2020 ) and the AAS Decadal planning process ( Australian Academy of Science 2025 ) . In this regard the Australian radio astronomy community may not be unique but its experiences may be representative, and help inform the wider international community as we adopt a new generation of radio telescopes. To identify the key advantages and issues of commercial HPC, we report results from a qualitative research study carried out via interviews with a number of Australian researchers with recent practical experience of commercial supercomputing in Australia. We have not intended to develop an authoritative summary of the views of the Australian radio astronomy community. Also, we have not sought to collect data on or critique existing widely-used publicly provided HPC systems, nor provide a comparison between public and commercial HPC. We hope to disseminate to the wider radio astronomy community some of the experiences and lessons learned by researchers with practical experience of commercial HPC. We have applied a rigorous qualitative research technique borrowed from the social sciences. We have generated knowledge by interviewing astronomers and computing specialists who have used commercial supercomputing in astronomy or similar data-intensive science, to capture their learnings and reflections. We have also incorporated learnings from two of our own recent case studies using commercial HPC. 1.2 Real-time data processing Modern radio astronomy is critically dependent on HPC. As an example, the 36-dish Australian Square Kilometre Array Pathfinder (ASKAP) in Western Australia ( Schinckel et al. 2012 ; Hotan et al. 2021 ) , sends up to 2.5 GB/s of visibility data to a Science Data Processor system (SDP), which carries out calibration, imaging, source finding and cataloguing, and stores the processed data to the CASDA repository ( Huynh et al. 2020 ) . All data are processed through the SDP, which is implemented on the Setonix supercomputer at the Pawsey Supercomputing Centre in Perth 1 1 1 https://pawsey.org.au/ . This machine, rated at 27 PFLOP/s processing power, is the most performant publicly disclosed computer in Australia, and at November 2024 appeared as number 45 in the world Top500 list ( Top500.org 2024 ) . It is a general purpose system used for a variety of other processing. The ASKAP SDP has access to 180 dual 2.45GHz AMD EPYC “Milan” 64-core CPU nodes, each delivering 2.5TFLOP/s ( Trader 2021 ) , so we estimate that the SDP requirement is met with 450 TFLOP/s. The second major SKA precursor in Australia, the 256-tile Murchison Widefield Array (MWA) ( Lonsdale et al. 2009 , Tingay et al. 2013 , Wayth et al. 2018 ), is capable, with its recently upgraded correlator, of sustainably sending 22.8GB/s ( Sleap 2025 ) , however in its normal operating modes it transmits 2.3GB/s. In contrast to ASKAP, visibility data from MWA is sent direct to archive at the Pawsey centre. Basic processing including calibration and RFI flagging is done later, either using the All-sky Virtual Observatory (ASVO) system implemented on Setonix 2 2 2 https://asvo.mwatelescope.org/ , or by astronomers using their own systems (which may include users’ merit based time allocation on Setonix itself). The MWA ASVO system processes data via 10 CPU nodes, which we estimate provide 25 TFLOP/s. Note that the ASKAP SDP needs to be able to process data in real time, whereas the MWA equivalent processes data on demand. Data flows and SDP processing requirements are soon to increase by two orders of magnitude as the next generation of radio telescopes are implemented. The SDP for the Next Generation Very Large Array (ngVLA, currently at the design stage) will be required to process visibility data at up to 133 GB/s ( Hiliart 2019 ) . The 197-dish Square Kilometre Array Mid-Frequency telescope (SKA-Mid, currently under construction) will send up to 292 GB/s of visibility data to the SDP ( McMullin et al. 2022 ) , which will implement an expanded set of workflows to produce a wider range of science-ready data products than does today’s ASKAP ( Nikolic and SDP Consortium 2014 ) . The SKAO organisation currently estimates that the SDP for each of the two SKA Phase I telescopes would be appropriately sized at 135PFLOP/s ( SKAO 2024a ) . 1.3 Non-real-time data processing The processing of data streaming from the telescope is only one use of HPC. Astronomers carry out extensive processing to extract scientifically useful information from the standardised science ready products generated by the SDP, or to carry out custom calibration and imaging. This processing presents a wide and unpredictable range of requirements. Some useful data on non-real-time processing were provided by the AusSRC Radio Astronomy Data User Community Survey Report ( Quartermaine