Reproducibility of machine learning analyses of 21 cm reionization maps
K. Sooknunan, E. Chapman, L. Conaboy, D. J. Mortlock, J. R. Pritchard · 2024
Read Reproducibility of machine learning analyses of 21 cm reionization maps by K. Sooknunan, E. Chapman, L. Conaboy, D. J. Mortlock, J. R. Pritchard (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15893.txt.
Introduction
Sooknunan and co-authors test in 2024 whether machine learning tools for early universe maps can be trusted when the training data change. Such maps, read at a key radio wavelength, chart when the first stars lit up. Networks trained on one simulation often learn its quirks, not the physics, the team warns.
They rebuilt past networks and ran them on two unlike simulation sets. The networks often failed on data they had not trained on, missing the gas state and key traits. Case studies show what helps and what hurts. The authors stress that users must check where a model may be applied.
Catalog record
| CREATOR | K. Sooknunan, E. Chapman, L. Conaboy, D. J. Mortlock, J. R. Pritchard |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reproducibility of machine learning analyses of 21 cm reionization maps K. Sooknunan, E. Chapman, L. Conaboy, D. J. Mortlock, J. R. Pritchard Abstract Machine learning (ML) methods have become popular for parameter inference in cosmology, although their reliance on specific training data can cause difficulties when applied across different data sets. By reproducing and testing networks previously used in the field, and applied to 21cmFast and Simfast21 simulations, we show that convolutional neural networks (CNNs) often learn to identify features of individual simulation boxes rather than the underlying physics, limiting their applicability to real observations. We examine the prediction of the neutral fraction and astrophysical parameters from 21 cm maps and find that networks typically fail to generalise to unseen simulations. We explore a number of case studies to highlight factors that improve or degrade network performance. These results emphasise the responsibility on users to ensure ML models are applied correctly in 21 cm cosmology. 2024 Reproducibility of machine learning analyses of 21 cm reionization maps – References 1 Introduction Our understanding of the Universe has grown substantially over the last few decades, but there is still much we do not know about the Epoch of Reionization (EoR) – the period of \sim\!1 billion years after the Big Bang when light from the first stars ionized the hydrogen in the intergalactic medium (IGM). There are many possible astrophysical models for reionization, typically characterized by parameters such as the escape fraction of ionizing photons from galaxies ( Khaire et al., 2016 ; Mesinger et al., 2016 ) , and there is considerable uncertainty in the overall timing of the reionization process ( Greig & Mesinger, 2017a ; Kulkarni et al., 2019 ) . Current empirical constraints about the EoR are limited, coming mainly from cosmic microwave background (CMB) measurements ( Planck Collaboration, 2020 ) and quasar absorption spectra ( Fan et al., 2006 ; Becker et al., 2018 ; Keating et al., 2020 ; Bosman et al., 2022 ) . Recent observations from the James Webb Space Telescope (JWST) 1 1 1 https://www.stsci.edu/jwst have provided critical insights into the Epoch of Reionization, particularly in refining the timeline. JWST’s deep-field observations have revealed galaxies forming as early as 200 million years after the Big Bang ( Naidu et al., 2022 ; Finkelstein et al., 2023 ; Harikane et al., 2023 ; Curtis-Lake et al., 2023 ; Carniani et al., 2024 ) , pushing back the likely start of reionization. Furthermore, estimates from JWST data suggest that the star formation rates and escape fraction of photons might be higher than previously anticipated, especially in faint, early galaxies ( Atek et al., 2024 ) . JWST’s ability to study faint star-forming galaxies ( Endsley et al., 2023 ; Endsley et al., 2024 ) , constrain and detect the ionized bubbles surrounding galaxies ( Saxena et al., 2024 ; Hsiao et al., 2023 ) , as well as the measuring of neutral hydrogen fractions via Lyman-alpha absorption damping wings ( Umeda et al., 2024 ) , has been instrumental in mapping the progress of reionization. The most promising new probe of the EoR is the redshifted 21 cm line of neutral hydrogen ( Pritchard & Loeb, 2012 ; Liu & Shaw, 2020 , for a review see e.g.) . This is the radiation released by a neutral hydrogen atom when its electron undergoes a spin flip transition ( Field, 1958 ; Field, 1959 ) . A new generation of low-frequency radio telescopes, such as the Low Frequency Array ( van Haarlem et al., 2013 , LOFAR 2 2 2 www.lofar.org ,) , the Hydrogen Epoch of Reionization Array (HERA 3 3 3 http://reionization.org , DeBoer et al. (2017) ), and the Square Kilometre Array ( Braun et al., 2015 , SKA 4 4 4 www.skao.int ,) , should be able to directly observe the evolution of the neutral hydrogen distribution during the EoR. An open problem in EoR research is how best to constrain the reionization history using 21 cm intensity measurements. One common approach is to use summary statistics such as the power spectrum ( Trott et al., 2016 ; Pober et al., 2014 ) ; and fully Bayesian parameter inference methods have also been developed for this task ( Greig & Mesinger, 2015 ; Greig & Mesinger, 2017b ) . However, the 21 cm signal from the EoR is expected to be highly non-Gaussian, implying that there is significant information encoded in higher-order statistics such as the bispectrum ( Majumdar et al., 2018 ; Shimabukuro et al., 2016 ) . But all such summaries inevitably entail some loss of information. The SKA will produce tomography throughout the EoR, allowing the possibility of inferring parameters from the maps themselves. However, the amount of data and large parameter space pose significant challenges for traditional statistical inference approaches, motivating the use of more flexible machine learning (ML) methods. ML methods have already been shown to be effective in the analysis of 21cm simulations. Artificial neural networks (ANNs) have been used for emulation to accelerate the speed of modelling ( Schmit & Pritchard, 2018 ; Bevins et al., 2021 ; Bye et al., 2022 ; Sikder et al., 2024 ) , to improve foreground mitigation ( Mertens et al., 2024 ) and for inference of parameters from the power spectrum ( Shimabukuro & Semelin, 2017 ; Choudhury et al., 2020 ; Choudhury et al., 2021 ; Choudhury et al., 2022 ; Tiwari et al., 2022 ; Shimabukuro et al., 2022 ) . Convolutional neural networks (CNNs) have been used to emulate summary statistics ( Breitman et al., 2024 ) , infer parameters from power spectra ( Doussot et al., 2019 ) , images ( Gillet et al., 2019 ; Kwon et al., 2020 ; Neutsch et al., 2022 ; Choudhury et al., 2022 ) and lightcones ( Zhao et al., 2022 ; Prelogović et al., 2022 ) , to infer topological statistics ( Bianco et al., 2021 ) , constrain cosmological models ( Sabiu et al., 2022 ) and for foreground removal ( Villanueva-Domingo & Villaescusa-Navarro, 2021 ; Makinen et al., 2021 ; Ni et al., 2022 ; Bianco et al., 2024 ; Sabti et al., 2024 ) . In this paper, we investigate the reproducibility and generalisability of currently proposed CNN-based methods of reionization parameter inference from 21 cm images. After briefly summarising 21 cm cosmology in Sec. 2 and CNNs in Sec. 3 , we then look at several case studies: • In Sec. 4 we create a simulated 21cm signal and reproduce a common CNN methodology used in the field. We first attempt to recover the neutral fraction of hydrogen, before and after generalising the network. • In Sec. 5 we apply this same network to the simulated data set, but with the aim of recovering a wider range of astrophysical and cosmological parameters: \Omega_{\mathrm{m}},h,f_{\mathrm{esc}},\sigma_{8},C_{\mathrm{ion}}\text{ and }D_{\mathrm{ion}}. • In Sec. 6 we test the correlation between performance of the network and the number of training examples. We then look at adding redshift information structure into the network, such that we can recover parameters that capture the evolutionary or integrated nature of reionization. • In Sec. 7 we use the network to constrain the timescale