A crude but efficient pipeline for JWST MIRI imager : the case of sn1987A

R. Gastaud, A. Coulais Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, Gif-sur-Yvette, France [email protected] LERMA, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, Paris, France [email protected] · 2025

Read A crude but efficient pipeline for JWST MIRI imager : the case of sn1987A by R. Gastaud, A. Coulais Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, Gif-sur-Yvette, France [email protected] LERMA, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, Paris, France [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13926.txt.

Introduction

Rene Gastaud and Alain Coulais present a 2025 paper on processing images from the James Webb Space Telescope. They worked with the MIRI imager, the telescope's mid-infrared camera. Their test case was the supernova remnant SN 1987A, a famous exploded star studied for decades.

Big observatories build official software over years with large teams, which the authors liken to cathedrals. Their own pipeline is deliberately crude: a short script in a high-level language, built from basic operators. Yet it turned raw data into outputs fit for science. They argue that large projects should be designed so small, cheap tools can plug in and offer rival processing at very low cost.

Catalog record

CREATORR. Gastaud, A. Coulais Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, Gif-sur-Yvette, France [email protected] LERMA, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, Paris, France [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A crude but efficient pipeline for JWST MIRI imager : the case of sn1987A R. Gastaud, A. Coulais Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, Gif-sur-Yvette, France [email protected] LERMA, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, Paris, France [email protected] Abstract Most of the space projects or large observatories do have official tools like simulators, end-to-end pipelines developed during years by a large team of contributors. They are like cathedrals . In this paper, we show that very simplistic code using basic operators provided by high level language like GDL allows to write quickly high quality code to process raw data into scientifically validated outputs. This is bazaar . In this paper we argument why we consider large infrastructure should be designed to allow small ones to benefit from it and allow to graft better alternative processing at very low cost. 1 introduction Observations in imager mode of supernova sn1987A with MIRI have been made in July 2022, just after Performance Verification Phase of the instruments onboard space mission Webb (aka "James Webb Space Telescope" or JWST) ( Wright & al. 2023 ) . We quickly realized that the official pipeline was too limited at that time and does not give the high quality level we need. So we developed in IDL/GDL a simple but efficient pipeline to process our data from scratch. Being able to code with our favorite language (GDL : Park & et al 2022 ; Coulais & Duvert 2024 ) was a way to quickly test new ideas and have high quality results, without waiting for official product. 2 What can we expect from a pipeline ? What is a pipeline ? A pipeline ingests the data from any instrument, any observation mode, and processes it by steps to a meaningfully science data. Pipeline automatically runs on all data, do not confuse with Data Analysis Tools which requires science decisions (human interaction), but a good pipeline is flexible and can be used as a Data Analysis Tool. What is a good pipeline : this depends upon the user. An average user wishes to have: • easy and quick installation on a personal computer • good documentation, with a lot of examples (notebook or python scripts) • speed and memory : could be run on a personal computer on reasonable amount of time • input/output for each stage/step should be stored in self-documented open-format files containing information on the previous steps, with the name of the calibration files used, and the version of the pipeline. An advanced user wishes more: • modularity: each stage/step can be individually called • flexibility: each step has default method and parameters which can be over-ridden • parametrable log facility • documentation on each step (methods and parameters) A programmer wishes even more: • unit test for each stage/step • each step can be done by different methods, easily substituable • use of a configuration management for the code (eg Git or SVN) • documentation on the unit tests, heritage of each module, architecture, etc… • use of external libraries (do not reinvent the wheel) Figure 1. : Left : JWST pipeline, with in blue the home-made. Right : mosaicking images : top home-made, bottom official pipeline 3 JWST Official Pipeline Gordon & al. 2015 described the official JWST MIRI pipeline It respects most of the points we list in the previous section. We can discuss that the documentation is very limited, and the examples too. We were able to locate only a very limited set of unitary tests, poorly documented, and we don’t find end-to-end tests. JWST is an observatory coming with four instruments, each one are a combination of a large number of observing modes and optical configurations. For MIRI, we have LRS, MRS, imager, full or sub-array modes, etc… The installation of the official pipeline comes with all the modes, takes time, requires Conda and a lot of disk space. Even if you have very limited data (eg a small LRS observation) all the code for all the 4 instruments and all the modes are installed and calibrations products are downloaded. At the end, before running the code, the total amount of space is surprisingly large (tens of Go). FITS files haves a long story in the ADASS conferences. In the JWST case, the input/output of each step, intermediate or final, and the calibration data are stored in FITS files. These files contain the detailed information of the previous steps, the version of the pipeline and also the version of the associated calibration files (CDP). The very interesting point to use FITS files is it can be read back by most of the tools used in astronomy (ds9, fv, IDL/GDL, …). The pipeline is divided in 3 main stages. Each stage produces an output which data has different dimensions and or different unis, so this division is not arbitrary. Each stage is divided in several steps, which can be turned on/off, whose methods and parameters can be selected. A stage or a step can be executed alone. The input/output of each step can be stored (not the default option). This is great to tune the pipeline to your data, the problem is again the documentation (some name can be misleading). The pipeline is versatile : the same code for the different instruments as much as possible, both for storing data and processing data. This is possible with inheritance. The drawback is a code more difficult to understand. The pipeline uses as much as possible existing routines/libraries, for example the astropy package. 4 Dedicated Home-made Pipeline At the beginning of JWST scientific life, we had to process an observation of SN1987 made by the MIRI Imager ( Bouchet & al. 2024 ) . This observation was done with four different filters, and with a subarray 512x512 to avoid saturation (the sampling time is quicker than the full array). Four different exposures with different pointing were taken for each filter (Dithering) to reduce the impact of bad pixels. We simplify the steps and avoid to use the official calibration products : some were not available, some were not optimal (in our case), and some were not up to date. Our home-made pipeline starts from raw data and can be used only one mode of the observatory (see blue line in figure 1 left). We made significant improvements in data processing for • ramps to slopes, rewriting a method with a better Allan Variance (we use median filtering) • odd/even removal (less SNR on individual maps) : we remove dark subtraction step which only adds noise and compute a better background • mosaicking (better resolution in maps) we rebin of a factor 2 to take into account the half-pixel shifts When compared to the initial official product, clearly the revisited one was better, which was a must for the post-processing done in Bouchet & al. 2024 . Even if those improvements are now known and published, some are still not in the official pipeline. Thanks to our experience as data scientists, following KISS philosophy, we wrote the custom tools working on raw data, circumventing the constraints from official infrastructure, in a month. As usual, the end of the pipeline is not

Read the complete text