Lossless compression of simulated radio interferometric visibilities
A. R. Offringa & R. J. van Weeren · 2025
Read Lossless compression of simulated radio interferometric visibilities by A. R. Offringa & R. J. van Weeren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23490.txt.
Introduction
Radio telescopes produce huge data sets, and calibration requires storing predicted model signals that can be far larger than the original observations. A. R. Offringa and R. J. van Weeren present Sisco, a compression method for these noiseless simulated signals. Lossy compression, which works fine on noisy real data, corrupts the calibration step when applied to model data.
Sisco predicts each visibility value by polynomial extrapolation over time and frequency, then compresses only the leftover differences. Linear and quadratic prediction together worked best, shrinking model data to about a quarter of its original size on average across LOFAR, MeerKAT, and MWA data. Built as an open-source storage manager, Sisco drops into existing workflows without changing formats.
Catalog record
| CREATOR | A. R. Offringa & R. J. van Weeren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23490
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lossless compression of simulated radio interferometric visibilities A. R. Offringa & R. J. van Weeren Abstract Context. Processing radio interferometric data often requires storing forward-predicted model data. In direction-dependent calibration, these data may have a volume an order of magnitude larger than the original data. Existing lossy compression techniques work well for observed, noisy data, but cause issues in calibration when applied to forward-predicted model data. Aims. To reduce the volume of forward-predicted model data, we present a lossless compression method called Simulated Signal Compression (Sisco) for noiseless data that integrates seamlessly with existing workflows. We show that Sisco can be combined with baseline-dependent averaging for further size reduction. Methods. Sisco decomposes complex floating-point visibility values and uses polynomial extrapolation in time and frequency to predict values, groups bytes for efficient encoding, and compresses residuals using the deflate algorithm. We evaluate Sisco on diverse LOFAR, MeerKAT, and MWA datasets with various extrapolation functions. Implemented as an open-source Casacore storage manager, it can directly be used by any observatory that makes use of this format. Results. We find that a combination of linear and quadratic prediction yields optimal compression, reducing noiseless forward-predicted model data to 24% of its original volume on average. Compression varies by dataset, ranging from 13% for smooth data to 38% for less predictable data. For pure noise data, compression achieves just a size of 84% due to the unpredictability of such data. With the current implementation, the achieved compression throughput is with 534 MB/s mostly dominated by I/O on our testing platform, but occupies the processor during compression or decompression. Finally, we discuss the extension to a lossy algorithm. 1 Introduction Modern radio astronomical observatories produce vast amounts of visibility data, for which the complex values are typically stored as two 32-bit single-precision floating-point values. During the processing of such data, it is often necessary to store model data alongside the observed visibilities: these are forward-predicted values that simulate the response of a telescope given a sky model and are commonly used in two processing steps: i) (self-)calibration, where gains are solved to minimize the difference between the model visibilities and observed data; and ii) deconvolution, where they are used to compute the residual image in Cotton-Schwab-based techniques ( 29 ) . Such data are of the same size as the observed visibility data. Storing and transporting large data volumes is a major expense for observatories, which will be even greater for the Square Kilometre Array (SKA) currently under construction. The large size of the model data can be particularly problematic for multi-stage calibration approaches ( 6 ) or for direction-dependent self-calibration ( 36 ) . In the latter case, it may be necessary to store a model column for each calibrated direction ( 7 ) , which can reach about ten times the size of the observed data. The observed visibilities are noisy and can be compressed using lossy compression techniques. Dysco is a compression technique that automatically finds appropriate scaling for interferometric data and performs non-linear quantization and bit-packing ( 21 ) . With the Dysco compression technique, interferometric data can be compressed by approximately a factor of four with an insignificant increase in noise ( 21 ; 4 ) . By using non-linear quantization optimized for Gaussian-distributed data, it produces visibilities for which the sum of squared residuals is minimal for a given compression factor. The lossy compression method offered by the FITS format is aimed at imaging data ( 25 ) and is based on linear quantization and Rice encoding. More generic approaches also exist, such as the MultiGrid Adaptive Reduction of Data ( mgard ) technique ( 13 ) , which employs detrending, linear quantization, and Huffman encoding. This method has also been applied in the context of radio interferometric data ( 9 ) , and was shown to provide good compression with insignificant impact on the image quality. The mgard technique does not offer an automated way to perform scaling appropriately for radio interferometric data, in which different correlated visibilities may be formed from antennas with varying noise levels—a particularly significant effect in very long baseline interferometry (VLBI) — and currently requires manually setting an absolute or fractional error bound. Although the aforementioned compression techniques for noisy visibility data are available, applying quantization-based compression techniques with similar compression rates to model data adds noise to the otherwise noise-free model data. This causes problems in processing, particularly in calibration, where the observed data is calibrated using these model data. This makes lossy compression unsuitable for model data, and in this paper we therefore explore lossless compression. Model data are noiseless and generally smooth in the time and frequency directions. Although off-axis sources can reduce the smoothness of the data, the integration time and frequency resolutions are typically chosen to minimize smearing on the longest baselines, ensuring that the visibility data remain smooth on all shorter baselines. This effect is further mitigated by the attenuation of these distant sources by the primary beam. Many compression algorithms can exploit correlations between adjacent values, and a common approach for compressing such data is to predict the next value based on previous value(s) and compress the difference using an entropy encoder such as Rice-Golomb encoding ( 12 ; 27 ) or Huffman encoding ( 15 ) . One well-known example of such an algorithm is the Free Lossless Audio Codec (FLAC; 32 ), which compresses the difference between adjacent values using Rice encoding. However, FLAC is designed for compressing integer values, whereas visibilities are floating-point values, making the encoding not directly applicable. The FITS format supports lossless compression using Rice compression for image data but also supports only integer data in lossless mode ( 24 ) . Generic lossless algorithms, such as GZIP (which employs the deflate mechanism combining LZ77 and Huffman coding), are widely used due to their balance of speed and compression ratio. However, they do not exploit the spectrally and temporally smooth behaviour of radio astronomical model data that can be leveraged for better performance. Another method to reduce the volume of interferometric data is baseline-dependent averaging (BDA; 37 ). This can reduce data by a factor of a few but comes in most cases at the cost of some information loss: reducing the time and frequency resolution of the data may complicate or prevent data calibration or experiments requiring high time or frequency resolution, such as the search for radio signatures of star-planet interactions or exoplanets (e.g., 3 ), fast-radio bursts (e.g., 26 ), gamma-ray bursts (e.g., 28 ), spectral lines (e.g., 14 ) and polarized emission at non-zero Faraday depth (e.g., 23 ). In this paper, we develop a novel lossless compression framework for radio astronomy data that, for the first time, targets noiseless model data. It integrates prediction for floating-point data and compresses this with the deflate algorithm. By leveraging predictive preprocessing to reduce entropy, followed by the efficient dictionary-based and Huffman encoding of the off-the-shelf libdeflate library 1 1 1 https://github.com/ebiggers/libdeflate , our approach achieves significantly improved compression ratios over generic data compression algorithms. This algorithm is especially suitable for interferometric model data, as the memory footprint per baseline is minimal, thereby allowing on-the-fly compression. The algorithm is released