End-to-end design framework for compressed on-chip pixel-wise spectro-polarimeters
T.A. Stockmans, F. Snik, J.M. Smit, J.H.H. Rietjens, M. Esposito, C. van Dijk, C.U. Keller · 2025
Read End-to-end design framework for compressed on-chip pixel-wise spectro-polarimeters by T.A. Stockmans, F. Snik, J.M. Smit, J.H.H. Rietjens, M. Esposito, C. van Dijk, C.U. Keller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06724.txt.
Introduction
Stockmans and co-authors describe a design toolbox for tiny Earth-watching instruments, in a 2025 paper. New detector chips carry color and polarization filters on each pixel, shrinking a satellite spectrometer to camera size. Flying fewer filter passes saves time but leaves holes in the data cube.
Software must fill those holes, and the filter layout decides how well it can. The toolbox trains a neural filler and the filter values together, so layout and reconstruction improve as a pair. Training used airborne measurements over Earth from a sister instrument. The paper reports an optimized layout from that training. End-to-end design, where optics and algorithms are tuned jointly, is the authors' broader pitch for cheap space instruments.
Catalog record
| CREATOR | T.A. Stockmans, F. Snik, J.M. Smit, J.H.H. Rietjens, M. Esposito, C. van Dijk, C.U. Keller |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
End-to-end design framework for compressed on-chip pixel-wise spectro-polarimeters T.A. Stockmans, F. Snik, J.M. Smit, J.H.H. Rietjens, M. Esposito, C. van Dijk, C.U. Keller Abstract Modern detector manufacturing allows spectral and polarimetric filters to be directly integrated on top of separate detector pixels. This enables the creation of CubeSat-sized spectro-polarimetric instruments that are not much larger than the detector and a lens. Redundancy inherent to the observed scene, offers the opportunity for sparse sampling in the form of not scanning all filters at every location. However, when there are fewer pushbroom steps than filters, data are missing in the resulting data cube. The missing, largely redundant data can be filled in with interpolation methods, often called demosaicers. The choice of filters and their precise layout influences the performance of the instrument after the demosaicing process. In these proceedings we describe a part of a design toolbox for both the filter layout and the optimum parameters for the reconstruction to a full spectro-polarimetric data cube. The design tool is based on training a (neural) network and jointly updating the values of the filters and demosaicer. We optimized a filter layout by training on spectro-polarimetric remote observations of the Earth acquired by SPEX airborne. This optimised filter layout could reconstruct a validation scene from five overlapping snapshots (pushbroom steps), which would take 109 pushbroom steps when measuring with a classical layout and no reconstruction. a]Leiden University, Niels Bohrweg 2, 2333 CA Leiden, The Netherlands b]SRON Netherlands Institute for Space Research, Niels Bohrweg 4, 2333 CA Leiden, the Netherlands c]cosine Remote Sensing, Warmonderweg 14, 2171 AH Sassenheim, The Netherlands d]Lowell Observatory, 1400 W Mars Hill Rd, Flagstaff, AZ 86001, USA keywords spectro-polarimetry, filter array design, TensorFlow, Cubesat † † authorinfo: Further author information: (Send correspondence to T.A. Stockmans) T.A.Stockmans: E-mail: [email protected] 1 INTRODUCTION The first instrument that performed multiband polarimetry for remote sensing from space was the POLDER instrument. [ 1 ] Since then there have been many instruments, expanding for instance on spectral capabilities; see e.g. the overview by ref. [ 2 ] . For space applications, volume and mass are often the most important cost drivers and hence, when similar capabilities can be achieved at reduced mass and volume, more options for application in space become available. A first step in this direction has been made with the development of the HARP cubesat [ 3 ] and the SPEXone [ 4 ] and HARP2 polarimeters as part of the PACE mission. These instruments are significantly smaller than many of their predecessors. Still, further miniaturization while maintaining instrument capabilities would greatly simplify their use in e.g. satellite constellations, distributed systems, and on cubestats. An effective way to design smaller spectro-polarimetric instruments is by micro-patterning both the polarimetric and spectral filters directly on each individual detector pixel. The size reduction enabled by this production technique opens up the possibility to go to CubeSat-sized instruments with all their cost benefits. Details on the micro-patterning can be found in e.g. [ 5 ] . Some examples in which this technology is put to use can be found in [ 6 , 7 , 8 , 9 ] . Micro-patterned filters are not new, especially concerning color filters. RGB cameras have been working for over 45 years with these kind of filters. [ 10 ] To enable instruments with pixelated filters to create full images within one snapshot, algorithms have been created to reconstruct the missing information, which has been dubbed ”demosaicing”. These algorithms infer the unmeasured values at all pixels by making use of surrounding pixels and measured bands. These techniques have expanded beyond RGB imagers to hyperspectral imaging and spectro-polarimetry. Some examples for demosaicing for polarimetric division of focal plane images are [ 11 , 12 , 13 , 14 ] . The design of a filter array consists of 2 main parts. First, there are the specific parameters of the filters themselves. For spectral filters it would be the bandwidth over which it transmits; for linear polarization filters, it is the orientation angle of the linear polarizer. Second, there is the layout of these filters over all pixels; for example, the most widespread filter array layout for RGB imaging is the Bayer pattern [ 10 ] . We have developed a network that jointly optimizes the design of the color filters for hyperspectral imaging with a optimal form of demosaicer; see Stockmans et al. in prep . In these proceedings, we show the preliminary results of the addition of a micropolarizer array to this framework for the design of a spectro-polarimetric imager. 2 Methods The work presented here explores the addition of linear polarization measurements to our compressed sensing approach for hyperspectral imaging described in Stockmans et al. in prep . The main focus is the design of a filter layout that can be directly attached on all individual pixels of a detector. This instrument can take a snapshot image and move the instrument slightly so that every pixel is looking at the next patch of land or ocean. We will refer to these movements as pushbroom steps and put them in as a design parameter. In the paper by Stockmans et al. in prep we describe the layout design process of the spectral filter array on top of a CMOS detector of a hyperspectral instrument, with a demosaicing algorithm (linear reconstructor) being optimized simultaneously. The spectral filter array is designed to be incomplete, which means that every geometrical pixel is not imaged in every desired spectral band. The linear reconstructor acts as an interpolator between spectral bands and spatial connections to give an intensity estimate of every geometric pixel in all desired spectral bands. The design framework is built up in TensorFlow, a Python module capable of doing automatic backward propagation. The spectral filter array, detector, and reconstructor are all described as separate layers through which the data propagates from input datacube to output datacube. A layer is a TensorFlow class comprising of an input, output and updating function describing the relation between in- and output. The spectral filter array has as an input the full hyperspectral scene and the variables of the layer describe the filters themselves. The output of this layer is the filtered hyperspectral datacube. The reconstructor is a matrix of variable multipliers mapping the intensities coming out of a detector towards a full hyperspectral datacube. The variables of both layers are updated during training stage towards the values that result in the smallest difference between the reconstructed and original datacube. This difference is the so-called ”loss function”, a function of merit that needs to be minimized by the optimizing algorithm inherent to TensorFlow. The updating of the variables is done using back propagation, so a partial differential must be defined between the final loss function and the variables. Each variable could also be fixed to a certain value if desired, without losing the updating capabilities of the rest of the variables. The main addition described in this proceeding to the framework above, is the addition of one layer at the beginning that describes a microgrid structure of linear polarizers in different orientations. The input of the network can be either the combination of the degree of linear polarisation (DoLP),