Demonstrating Onboard Inference for Earth Science Applications with Spectral Analysis Algorithms and Deep Learning

Itai Zilberstein, Alberto Candela, Steve Chien, David Rijlaarsdam, Tom Hendrix, Léonie Buckley, Aubrey Dunne · 2025

Read Demonstrating Onboard Inference for Earth Science Applications with Spectral Analysis Algorithms and Deep Learning by Itai Zilberstein, Alberto Candela, Steve Chien, David Rijlaarsdam, Tom Hendrix, Léonie Buckley, Aubrey Dunne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15053.txt.

Introduction

Itai Zilberstein and co-authors report in 2025 a live demonstration of onboard data analysis on the CogniSAT-6 satellite. The craft carries a hyperspectral camera and neural network hardware. Working with an industry partner, the team runs deep learning and spectral analysis tools on the craft itself. The trials cover a range of Earth science tasks. Findings can then be drawn in orbit rather than after download.

Catalog record

CREATORItai Zilberstein, Alberto Candela, Steve Chien, David Rijlaarsdam, Tom Hendrix, Léonie Buckley, Aubrey Dunne
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Demonstrating Onboard Inference for Earth Science Applications with Spectral Analysis Algorithms and Deep Learning Itai Zilberstein, Alberto Candela, Steve Chien, David Rijlaarsdam, Tom Hendrix, Léonie Buckley, Aubrey Dunne Abstract In partnership with Ubotica Technologies, the Jet Propulsion Laboratory is demonstrating state-of-the-art data analysis onboard CogniSAT-6/HAMMER (CS-6). CS-6 is a satellite with a visible and near infrared range hyperspectral instrument and neural network acceleration hardware. Performing data analysis at the edge (e.g. onboard) can enable new Earth science measurements and responses. We will demonstrate data analysis and inference onboard CS-6 for numerous applications using deep learning and spectral analysis algorithms. fnum@section1 Introduction The capabilities of in-orbit assets to perform Earth science has skyrocketed in recent years. New space providers are deploying satellites that possess state-of-the-art multi- and hyperspectral instruments along with processors that raise the ceiling for onboard computation. CogniSAT-6/HAMMER (CS-6) is one such spacecraft that has a visible and near infrared range hyperspectral instrument and AI acceleration hardware, enabling advanced edge data analysis [ 1 ] . CS-6, which is a 6U CubeSat, launched in March of 2024. It is in a sun-synchronous orbit at an orbital height of around 500 km. Onboard CS-6 there is a Myriad X Vision Processing Unit (VPU) which can perform rapid computer vision, image signal processing, and neural network execution. CS-6 employs the HyperScape 100 instrument that can take hyperspectral measurements in the range 440\text{nm}{-}884 nm and achieves a ground sample distance of 5 meters per pixel. Analyzing data onboard serves several key functionalities including rapid response to detected phenomena and reducing data volume through identification of unusable data. The former relies on inferring science properties of a data acquisition to direct intelligent planning of future measurements. Performing this computation at the edge (e.g. onboard) is key to enabling new, time-sensitive measurements of rare Earth phenomena that would otherwise be unobtainable if ground analysis was required. A spacecraft can self-cue and take another measurement in the same overflight using the knowledge from the first acquisition to direct the second one. This concept, called dynamic targeting , would enable higher-resolution pinpoint measurements [ 2 , 3 ] . Alternatively, a spacecraft can cross-cue another spacecraft to obtain rapid follow-up imagery of key targets. Onboard data analysis realizes a key component of NASA’s New Observing Strategies (NOS) program which aims to advance observation systems. Figure 1: Overview of a rapid response workflow for volcano monitoring. Image sources: Ubotica Technologies (steps 1 & 2), Planet Labs (step 3) . Figure 1 illustrates a rapid-response use-case for onboard data inference. Detection of thermal activity triggers communication to ground stations or other assets for quick follow-up actions. Inter-satellite links (ISL) enables satellites such as CS-6 to cross-cue other sensing assets or participate in decentralized scheduling [ 4 ] . In this work, we focus on the development and deployment of onboard algorithms, including spectral analysis algorithms and deep neural networks such as convolutional neural networks (CNNs), to perform inference for both image analysis and spectral signature detection in the visible and near infrared spectral ranges. Leveraging AI acceleration hardware to perform spectral analysis is novel and pioneers new avenues for edge computing. We target numerous Earth science applications ranging from the detection of clouds, wildfires, volcanic activity, and harmful algal blooms to surface water, vegetation, and mineral mapping and land-use classification. This paper details the development of inference algorithms that will be demonstrated onboard CS-6 and other spacecraft. This ongoing effort consists of composing datasets for each application using automated labeling techniques, designing and implementing algorithms within the constraints of flight hardware including training machine learning models, validating inference and execution, and a flight demonstration. fnum@section2 Methods 2.1 Datasets CS-6 data is limited due to its recent launch (March 2024). For spectral analysis, we leverage data from the USGS spectral library [ 5 ] . The image datasets are composed of 190 scenes from the Menut satellite operated by Open Cosmos and several hundred scenes from Planetscope data. We use the red, blue, green, and near infrared bands from these data products. CS-6 bands are stretched to enhance the data and enable compatibility with models trained on other satellite scenes. Let q_{1} and q_{99} be the 1 st and 99 th quantiles of a band. We stretch (onboard) each pixel, p , in a band so that it is in the range [v_{min},v_{max}] using the formula \textsc{min}\left[\textsc{max}\left(v_{min}+\frac{v_{max}-v_{min}}{(q_{99}-q_{1})}\cdot(p-q_{1}),v_{min}\right),v_{max}\right]{.} (1) We derive automated labeling techniques from the Haze Optimized Transform (HOT) method for clouds [ 6 ] , Normalized Difference Water Index (NDWI) method for surface water extent (SWE) [ 7 ] , band thresholds for thermal activity [ 8 ] , ESA’s WorldCover maps for land use [ 9 ] , and Sentinel-2 imagery for algal blooms. Methods such as HOT and NDWI can be directly computed from VNIR data. We compute the NDWI mask using the equation NDWI=\frac{Green-NIR}{Green+NIR}. (2) The NDWI product is then thresholded using Otsu’s method to obtain a binary surface water mask for a scene. The HOT method leverages the relationship between blue and red band values for non-cloudy pixels. The red band is less affected by atmospheric haze, while the blue band has more scattering. The clear-sky line defines this correlation and points that fall close to this line signify clear pixels, while points far from this line signify cloud or haze interference. Note that Planetscope scenes are delivered with cloud masks, therefore we apply the HOT method to label Menut and CS-6 data products. We compute the clear-sky line by selecting the 0.15\% of data points with the smallest blue band value. These points are divided into 20 bins, and for each bin the 20 points with the highest red band value are selected. Given these 400 points, we fit a line using linear regression. Let m and b define the clear-sky line. We compute the HOT value as HOT=|m\cdot Blue-Red|+\frac{b}{\sqrt{1+m^{2}}}. (3) Like NDWI, the HOT product is thresholded using Otsu’s method to obtain a binary cloud mask. Despite using automated techniques, the labelled data is human-verified to ensure quality. Due to CS-6 having only visible and near infrared data products, the science return is limited for certain applications. Without a higher wavelength, cloud classification includes noise from snow, shorelines, and other high-reflectance objects. SWE has noise from shadows and urban areas. There are obvious limitations in thermal detection without VSWIR or TIR sensing. Despite these drawbacks, the models perform with high accuracy as shown in the evaluation. 2.2 Spectral Analysis Algorithms Spectral analysis is used for two different applications: mineral and vegetation mapping. We engineer these algorithms to leverage the AI acceleration hardware onboard CS-6, a novel approach to deploying spectral algorithms. We use three common methods: spectral angle mapper (SAM), matched filters (MF), and the Reed-Xiaoli anomaly detector (RX). SAM is a function that measures similarity between any two spectra: x and y . It generalizes the notion of an angle between two vectors in N-space. Small SAM values indicate a high similarity, and vice versa. It is computed as follows: SAM(x,y)=\cos^{-1}\left(\frac{x\cdot y}{||x||^{2}~||y||^{2}}\right). (4) MF also quantifies similarity between two spectra, but it scales and normalizes the response by using scene statistics \mu (mean) and \Sigma (covariance). Larger MF values indicate a

Read the complete text