Self-supervised and Multi-fidelity Learning for Extended Predictive Soil Spectroscopy Journal: TBD
Luning Sun, José L. Safanelli, Jonathan Sanderman, Katerina Georgiou, Colby Brungard, Kanchan Grover, Bryan G. Hopkins, Shusen Liu, Timo Bremer · 2025
Read Self-supervised and Multi-fidelity Learning for Extended Predictive Soil Spectroscopy Journal: TBD by Luning Sun, José L. Safanelli, Jonathan Sanderman, Katerina Georgiou, Colby Brungard, Kanchan Grover, Bryan G. Hopkins, Shusen Liu, Timo Bremer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15965.txt.
Introduction
Luning Sun, José L. Safanelli, and seven co-authors build a self-supervised framework to forecast soil traits from spectra. Soil spectroscopy reads how soil reflects light in mid- and near-infrared ranges to infer traits cheaply. But labeled samples are scarce, and the richer mid-infrared libraries are costly to use in the field. Their method first learns from unlabeled spectra alone.
A variational autoencoder compresses the large mid-infrared library into embeddings, compact learned summaries. Scan repeats augment training. A frozen decoder then turns cheap portable near-infrared scans into forecast mid-infrared spectra. Downstream models use raw spectra, forecast spectra, and embeddings to predict nine soil traits, matching or beating baselines on a gold-standard test set.
Catalog record
| CREATOR | Luning Sun, José L. Safanelli, Jonathan Sanderman, Katerina Georgiou, Colby Brungard, Kanchan Grover, Bryan G. Hopkins, Shusen Liu, Timo Bremer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15965
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-supervised and Multi-fidelity Learning for Extended Predictive Soil Spectroscopy Journal: TBD Luning Sun, José L. Safanelli, Jonathan Sanderman, Katerina Georgiou, Colby Brungard, Kanchan Grover, Bryan G. Hopkins, Shusen Liu, Timo Bremer Abstract We propose a self-supervised machine learning (SSML) framework for multi-fidelity learning and extended predictive soil spectroscopy based on latent space embeddings. A self-supervised representation was pretrained with the large USDA NSSC-KSSL mid-infrared (MIR) spectral library and the Variational Autoencoder algorithm to obtain a compressed latent space (32 features) for generating spectral embeddings. At this stage, only unlabeled spectral data were used, allowing us to leverage the full spectral database and the availability of scan repeats for augmented training (n scans = 334,665). We also leveraged and froze the trained MIR decoder for a spectrum conversion task by plugging it into a near-infrared (NIR) encoder to learn the mapping between NIR and MIR spectra in an attempt to leverage the predictive capabilities contained in the large MIR library with a low cost portable NIR scanner. This was achieved by using a smaller subset of the KSSL library, which contains paired NIR and MIR spectra (n = 2,106). Downstream machine learning models were then trained to learn the mapping between original spectra, predicted spectra, and latent space embeddings for nine soil properties: total carbon (TC), total nitrogen (TN), inorganic carbon (IC), estimated organic carbon (EOC), soil pH (pH), cation exchange capacity (CEC), clay, silt, and sand. The performance of soil property prediction was evaluated independently of the KSSL training data using a gold-standard test set derived from the North American Proficiency Test (NAPT) Program soil archive, along with regression goodness-of-fit metrics. Compared to baseline models, the proposed SSML and its embeddings yielded similar or better accuracy in all soil properties prediction tasks. Predictions derived from the spectrum conversion (NIR to MIR) task did not match the performance of the original MIR spectra but were similar or superior to predictive performance of NIR-only models, suggesting the unified spectral latent space can effectively leverage the larger and more diverse MIR dataset for prediction of soil properties not well represented in current NIR libraries. 1 Introduction Measuring and monitoring soils at scale is costly ( 12 ) . The demand for high-quality soil information is surging with the increased recognition of the soil’s role in climate regulation and adaptation, sustainable agricultural production, and its importance in several other ecosystem services ( 1 ) . Emerging technologies for generating basic soil data in response to this growing demand have been maturing in academia and are now being tested in commercial setups ( 24 ) . Among the range of potential measurement technologies, diffuse reflectance spectroscopy (DRS) is one of the most promising ( 34 ; 14 ) , as the spectral acquisition of a soil sample is fast (often takes less than a minute), requires minimum sample preparation, does not generate chemical residues, and is sensitive to some soil properties variations ( 28 ; 25 ; 27 ) . This happens because the resulting DRS spectra are characterized by a combination of interactions between light and organic, mineral, and sample bulk constituents, which can be explored using machine learning when analytical reference data are available. Still, challenges related to handling spectral data more effectively, addressing variable fidelity between spectral representations and relevant soil properties, and increasing the reliability of generated predictions, all remain as research opportunities to unlock this technology as a viable soil measurement tool. DRS spectra measure the amount of light reflected from the surface of a soil sample in relation to a reference material. This interaction can be measured across the visible (Vis, 400-700 nm), near (NIR, 700-2500 nm), and mid-infrared electromagnetic region (MIR, 2500-25000 nm, i.e., 4000-400 cm -1 ), depending on the instrument characteristics and study goals. The measured signal is represented by its intensity as a function of evenly spaced contiguous channels, measured across the electromagnetic range of interest (e.g., VisNIR, NIR, or MIR). Therefore, soil spectral data are inherently high-dimensional (hundreds of wavelengths scanned contiguously), multicollinear (nearby and distant wavelengths are strongly correlated), hence redundant ( 20 ) . Multicollinearity of spectral data can be partially addressed through preprocessing techniques, such as employing the Standard Normal Variate (SNV) method ( 3 ) , which centers and scales each spectrum to its origin and unit across the wavelength range, respectively. Still, remaining redundancy and the high-dimensionality issues often require specific strategies to calibrate more consistent prediction models. Partial Least Squares Regression (PLSR) and its variants are examples of multivariable and/or multivariate algorithms that are capable of decomposing the high dimensionality and redundancy of spectral data via orthogonalization while simultaneously maximizing the variance of a single or a set of target variables, which are later combined with multiple linear regression ( 7 ) . PLSR is considered the standard fitting algorithm in chemometrics or machine learning with spectral data, and is often used as a baseline. Similarly, other linear orthogonalization algorithms like principal component analysis (PCA) have also been combined with statistical or machine learning algorithms in predictive soil spectroscopy to control these issues, which often benefits model performance ( 33 ; 4 ; 25 ) . With the advent of deep learning, which offers greater flexibility in designing complex architectures and loss functions for exploring linear and non-linear relationships of data ( 13 ) , several architectures have been tested in soil spectroscopy. This includes simpler models with multilayer perceptrons ( 37 ; 36 ) to more sophisticated architectures like convolutional and recurrent neural networks ( 15 ; 18 ; 31 ; 38 ; 17 ) , as well as novel algorithms based on autoencoders and transformers that are forming the basis of foundation models in several fields, including soil spectroscopy ( 10 ; 9 ) . A paradigm that has recently emerged is self-supervised learning, where a model is trained on unlabeled data ( 2 ) . This is especially interesting for soil spectroscopy, as an autoencoder can effectively compress the spectra into a latent space (i.e., a lower-dimensional representation) to produce discriminative features (embeddings) while eliminating redundancy and reducing the high dimensionality of the spectral data by orders of magnitude. 32 explored this type of architecture and effectively demonstrated the enhancement of prediction accuracy based on soil spectral data. In this paper, we leverage self-supervised learning on a large MIR soil spectral library to learn a latent feature space of 32 dimensions. We extend our study from previous work with an additional hypothesis: the latent space represents a manifold of the fundamental infrared spectral region, enabling interoperability between different fidelities, i.e., NIR and MIR (Fig. 3 ). This hypothesis was explored because the NIR range contains broad features, overtones, and combinations of overtones from the fundamental MIR range at higher frequencies ( 20 ; 35 ) . Hence, NIR spectral features are highly overlapping and less sensitive to soil variations (low-fidelity), such that making a bridge between the NIR and MIR could extend or enhance the predictive capacity of a low-fidelity scanner with access to larger MIR datasets. This is especially compelling for scaling soil spectroscopy as a measuring and monitoring tool because low-fidelity NIR scanners are an order of magnitude cheaper than bench-top MIR instruments