Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation
Jan Linnenbrink*, Jakub Nowosad*, Hanna Meyer · 2026
Read Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation by Jan Linnenbrink*, Jakub Nowosad*, Hanna Meyer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13689.txt.
Introduction
Jan Linnenbrink, Jakub Nowosad and Hanna Meyer call for situation-aware evaluation in environmental modelling. Map scores are ideally gauged from a fresh chance sample of the map area. Random checks suit spread training points, while spatial checks suit new areas, yet most cases sit between those extremes. Situation-aware methods freely adapt to the map setting, giving the most sound map scores across scenarios. The authors ground the case by redoing an earlier test study and weighing check methods.
Catalog record
| CREATOR | Jan Linnenbrink*, Jakub Nowosad*, Hanna Meyer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33391 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation Jan Linnenbrink*, Jakub Nowosad*, Hanna Meyer Abstract With the growing application of spatial predictive modeling in ecology, the question of how to appropriately evaluate the resulting maps has gained increasing attention. While there is consensus that map accuracy is ideally estimated using an independent probability sample of the prediction area, there is still no agreement on the most appropriate way to conduct an evaluation for the common case when such a sample is not available. Cross-validation, which involves multiple train–test splits, is commonly applied not only to estimate final model accuracy but also to guide model tuning and selection. Many different spatial and non-spatial approaches to cross-validation have been proposed, and approaches in both groups have faced substantial criticism. It has been shown that random cross-validation methods are suitable when the training points are randomly distributed in the prediction area, while spatial cross-validation is better suited towards extrapolation situations. In practice, however, there is a continuum and most cases are between those two extremes. To address this gap, we advocate for a new category of cross-validation methods to account for this: prediction-domain adaptive evaluation . Methods in this category flexibly adapt to the prediction situation, yielding most reliable estimates of map accuracy across different scenarios. To ground this perspective empirically, we reproduce a simulation study that was used in earlier research and systematically compare different evaluation methods and discuss their purpose. Keywords area of applicability \cdot data leakage \cdot machine learning \cdot map accuracy \cdot model validation \cdot spatial prediction 1 Introduction Spatial predictive modeling and mapping are essential in ecology and geosciences to assess species distributions, habitat suitability, and ecosystem dynamics across landscapes. The process typically requires two sets of data: response data representing the variable of interest, which are often drawn from large, opportunistic ecological databases (e.g., GBIF for biodiversity data: 12 , WoSIS Soil Profile Database: 3 , Fluxnet for gas exchange measurements: 28 , or sPlotOpen: 31 , a global dataset of vegetation surveys to mention just a few), and predictor variables, commonly derived from remote sensing products or climate models. Machine learning methods are then frequently used to train models on these data, as they are well suited to capture complex, non-linear relationships between environmental predictors and responses. Once trained, the model is applied across the entire spatial domain to generate predictions that are visualized as maps. Models may encounter different prediction scenarios, ranging from spatial interpolation (i.e., the whole prediction area is well covered by training points) to spatial extrapolation (i.e., the prediction area is not covered at all). To assess the quality of the derived map, especially as a baseline for decision-making, an evaluation is performed where the predictions are compared to data points that have not been used during model training. The question of how the evaluation should be performed, however, is a topic that is currently the subject of ongoing discussion. Established approaches, such as design-based inference relying on probability samples of the prediction area with known inclusion probabilities, are frequently infeasible, particularly in the absence of a formal sampling design when using large opportunistic ecological databases. As outlined in standard textbooks on statistical learning ( 14 ; 17 ) , it is recommended that, if the dataset is large enough, evaluation is conducted by splitting the available data into training, validation and test data, where validation data are used for model selection and test data are used to evaluate the final model predictions. An alternative evaluation strategy that allows the entire dataset to be used for model training is cross-validation. Cross-validation involves splitting the data into multiple folds and iteratively training models, each time using one fold as the test set and the remaining folds for training. The resulting performance statistics are then communicated to describe the quality of the predictions for the study area (i.e., the map accuracy). Cross-validation has proven especially useful during model selection, including hyperparameter tuning ( 33 ; 15 ) and predictor selection ( 25 ; 26 ) . However, if cross-validation is used both for model selection and for evaluating the final model, it can lead to data leakage ( 14 ; 17 ) , hence using a separate test set is favourable. Still, when the training dataset is small, as is often the case in ecology and geosciences, relying on cross-validation performance to estimate final map accuracy may represent a compromise between obtaining an unbiased evaluation and ensuring adequate model training. Consequently, in the majority of studies conducted in these fields, the model selection – and often even the final map evaluation – is solely based on cross-validation ( 37 ; 2 ; 7 , e.g.,) . Both strategies – using a distinct test set, or applying cross-validation – rely on splitting the data into training and test data. In this context, the way the data are split becomes critical. In the following, we first discuss existing approaches to data splitting before proposing a new strategy for data partitioning that, we argue, provides a consolidated perspective on existing strategies. Traditionally, and coming from an assumption of data being independent and identically distributed (iid), this data splitting was done in a random way (i.e., a single random subset used as test data or random cross-validation). However, since the distribution of spatial ecological data can rarely be regarded as iid ( 34 ; 16 ; 24 , see Figure 1 in) , spatial cross-validation approaches have been suggested. Spatial cross-validation refers to a group of cross-validation strategies where spatial resampling is applied to prevent test points from being located close to the training points. These spatial resampling methods can be similarly applied to split the data into a single training and test set. A common approach to spatial cross-validation is spatial block cross-validation where the spatial domain is divided into several blocks, and then iteratively, all data falling into one of the spatial blocks are held back for evaluation ( 30 ; 36 ) . Several other spatial cross-validation strategies have been suggested (see section 3 ), all of them striving for independence between training and test data. Spatial cross-validation has been adopted and is currently used in many ecological fields such as species distribution modeling ( 8 ; 10 , e.g.) or vegetation modelling ( 20 , e.g.) . Notably, and as discussed by 38 , spatial cross-validation is still poorly defined, encapsulating various approaches with numerous assumptions and parameters being proposed. Previous studies have shown that our perception of prediction performance can vary considerably depending on the evaluation strategy used. For example, in the case study presented in 29 , the R^{2} for predicting aboveground forest biomass in central Africa varied between 0.14 and 0.53, depending on the evaluation strategy. 19 demonstrated large differences between the estimated map accuracy resulting from random and a spatial cross-validation for global maps of nematode abundances ( R^{2} of 0.46 vs. 0.19) and specific leaf area ( R^{2} of 0.62 vs. 0.30). Due to the large differences in estimated map quality, a discussion has emerged regarding which of the many resampling strategies should be used in the context of machine learning for