Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances

Michael Balzer · 2025

Read Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances by Michael Balzer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13682.txt.

Introduction

Balzer wrote a solo study in twenty twenty-five on how to fit models to data that varies across place. Nearby towns and zones tend to sway each other, so plain fits can miss that link. He puts forth a step-by-step boost method built for such place-based models.

The method can pick which facts to keep, even when there are more facts than cases to learn from. It also curbs wild swings in the fit, which helps it guess well on new places. Tests with fake data show the method works as hoped. A case study then fits life span across German zones with place links in mind.

Catalog record

CREATORMichael Balzer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13682

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances Michael Balzer Abstract Researchers in urban and regional studies increasingly deal with spatial data that reflects geographic location and spatial relationships. As a framework for dealing with the unique nature of spatial data, various spatial regression models have been introduced. In this article, a novel model-based gradient boosting algorithm for spatial regression models with autoregressive disturbances is proposed. Due to the modular nature, the approach provides an alternative estimation procedure which is feasible even in high-dimensional settings where established quasi-maximum likelihood or generalized method of moments estimators do not yield unique solutions. The approach additionally enables data-driven variable and model selection in low- as well as high-dimensional settings. Since the bias-variance trade-off is also controlled in the algorithm, implicit regularization is imposed which improves prediction accuracy on out-of-sample spatial data. Detailed simulation studies regarding the performance of estimation, prediction and variable selection in low- and high-dimensional settings confirm proper functionality of the proposed methodology. To illustrative the functionality of the model-based gradient boosting algorithm, a case study is presented where the life expectancy in German districts is modeled incorporating a potential spatial dependence structure. keywords Spatial regression models, Gradient boosting, Statistical learning, Variable selection, Generalized moment estimator 1 Introduction In various real-world applications, applied researchers have to engage increasingly with data that exhibits distinct locational attributes, which establishes a connection between objects in space. Such spatial data usually includes a cross-sectional variable in which each observation corresponds to a spatial unit like a geographical location. Thus, dealing with spatial data poses certain challenges since classical statistical regression models heavily rely on the independence assumption between observations which simplifies the model but is commonly violated in the presence of spatial units. Therefore, so-called spatial regression models have been developed which explicitly incorporate the spatial dependence structure of the data ( LeSage and Pace 2009 ) . Particularly, spatial dependencies can arise due to spatial autocorrelation which describes similarities between geographical locations in space. For instance, housing prices in cities might be more similar in neighboring locations of close proximity. Thus, under the assumption that the spatial dependencies only arise through the error terms, Cliff and Ord 1973 proposed based on Whittle 1954 to model the spatial autocorrelation in the disturbance process. Then, for any given variable of interest, the error term for each location depends on a weighted average of the disturbances in connected locations. Thus, spatial spillovers between observations are quantified through the spatial structure of the disturbances ( Anselin 1988 ) . Considering that the complexity of the linear predictor can potentially grow quickly in size depending on the modeling choices of the practitioner, approaches to model choice and variable selection have become increasingly more important. Particularly, many candidate spatial regression models with autoregressive disturbances can be chosen by the practitioner by the inclusion of additional spatially lagged independent variables in the linear predictor. However, model choice is non-trivial since modeling spatial dependencies can induce a non-nested model structure. If model estimation is based on the quasi-maximum likelihood (QML), a quantifiable value for model comparison can be obtained based on a likelihood ratio test ( Lee 2004 ; Liu and Lee 2019 ) . Instead of utilizing specifically designed criteria, regularization techniques are popular alternatives for model choice and variable selection ( Fahrmeir et al 2013 ) . Particularly, the least absolute shrinkage and selection operator ( Tibshirani 1996 ) is a popular example of such a regularization technique which has also been recently extended to spatial regression model with autoregressive disturbances ( Cai et al 2019 ; Cai and Maiti 2020 ) . Another option in classical linear regression models is the so-called model-based gradient boosting algorithm ( Fahrmeir et al 2013 ; Hepp et al 2016 ) . Although originally proposed in the domain of machine learning for classification problems, gradient boosting has been extended for statistical regression models where it is known as component-wise, model-based or statistical gradient boosting. In principle, the algorithm is iterative in nature where the estimation problem reduces to fitting base-learners to the negative gradient of a prespecified loss function related to the statistical model of interest. In each iteration of the algorithm only the best performing base-learner is chosen from which a small fraction is added to the current linear predictor. Stopping the algorithm early allows for data-driven variable selection and yields interpretable results at each iteration ( Mayr et al 2014 ) . Due to the modular and iterative nature of model-based gradient boosting, it remains a feasible approach even in high-dimensional data settings where the number of variables exceeds the number of observations ( Bühlmann 2006 ; Bühlmann and Hothorn 2007 ) . Therefore, it is not surprising that model-based gradient boosting has been extended to a variety of regression models of increased complexity (see, for example, Jobst et al 2024 ; Balzer et al 2025 ). Although sparse boosting for semi-parametric additive spatial autoregressive models has been recently explored ( Yue and Xi 2025 ) , no prior work appears to have addressed a potential extension of model-based gradient boosting for (parametric) spatial regression models with autoregressive disturbances. To this end, the model-based gradient boosting algorithm in the mboost package ( Bühlmann and Hothorn 2007 ; Hothorn et al 2010 ; Hofner et al 2014 ; Hofner et al 2015 ) for fitting generalized linear, additive and interaction models of potential high-dimensionality in the programming language R ( R Core Team 2025 ) is extended to accommodate spatial regression models with autoregressive disturbances. To investigate proper functionality of the proposed model-based gradient boosting algorithm, in-depth simulation studies in low- and high-dimensional linear settings are conducted. The focus lies primarily on the evaluation of estimation, variable selection and prediction. To illustrate the potential application of model-based gradient boosting for spatial regression models with autoregessive disturbances, an application concerned with modeling the life expectancy in German districts is presented which heavily draws on the ”Indikatoren und Karten zur Raum- und Stadtentwicklung” (INKAR) data base which includes a rich variety of variables ( Bundesinstitut für Bau-, Stadt- und Raumforschung (2024) BBSR ) . The structure in this article is as follows: In Section 2 , the mathematical, theoretical framework of spatial regression models with autoregressive disturbances and model-based gradient boosting is introduced where both concepts are combined thereafter. Afterward, outcomes for the simulation studies are discussed in Section 3 . A description of the context, data situation and variables for the case study as well as the results for the application of model-based gradient boosting are presented in Section 4 . The article finishes then with a conclusion and a discussion in Section 5 . 2 Methodology 2.1 Spatial regression models with autoregressive disturbances Let n\in\mathbb{N} denote the number of observations in a spatial dataset. For each i\in\{1,\dots,n\} , consider following spatial regression model with autoregressive disturbances \displaystyle\bm{y} \displaystyle=\bm{X}\bm{\beta}+\bm{W}\bm{X}\bm{\theta}+\bm{u} (1) \displaystyle\bm{u} \displaystyle=\lambda\bm{W}\bm{u}+\bm{\epsilon} where \bm{y} is a n\times 1 vector of observations, \bm{X} is the n\times p design matrix of p\in\mathbb{N} exogenous variables, \bm{\beta} are corresponding p\times 1 coefficients and \bm{u} is the n\times 1 vector of disturbances. The spatial dependence in the data is assumed to enter the model in two ways.

Read the complete text