On cross-validation for small area estimators
Qianyu Dong, Zehang Richard Li · 2026
Read On cross-validation for small area estimators by Qianyu Dong, Zehang Richard Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23464.txt.
Introduction
Dong and Li propose a design-based cross-validation method for judging small-area estimators. These tools fill in local health and development figures where survey data are thin. Their score rests on a decomposition of estimation error, with a bound on an unidentifiable bias serving as a ranking threshold. They also show a popular leave-one-area-out shortcut targets the wrong error and can flip the ranking. They apply the method to female literacy estimates in Zambia.
Catalog record
| CREATOR | Qianyu Dong, Zehang Richard Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76033 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23464
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On cross-validation for small area estimators Qianyu Dong, Zehang Richard Li Abstract Subnational monitoring of public health often relies on household surveys where data are sparse at the desired spatial resolution. Small area estimation (SAE) methods address this challenge by borrowing strength across areas and incorporating auxiliary information. However, comparing these estimators remains difficult in the absence of ground truth. We propose a cross-validation framework for evaluating small area estimators that accommodates complex survey designs. Our approach enables model-agnostic comparisons between area-level and unit-level models. Central to our framework is a decomposition of the cross-validated squared error in the context of SAE, which reveals both identifiable bias and unidentifiable components that can be bounded. Our theoretical results and simulation studies show that conventional approaches, such as leave-one-area-out cross-validation, can yield misleading model rankings, whereas the proposed approach offers more robust and interpretable model comparison with uncertainty quantification. We demonstrate the procedure for comparing SAE models for mapping the female literacy rate using Demographic and Health Surveys from Zambia. keywords survey sampling; small area estimation; spatial smoothing; model checking; cross-validation. 1 Introduction Small area estimation (SAE) refers to the process of estimating quantities of interest for specific geographic areas or subpopulations using survey data. When within-area sample sizes are too small for design-based direct estimation, model‐based methods borrow information across areas and incorporate auxiliary information to improve the precision of estimates ( Rao & Molina 2015 ) . In most low- and middle-income countries (LMICs), census or administrative microdata are limited. Routinely collected household surveys serve as the only reliable sources for many key health and demographic indicators. Demographic and Health Surveys (DHS) ( The DHS Program 2026 ; Corsi et al. 2012 ) and the Multiple Indicator Cluster Surveys (MICS) ( UNICEF 2026 ; Khan & Hancioglu 2019 ) are two prominent examples that have been implemented in many countries over several decades. Small area estimation methods have been developed and applied in this setting for outcomes including poverty, child mortality, fertility, vaccination, and HIV ( Chi et al. 2022 ; Mercer et al. 2015 ; Li et al. 2019 ; Wakefield et al. 2019 ; Wu et al. 2021 ; Saha et al. 2023 ; Dong & Wakefield 2021 ; Wakefield et al. 2020 , see e.g., ) . SAE models are generally categorized into either area-level or unit-level models. The most widely used area-level model is the Fay-Herriot model ( Fay & Herriot 1979 ) , where the direct estimates and their associated estimates of the sampling variances are used as response data in a smoothing model. In contrast, unit-level approaches model individual survey responses, often via generalized linear mixed models. No single modeling strategy performs uniformly well across different survey designs and spatial dependence structures. Model evaluation and selection, however, remains challenging for SAE models. Existing approaches are either restricted to comparing models within shared likelihood families or require external benchmark data or assumptions of the true data generating mechanism. There is a critical need for a unified, model-agnostic framework for comparing candidate estimators on a common and scientifically meaningful scale. In this paper, we develop a cross-validation framework for comparing small area estimators fitted to complex survey data. Our approach evaluates estimators through held-out, design-based direct estimates constructed from sample splits that preserve the survey design. This strategy enables model-agnostic comparisons between area-level and unit-level models on a common scale. We derive a decomposition of conditional mean squared errors for small area estimators, leading to a consistent cross-validation score. In finite samples, we show that the unidentifiable bias in score difference between two models can be bounded, which provides a principled threshold for deciding whether an observed score difference is large enough to support model ranking. We also demonstrate that popular alternative model comparison tools, such as leave-one-area-out cross-validation, produce systematic bias in model ranking. All theoretical results are supported via design-based simulations based on the 2024 Zambia DHS. The rest of the paper is organized as follows. Section 2 reviews current approaches to evaluating small area estimators in the literature and discusses their limitations. Section 3 develops the proposed scoring and cross-validation framework. Section 4 reports simulation results and applies the methodology to compare estimators of female literacy rate in Zambia. Section 5 concludes with a discussion of future directions. All proofs are deferred to the Supplementary Materials. 2 Current approaches Existing model evaluation approaches fall broadly into three categories: benchmark-based, likelihood-based, and empirical simulation. None of the approaches provides a generally applicable, model-agnostic framework for comparing estimators. The first class of methods involves evaluating candidate estimators in terms of their agreement with some benchmark values, and is often model-agnostic. External validation against census data is widely regarded as the gold standard for model assessment ( Merfeld et al. 2024 ; Merfeld et al. 2022 ) , and when census is not available, validation against estimates from independent surveys has been proposed ( Brown et al. 2001 ; Dorfman 2018 ) . However, external census or surveys are rarely available at the required spatial resolution. Alternatively, one may first aggregate SAE estimates to coarser levels, e.g., a single national estimate, and then compare them with external benchmark values. While being useful for falsifying estimators, aggregated evaluation changes the target estimand completely. Estimators that perform well at the aggregated level can still be significantly biased at the finer levels, especially with over-smoothed models. When no external data are available, direct estimates from the same sample are also sometimes used to assess small area estimators ( Lahiri & Pramanik 2019 ) . However, such evaluation suffers from double dipping and systematically favors models that track the noise in direct estimates. Traditional model assessment and validation tools, including goodness-of-fit tests ( Rao & Scott 1981 ; Rao & Scott 1984 ; Thomas & Rao 1987 ; Lumley & Scott 2014 ; Lumley & Scott 2017 ) , information criteria ( Fabrizi & Lahiri 2013 ; Lumley & Scott 2015 ) , and generalization error measures ( Holbrook et al. 2020 ) , have been developed for regression models that incorporate survey weights, with adjustments to account for complex sampling designs. Similar approaches for comparing Bayesian SAE models using measures such as DIC and WAIC have also been adopted in the literature ( Trevisani et al. 2017 ; Gomez-Rubio et al. 2008 ; Franco & Maitra 2023 , see, e.g., ) . While likelihood-based methods are powerful and widely used, they are generally limited to comparing models that share a comparable likelihood formulation. Consequently, they are not well suited for comparing unit-level with area-level models, or models that target the same estimand through fundamentally different constructions. Lastly, empirical simulation approaches are also commonly used in the SAE literature ( Bradley et al. 2015 ; Janicki et al. 2022 ) . These approaches perturb observed direct estimates by adding noise, and then evaluate models fitted to the synthetic direct estimates against the original estimates. When some individual-level micro data from the population exist, design-based simulation may also be carried out by generating repeated surveys based on a synthetic population dataset ( Alfons et al. 2010 ; Wieczorek & Franco 2013 ) . Both approaches evaluate model performance for a hypothetical