Decision-aware Training of Spatiotemporal Forecasting Models to Select a Top-K Subset of Sites for Intervention

Kyle Heuton, F. Samuel Muench, Shikhar Shrestha, Thomas J. Stopka, Michael C. Hughes · 2025

Read Decision-aware Training of Spatiotemporal Forecasting Models to Select a Top-K Subset of Sites for Intervention by Kyle Heuton, F. Samuel Muench, Shikhar Shrestha, Thomas J. Stopka, Michael C. Hughes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05622.txt.

Introduction

Kyle Heuton and four co-authors wrote a 2025 paper on training forecast models to choose where interventions will do the most good. Agencies can treat only a few sites, say for overdose prevention. Models that forecast event counts at each site could guide the choice, but forecast accuracy and decision quality are different targets.

The team uses a metric comparing a chosen top-K set's reach against the best possible set in hindsight. Ranking sites by predicted mean turns out to be suboptimal for that metric; decision theory yields a better ranking from a probabilistic model. Training toward the metric is harder, because picking a discrete set kills gradients. Perturbed optimizer techniques restore them. A combined objective keeps forecasts honest while sharpening the ranking. Tests on overdose site selection show the gains.

Catalog record

CREATORKyle Heuton, F. Samuel Muench, Shikhar Shrestha, Thomas J. Stopka, Michael C. Hughes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decision-aware Training of Spatiotemporal Forecasting Models to Select a Top-K Subset of Sites for Intervention Kyle Heuton, F. Samuel Muench, Shikhar Shrestha, Thomas J. Stopka, Michael C. Hughes Abstract Optimal allocation of scarce resources is a common problem for decision makers faced with choosing a limited number of locations for intervention. Spatiotemporal prediction models could make such decisions data-driven. A recent performance metric called fraction of best possible reach (BPR) measures the impact of using a model’s recommended size K subset of sites compared to the best possible top-K in hindsight. We tackle two open problems related to BPR. First, we explore how to rank all sites numerically given a probabilistic model that predicts event counts jointly across sites. Ranking via the per-site mean is suboptimal for BPR. Instead, we offer a better ranking for BPR backed by decision theory. Second, we explore how to train a probabilistic model’s parameters to maximize BPR. Discrete selection of K sites implies all-zero parameter gradients which prevent standard gradient training. We overcome this barrier via advances in perturbed optimizers. We further suggest a training objective that combines likelihood with a BPR constraint to deliver high-quality top-K rankings as well as good forecasts for all sites. We demonstrate our approach on two where-to-intervene applications: mitigating opioid-related fatal overdoses for public health and monitoring endangered wildlife. 1 Introduction Statistical machine learning methods for spatiotemporal forecasting can play a vital role in high-stakes applications from mitigating overdoses in public health ( Marks et al., 2021b ) to forest fire management ( Cheng and Wang, 2008 ) to wildlife monitoring ( Golden et al., 2022 ; Hefley et al., 2017 ) . Across these domains, there is a pressing need for predictive models that can make accurate predictions of near-term future events at fine spatiotemporal resolutions. Such models can enable inform data-driven decisions about how to allocate limited resources to maximize utility. In this work, we seek to help decision-makers select where to intervene . Given historical data for a fixed set of S candidate spatial sites, we develop models that can recommend a specific subset of given size K for some action or intervention. We think of hyperparameter K as setting the budget for interventions. In an ideal world, decision-makers could afford interventions in all S sites. However, when resource constraints allow only K sites to receive interventions, a decision selecting a specific K -of- S subset is required. While such decisions may often be heuristic in current practice, we hope to offer data-driven solutions. With this goal in mind, choosing a sensible performance metric is critical to assessing which models have real-world utility. Common metrics such as squared error or absolute error are not well matched to where-to-intervene decisions because they treat all sites equally. Recent work on overdose forecasting has suggested a metric termed the fraction of best possible reach , or BPR ( Heuton et al., 2022 ; Heuton et al., 2024 ) . BPR measures a ratio of event counts. The numerator sums over the model’s recommended K sites, while the denominator sums over the best possible K selected in hindsight. This type of evaluation has been used in a preregistered trial ( Marshall et al., 2022 ) for assessing forecasts of opioid overdoses in Rhode Island, as well as a follow-up feasability study ( Allen et al., 2023 ) . BPR is applicable to many where-to-intervene problems beyond public health. † † footnotetext: Code: https://github.com/tufts-ml/decision-aware-topk While some publications have reported BPR in evaluations, we suggest that a natural goal would be for this performance metric to inform two other key parts of data-driven decision-making: model-based ranking and model training . By ranking, we mean that given fixed model parameters, the knowledge of BPR as the metric of interest should impact the numerical score assigned to each site to determine the top K . By training, we mean how to update model parameters to achieve high BPR. This paper contributes new methods for solving both ranking and training problems when BPR is the preferred metric. Our work overcomes several technical barriers. The first barrier is in ranking. Given a fixed probabilistic model, determining how to compute a per-site score, which will later be sorted to find the top K , is not obvious. It may be tempting to use the model’s per-site mean, but decision theory suggests not all loss functions recommend the per-site mean as the best estimator ( Berger, 2013 ; Murphy, 2022 ) . As a relatively new metric, the problem of how to assign a numerical ranking to sites for optimal BPR decision-making is currently open. We contribute a tractable ranking method that is provably best for a reasonable bound on BPR. We further show how the score function estimator ( Kleijnen and Rubinstein, 1996 ; Mohamed et al., 2020 ) can be used to calculate gradients of this ranking with respect to parameters. The second barrier prevents training parameters to improve BPR. First-order gradient descent is a common, effective algorithm we would like to use. However, gradients of BPR with respect to parameters are problematic. While small changes to parameters induce some changes to per-site scores, only changes large enough to move a site into or out of the top-K ranked sites will adjust BPR. Thus, gradients of BPR with respect to parameters will be zero almost everywhere, preventing gradient methods from ever moving beyond subpar initial parameters. To fix this, we leverage recent advances in perturbed optimizers ( Abernethy et al., 2016 ; Berthet et al., 2020 ) to yield effective and efficient gradient estimation for BPR. Our team explored this idea for optimizing BPR alone in earlier non-archival work ( Heuton et al., 2023 ) ; this paper offers an expanded treatment with more accessible presentation, while addressing two more barriers. The final barrier is designing a training objective to achieve applied goals. We find that optimizing BPR alone can lead to predictions with far lower likelihood than conventional training. This raises concerns about overall model quality and generalization. To address this, we pose a constrained optimization problem to maximize likelihood subject to a BPR quality constraint. This combined objective delivers quality top-K recommendations and good forecasts for all sites. Our objective is reminiscent of past additive combinations of a regression loss and decision loss ( Kao et al., 2009 ) . Unlike that work, ours pursues non-convex losses and directly enforces decision quality via constraints. We ultimately contribute methods for how-to-rank and how-to-train when making where-to-intervene decisions. Using these tools, a variety of models can be directly optimized to make effective top K site recommendations. We demonstrate these contributions first on synthetic data, where we reveal how off-the-shelf methods without our innovations can be suboptimal for decision-making. We further evaluate against alternatives on two applications: mitigating opioid-related fatal overdoses and monitoring endangered birds. We hope our contributions spark interest in where-to-intervene problems in the methodological community and also lead to effective deployments of data-driven top-K decision-making in public health and beyond. Related work. The application of machine learning to decision-making problems is widely studied in operations research literature ( Bertsimas and Kallus, 2020 ; Sadana et al., 2025 ) , including the problem of

Read the complete text