Bounded-Abstention Multi-horion Time series Forecasting Luca Stradiotti, Laurens Devos, Anna Monreale, Jesse Davis, Andrea Pugnana Abstract Multi-horizon time-series forecasting involves simultaneously making predictions for a consecutive sequence of subsequent time steps. This task arises in many application domains, such as healthcare and finance, where mispredictions can have a high cost and reduce trust. The learning with abstention framework tackles these problems by allowing a model to abstain from offering a prediction when it is at an elevated risk of making a misprediction. Unfortunately, existing abstention strategies are ill-suited for the multi-horizon setting: they target problems where a model offers a single prediction for each instance. Hence, they ignore the structured and correlated nature of the predictions offered by a multi-horizon forecaster. We formalize the problem of learning with abstention for multi-horizon forecasting setting and show that its structured nature admits a richer set of abstention problems. Concretely, we propose three natural notions of how a model could abstain for multi-horizon forecasting. We theoretically analyze each problem to derive the optimal abstention strategy and propose an algorithm that implements it. Extensive evaluation on 24 datasets shows that our proposed algorithms significantly outperforms existing baselines. 1 Introduction Figure 1: Illustration of possible forecasts when using abstention in a multi-horizon setting. (a) The model offers a forecast for the entire horizon, i.e. , the full forecast is accepted. (b) Standard abstention decides independently for each time step whether to accept or abstain from making a prediction, hence the forecast is not guaranteed to be provided for a contiguous time interval. (c) Full abstention only allows rejecting the forecast for the entire horizon. (d) Partial abstention lets the model provide a forecast for part of horizon: the forecast must start from the first time step but it may choose the end time step. (e) Interval abstention offers the most flexibility: it can offer a forecast for any contiguous interval within the horizon, i.e. , it can choose any start and end point. Multi-horizon time-series forecasting involves simultaneously making predictions for a consecutive sequence of subsequent time steps. This problem arises in a wide range of applications, including predicting asset prices [ 4 , 33 ] , providing weather forecasts [ 30 ] , modeling a patient’s prognosis [ 1 , 21 ] , and predicting how an athlete’s skill will change over time [ 38 ] . Many multi-horizon problems occur in high-stakes settings ( e.g. , medicine) where incorrect predictions can have a high cost [ 29 ] . Therefore, it is appealing to allow a model to abstain from making a prediction in certain situations [ 31 ] . For example, abstaining may be appropriate when a model is at heightened risk of making a misprediction, e.g. , because it is highly uncertain about its prediction. In these cases, a decision could be deferred to a human or more information can be collected [ 18 ] . However, most existing abstention strategies target standard classification and regression problems where a single value (or class) is predicted for each instance. Unfortunately, these methods are ill-suited for the multi-horizon setting because they all treat each prediction separately. That is, for each time step in the horizon, they will independently decide whether to offer a prediction or abstain. This is disadvantageous because, on the one hand, it disregards the clear correlations between consecutive time steps. On the other hand, ignoring the structure of the predictions can lead to fragmented forecasts as illustrated in Figure 1 b. However, in many applications it is necessary to receive a continuous (sub)sequence of predictions for future time steps to identify trends or the onset events. For instance, in weather forecasting, a user relies on a continuous sequence of daily predictions to effectively plan or pack for a trip. Therefore, our goal is to generalize abstention strategies to account for the inherently richer and more structured nature of the multi-horizon setting. Concretely, we propose and explore three natural multi-horizon abstention settings: Full abstention is where the forecast is either given for all time steps or none as illustrated in Figure 1 c. This can be viewed as the natural adaptation of abstention for single step predictions to the multi-horizon setting because for each test time series, either a forecast is offered or it is not. For example, this may be appropriate when the test time series is overly noisy due to, e.g. , a poorly attached sensor that produces motion artifacts. Partial abstention is where the forecast is offered for a consecutive sequence of time steps starting from the first time step in the prediction horizon as shown in Figure 1 d. For example, a model may produce a more confident forecast about an athletes’ performance for the next several seasons, but may highly uncertain about its estimates of later career performance as age-related declines vary substantially among athletes. Interval abstention generalizes the partial abstention setting by offering a forecast for a consecutive sequence of time steps that may start from any point in the prediction horizon as shown in Figure 1 e. For example, in traffic forecasting, it may be difficult to estimate delays immediately following a crash, 1 1 1 This depends on severity, emergency services’ proximity, etc. but the model will be more certain about its predictions in hour as traffic should return to normal by then. We formalize each of these settings using the bounded-abstention framework [ 11 ] , which aims to optimize forecasting performance subject to a target coverage constraint ( i.e. , the proportion of examples for which a forecast is offered). For each abstention setting, we theoretically characterize the problem, provide the optimal abstention strategy, and propose an algorithm that approximates the optimal abstention strategy. We extensively validate our framework on 24 real-world datasets, demonstrating that our approaches significantly outperform several baselines. 2 Formalizing bounded-abstention for multi-horizon time-series forecasting We study the forecasting setting where the goal is to learn a forecasting model that generalizes to unseen time series ( e.g. , learn from past patient data to provide forecasts for future patients). We assume time series of the form y_{1:T}:=(y_{1},y_{2},...,y_{T}) , where y_{i}\in\mathbb{R}^{d} and i represents a time step. We define a selective multi-horizon forecaster m as a pair (f,g) , where f:\mathcal{Y}^{1:T}\to\mathcal{Y}^{T+1:T+H} is a multi-horizon time-series forecaster that given an input time series y_{1:T} , predicts the values of its subsequent H time steps, where H is called the forecast horizon; g:\mathcal{Y}^{1:T}\to\{1,\dots,H\}\times\{0,\dots,H\} is a selection function that determines the window within the horizon H for which a forecast is offered (accepted steps). For any time step outside this interval, the forecaster abstains (rejected steps). That is, g\left(y_{1:T}\right)=(s,e) with s\geq 1 and e\leq H denoting the start and the end of the forecast interval. For ease of notation, we omit the explicit dependence on the input time-series and use (s,e) instead of (s(y_{1:T}),e(y_{1:T})) . We adopt the convention that (s=1,e=0) denotes that the entire horizon is rejected; apart from this exception we require s\leq e . Formally, the output of m for a time point T+1\leq i\leq T+H is defined as: m_{i}\left(y_{1:T}\right)=\begin{cases}\text{\textregistered}&\text{if $ie$}\\ f_{i}\left(y_{1:T}\right)&\text{otherwise}.\end{cases} (1) That is, the forecasts for the time steps that satisfy the first condition are rejected whereas the forecasts for the remaining time steps are accepted . We measure the performance of a selective multi-horizon forecaster using the selective risk , i.e. , the expected loss over the accepted steps. Letting h(g)=e-s+1 be the number of accepted steps and \phi(g)=\mathbb{E}_{Y_{1:T}}\left[h(g)\right] , we define the selective risk as: \mathcal{R}(f,g)=\frac{\mathbb{E}_{Y_{1:T+H}}\left[\sum_{t=T+s}^{T+e}\ell\left(y_{t},f_{t}\left(y_{1:T}\right)\right)\right]}{\phi(g)} (2) where \ell\left(y_{t},f_{t}\left(y_{1:T}\right)\right) is the loss at time-step t . Intuitively, the numerator represents the total expected loss over the accepted steps, while the denominator represents the expected length of the forecast interval. To avoid degenerate solutions where the forecaster always abstains, we enforce a target coverage c , requiring the model to offer forecasts for at least a fraction c of the steps in the horizon on average. Therefore, the objective is to find a selection function g that minimizes the selective risk subject to this coverage constraint [ 27 ] . The general optimization problem can be formalized as: g^{*}=\underset{g}{\arg\min}\;\mathcal{R}(f,g)\quad\text{s.t.}\quad\phi(g)\geq cH. (3) We now define three different ways to impose structural constraints on how g selects the indices s and e , each of which yields a distinct bounded-abstention problem. We theoretically analyze each problem and derive the optimal selection function. In our analysis, we make two assumptions. First, we assume exchangeability across different time-series , but we avoid assumptions about dependencies across time steps [ 34 ] . Second, we assume that f produces all H predictions simultaneously from the input sequence ( i.e. , they are not generated autoregressively). This choice is motivated by the robustness of this approach to error accumulation and conditionally independent predictions [ 39 ] . 2.1 Full abstention for multi horizon forecasting We first consider abstention at the time-series level : Definition 1 (Full Abstention) . The selective forecaster must accept or reject the entire forecast horizon. Therefore, the selection function in the full abstention setting is defined as g_{\texttt{FA}}:\mathcal{Y}^{1:T}\rightarrow\{(1,H),(1,0)\} . The goal is to minimize the risk over the accepted time-series while ensuring that, on average, forecasts are provided for at least a fraction c of the time-series. 2 2 2 This is equivalent to forecast cH time steps on average. Ideally, g should abstain from those time-series with higher expected risk, i.e. , those for which the forecaster is most uncertain. The core challenge lies in evaluating the total expected risk across the entire horizon, while traditional abstention strategies evaluate it for a single prediction. We now show that the optimal selection function abstains whenever the sum of conditional risks over the entire horizon exceeds a threshold: Theorem 1 . Given a forecaster f and a target coverage c\in(0,1] , the optimal selection function g^{*}_{\texttt{FA}} that is a solution to Equation 3 in full abstention is: g^{*}_{\texttt{FA}}\left(y_{1:T}\right)=\begin{cases}(1,H)&\text{ if }\quad\sum_{t=T+1}^{T+H}\rho_{t}\left(y_{1:T}\right)<\tau^{*}_{c}\\ (1,\xi)&\text{ if }\quad\sum_{t=T+1}^{T+H}\rho_{t}\left(y_{1:T}\right)=\tau^{*}_{c}\\ (1,0)&\qquad\;\text{ otherwise }\end{cases} (4) where \xi is a random variable such that \xi=H with probability \kappa and \xi=0 with probability 1-\kappa , and \rho_{t}\left(y_{1:T}\right)=\mathbb{E}_{Y_{T+1:T+H}|Y_{1:T}}\left[\ell\left(y_{t},f_{t}(y_{1:T})\right)\mid Y_{1:T}=y_{1:T}\right] \tau^{*}_{c}=\inf\left\{v\mid\mathbb{E}_{Y_{1:T}}\left[\mathds{1}\left\{\sum\limits_{t=T+1}^{T+H}\rho_{t}\left(y_{1:T}\right)