Enhancing Forecasts Using Real-Time Data Flow and Hierarchical Forecast Reconciliation, with Applications to the Energy Sector
Lukas Neubauer, Peter Filzmoser · 2024
Read Enhancing Forecasts Using Real-Time Data Flow and Hierarchical Forecast Reconciliation, with Applications to the Energy Sector by Lukas Neubauer, Peter Filzmoser (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01528.txt.
Description
Abstract August 24, 2026 Abstract A novel framework for hierarchical forecast updating is presented, addressing a critical gap in the forecasting literature. By assuming a temporal hierarchy structure, the innovative approach extends hierarchical forecast reconciliation to effectively manage the challenge posed by partially observed data.
Catalog record
| CREATOR | Lukas Neubauer, Peter Filzmoser |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01528
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Forecasts Using Real-Time Data Flow and Hierarchical Forecast Reconciliation, with Applications to the Energy Sector Lukas Neubauer, Peter Filzmoser Abstract August 24, 2026 Abstract A novel framework for hierarchical forecast updating is presented, addressing a critical gap in the forecasting literature. By assuming a temporal hierarchy structure, the innovative approach extends hierarchical forecast reconciliation to effectively manage the challenge posed by partially observed data. This crucial extension allows, in conjunction with real-time data, to obtain updated and coherent forecasts across the entire temporal hierarchy, thereby enhancing decision-making accuracy. The framework involves updating base models in response to new data, which produces revised base forecasts. A subsequent pruning step integrates the newly available data, allowing for the application of any forecast reconciliation method to obtain fully updated reconciled forecasts. Additionally, the framework not only ensures coherence among forecasts but also improves overall accuracy throughout the hierarchy. Its inherent flexibility and interpretability enable users to perform hierarchical forecast updating concisely. The methodology is extensively demonstrated in a simulation study with various settings and comparing different data-generating processes, hierarchies, and reconciliation methods. Practical applicability is illustrated through two case studies in the energy sector — energy generation and solar power data — where the framework yields superior results compared to base models that do not incorporate new data, leading to more precise decision-making outcomes. 1 Introduction In many time series forecasting scenarios, a temporal hierarchical structure exists. This hierarchy is constructed implicitly once one or multiple steps of temporal aggregation are applied to a time series of interest. In a common example, one considers both quarterly and annual observations of an economic indicator where aggregation by summing is appropriate. Naturally, the aggregated quarterly observations should equal the annual value. When addressing the forecasting aspect of this problem, several challenges arise. Typically, the different levels of the hierarchy are modeled and forecasted independently. This approach allows for flexibility in selecting the modeling techniques. However, this leads to incoherent forecasts, i.e. the aggregated quarterly forecasts do not align with the annual forecast. To resolve this issue, hierarchical forecast reconciliation has been developed to adjust those base forecasts into coherent forecasts through a post-hoc procedure. Initially applied to cross-sectional hierarchies, the concept of hierarchical forecast reconciliation has been extended to temporal hierarchies, too. The reconciled forecasts not only become coherent but also achieve greater accuracy. Thus, hierarchical forecast reconciliation serves as a correction for model misspecification in the base models. In analyzing temporal hierarchies of time series, an important question emerges. The fact that lower levels of the hierarchy are observable at a higher frequency should be reflected throughout the entire hierarchy. For instance, if two quarters have already been observed, this should impact both the remaining quarterly forecasts and the annual forecast. Generally, the higher frequency time series can be updated more frequently, and each update should affect higher level forecasts accordingly. A commonly used and effective method is bottom-up aggregation. [ Koreisha and Fang, 2004 ] show that in the setting of temporally aggregated ARMA models, bottom-up forecasts tend to provide more accurate forecasts for the aggregated time series compared to forecasts derived from the aggregated data. This advantage becomes more pronounced as more lower level data becomes available. Interestingly, this effect is only significant for short forecast horizons. Another common concept in this context is Nowcasting . Originally developed for weather forecasting, nowcasting has found increasing applications in economics such as forecasting GDP or inflation using higher frequency indicators. The statistical background is rather complex. By modeling the data via a state space representation, the ”news” can be appropriately incorporated. Examples of such representations are factor models and vector autoregressive models. A traditional approach involves bridging equations to relate lower frequency data to higher frequency data. Often, these models are tailored for specific hierarchies, such as aggregating daily data to monthly, making them less flexible overall. It is important to note that, in nowcasting scenarios, the goal is usually not to explain aggregated data by using the same data at a higher frequency. Instead, a time series of lower-frequency data is explained by other, higher-frequency variables. An extensive overview can be found in [ Bańbura et al., 2013 ] . When discussing the handling of time series of different frequencies, MIxed DAta Sampling (MIDAS) models ( [ Ghysels et al., 2007 ] ) are frequently encountered. These regression models are able to explain a lower frequency target variable by using higher frequency regressors, making them suitable for nowcasting scenarios as well. Initially developed for economic applications, the primary aim of those models was to regress a target variable on indicators observed at a higher frequency. However, our motivation differs, as we seek to forecast a time series of interest aggregated at various frequencies and update forecasts within this hierarchy of aggregated time series. Additionally, the traditional MIDAS models were not intended to use partially observed data, whereas our approach intends to do so. It was only in subsequent advancements, such as those of [ Mikosch and Neuwirth, 2015 ] , that the use of real-time data and forecast updating were considered. The post-hoc framework for hierarchical forecast reconciliation is currently designed to work only with fully observed data. For example, in the case of a quarterly and annual hierarchy, an entire year of data must be available. Recent research by [ Di Modica et al., 2021 ] presents a recursive and adaptive approach in which an online variant is proposed. However, the authors still do not account for partially observed data and instead rely on complete rows of new data, which does not align with the issue we are facing. Thus, this work addresses the need for a methodology to update forecasts in a hierarchical structure using partially observed data, while still allowing for flexibility in both the model and the hierarchy. Motivated by the benefits and recent research advances of hierarchical forecast reconciliation, we investigate this very flexible framework to perform forecast updating while ensuring coherence across the entire hierarchy. We propose a simple method to employ existing reconciliation techniques using partially observed data to generate updated forecasts. This approach is very flexible as the base models can be arbitrarily chosen, and the reconciliation method can be chosen based on the specific application. Once new data is available, the base forecasts can be updated, followed by a pruning step. On the pruned hierarchy, the actual reconciliation is performed, leading to updated and coherent forecasts. This procedure allows us to theoretically show an improvement in forecast accuracy as new data becomes available. Notably, the rate of improvement increases as the amount of new data grows. The potential of forecast improvements is also demonstrated on simulated data as well as real data examples from the energy sector. The paper is structured as follows. In Section 2 relevant literature is summarised. A short introduction to hierarchical forecast reconciliation is given in Section 2.1 , followed by its extension to temporal hierarchies (Section 2.2 ). The hierarchical forecast updating framework is given in Section 3.1 and is analyzed theoretically in Section 3.2 . Finally, extensive simulation studies (Section 4 ) and real data applications (Section 5 ) are presented. Here, real data from the energy