Frequentist forecasting in regime-switching models with extended Hamilton filter

2025

Read Frequentist forecasting in regime-switching models with extended Hamilton filter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18149.txt.

Introduction

Kento Okuyama and coauthors build a forecasting tool for models of psychological change, such as a student's drift toward dropping a math course. Such processes jump between hidden states, and regime switching state space models capture the jumps with intensive longitudinal data.

Their filter is the first frequentist one for this model class, built on Chow and Zhang's extended Kim filter. It lets the hidden Markov switching depend on latent traits measured both within and between people. Forecasts can then update in real time while data are still being collected. In a study of math dropout, the filter forecasts emotions and behavior tied to quitting. Simulations check how well it recovers parameters and predicts the hidden regimes.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Frequentist forecasting in regime-switching models with extended Hamilton filter Abstract Psychological change processes, such as university student dropout in math, often exhibit discrete latent state transitions and can be studied using regime-switching models with intensive longitudinal data (ILD). Recently, regime-switching state-space (RSSS) models have been extended to allow for latent variables and their autoregressive effects. Despite this progress, estimation methods for handling both intra-individual changes and inter-individual differences as predictors of regime-switches need further exploration. Specifically, there’s a need for frequentist estimation methods in dynamic latent variable frameworks that allow real-time inferences and forecasts of latent or observed variables during ongoing data collection. Building on Chow and Zhang’s (2013) extended Kim filter, we introduce a first frequentist filter for RSSS models which allows hidden Markov(-switching) models to depend on both latent within- and between-individual characteristics. As a counterpart of Kelava et al.’s (2022) Bayesian forecasting filter for nonlinear dynamic latent class structural equation models (NDLC-SEM), our proposed method is the first frequentist approach within this general class of models. In an empirical study, the filter is applied to forecast emotions and behavior related to student dropout in math. Parameter recovery and prediction of regime and dynamic latent variables are evaluated through simulation study. \submit keywords regime-switching, state-space, forecasting, Kim filter, frequentist Kento Okuyama Tim Fabian Schaffland Pascal Kilian Holger Brandt Augustin Kelava \reprints Correspondence should be sent to Kento Okuyama, Methods Center, University of Tübingen, 72076, Germany. E-Mail: [email protected] \RepeatTitle Frequentist forecasting in regime-switching models with extended Kim filter Abstract Recently, the increasing availability of intensive longitudinal data (ILD) has been brought about by digital devices such as mobile phones and smartwatches. ILD typically consists of data from several subjects, with around thirty to a hundred repeated measures per subject. The data could take various forms, such as self-report, physiological data, and behavioral, environmental, and geographic data ( 24 ) . Parts of such data may be time-invariant, i.e., stable across time points ( inter-individual ; e.g., dispositions, personal traits). Other parts of the data consist of repeated measures whose states are subject to dynamic changes across time within individuals ( intra-individual ; e.g., affective states). Structural equation model (SEM) has been one of the most important methods in psychometrics. Dynamic SEM ( 23 , DSEM;) , which combines SEM and time-series analysis, has been actively studied to investigate the change process in ILD. With increasing number of substantive applications of DSEMs, a number of new ILD methods have been developed ( 15 , e.g.,) . Many psychological change processes are characterized by regime-switching , in which a change process is described by discontinuous transitions from one discrete latent state (i.e., regime) to another at a particular time point ( 10 ; 2 , see e.g.,) . Individuals are allowed to switch between regimes over time. In some cases, it is expected that the transition mostly occurs in a unidirectional manner e.g., from an intention to stay at math lecture to an intention to quit ( 1 , cf., Rubicon model;) . In other cases, subjects could repeatedly switch back and forth between two or more states such as the sleep-wake cycle ( 13 ) . Studying such processes on a within-individual level requires a regime-switching model that assumes that each observation comes from one of the distinct distributions, each of which has a unique set of parameters (e.g., mean, variance, autocorrelation). Since Markov-switching autoregressive (MSAR) models ( 11 ) were introduced, regime-switching models have mostly gained the interest of econometricians. 18 generalized this approach to state-space models, which are called regime-switching state-space (RSSS) models. This framework accommodated a wide range of models, including latent variable models and regime-dependent time-series models ( 27 ) . 7 then developed nonlinear RSSS models, whose state-space representation is governed by nonlinear dynamics. The transition between regimes is characterized by a set of transition probabilities that are invariant across individuals over time. Their estimation approach, called the extended Kim filter, combines the extended Kalman filter and the Hamilton filter to handle the estimation of parameters and latent variables in their nonlinear RSSS models. Filtering ( 4 ) is an important technique as it allows forecasting i.e., real-time inference of unrealized future states and/or events based on available information. Upon each measurement, the prediction is continuously updated, making it suitable for applications which involve real-time monitoring and timely intervention. Upon arrival of the entire time-series observation, their estimation of the latent variables and the regime probabilities could be refined using a smoothing procedure called the extended Kim smoother. 16 addressed forecasting the precursor of student dropout, referred to as the intention to dropout , within their regime-switching modeling framework called the nonlinear dynamics latent class structural equation models (NDLC-SEM). Their flexible NDLC-SEM notation accommodates interactions between intra-individual factors and inter-individual factors in both their regime-switching model and their state-space model. An estimation procedure called the Forward Filtering Backward Sampling ( 25 , FFBS;) is implemented as a Bayesian counterpart to the extended Kim filter and extended Kim smoother. Accounting for unobserved heterogeneity in the hidden Markov(-switching) models characterizes the novelty of this approach. Limitations and scope These methods have enabled the estimation of diverse nonlinear RSSS models. Notably, they allowed for real-time inferences and forecasts of latent or observed variables based on ongoing collections of ILD. However, there are three limitations in the current state of RSSS models. First, the NDLC-SEM provides a very flexible approach of modeling RSSS models (including intra-individual factors for the prediction of regime membership). It is implemented using a Bayesian estimator which may become computationally very expensive, as it tries to obtain the complete parameter distribution. On the other hand, the goal of the frequentist estimator is to find a point estimate where the likelihood is maximized. Second, existing frequentist approaches do not allow for any forms of intra-individual change, inter-individual differences, and their interaction in transition probabilities. This heavily impacts their use for dynamic modeling of transition probabilities. From an applied perspective, a frequentist estimator that can include dynamic information in updating the state membership is an important next step. Third, the specification of priors is not necessary in the frequentist domain. These models are very complex and theories very often have very vague specification of expected distributions (if at all). Therefore, the lack of theoretical and empirical anticipation of effects (priors/beliefs) could be a motivation to adapt a frequentist approach. In this paper, we present a frequentist estimation approach for RSSS models which considers both intra- and inter-individual differences in hidden Markov(-switching) models. The Kim filter is extended for novelty, enabling parameter and latent variable estimation within a frequentist counter-part to the NDLC-SEM framework. Similar to 16 ’s, our approach accommodates unobserved heterogeneity in the hidden Markov(-switching) models. The remainder of this article is structured as follows. In the next section we introduce the RSSS modeling framework, followed by a detailed description of the model estimation and the implementation of the extended filter. In an empirical example, we illustrate the approach and its ability to forecast regime membership. Then, we provide further evidence of its performance using a simulation study. We end with a discussion of the strengths and weaknesses of the approach. Regime-Switching State-Space (RSSS) models In

Read the complete text