LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting

Sungjun Cho, Changho Shin, Suenggwan Jo · 2025

Read LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting by Sungjun Cho, Changho Shin, Suenggwan Jo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20952.txt.

Introduction

Sungjun Cho leads five co-authors on LBS, a forecasting framework that joins Bayesian state space models with a large language model so numbers and text can be predicted together. Real forecasts need both structured series and unstructured reports, yet older methods fix horizons in advance and give no uncertainty.

A state space backbone holds latent states that generate both numeric and textual observations, while the language model is adapted to encode text input into that backbone. The probabilistic design yields uncertainty bands alongside point forecasts. Flexible horizons let the same system answer varied planning questions in settings like finance and operations.

Catalog record

CREATORSungjun Cho, Changho Shin, Suenggwan Jo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting Sungjun Cho, Changho Shin, Suenggwan Jo Abstract Forecasting in the real world requires integrating structured time-series data with unstructured textual information, but existing methods are architecturally limited by fixed input/output horizons and are unable to model or quantify uncertainty. We address this challenge by introducing LLM-integrated Bayesian State space models (LBS), a novel probabilistic framework for multimodal temporal forecasting. At a high level, LBS consists of two components: (1) a state space model (SSM) backbone that captures the temporal dynamics of latent states from which both numerical and textual observations are generated and (2) a pretrained large language model (LLM) that is adapted to encode textual inputs for posterior state estimation and decode textual forecasts consistent with the latent trajectory. This design enables flexible lookback and forecast windows, principled uncertainty quantification, and improved temporal generalization thanks to the well-suited inductive bias of SSMs toward modeling dynamical systems. Experiments on the TextTimeCorpus benchmark demonstrate that LBS improves the previous state-of-the-art by 13.20% while providing human-readable summaries of each forecast. Our work is the first to unify LLMs and SSMs for joint numerical and textual prediction, offering a novel foundation for multimodal temporal reasoning . 1 Introduction Time-series forecasting, a core machine learning task, is traditionally centered on predicting future numerical values from past data (patchtst)nie2022time . However, in many real-world domains, contextual information expressed in natural language—such as clinical notes, financial reports, or weather descriptions—plays a critical role in forecasting. This complementary modality can offer valuable signals that cannot be fully extracted from numeric data alone liu2024time ; kim2024multi . Similarly, generating textual forecasts alongside numerical predictions can improve interpretability. This is particularly useful in high-stakes decision-making scenarios. These opportunities motivate the development of models that not only forecast from multimodal inputs, but also communicate their predictions through natural language, augmenting quantitative accuracy with qualitative explanations. Probabilistic state space models (SSMs) offer compelling advantages for time-series forecasting: they can quantify uncertainty and support flexible, variable-length prediction horizons. For this reason, it is intuitive to seek to integrate a probabilistic SSM with a pretrained large language model (LLM), enabling forecasting from and into both numeric and textual modalities. However, this integration presents two fundamental challenges: (C1) Text-conditioned posterior state estimation: How can we update the latent state of the SSM using a pretrained LLM and textual observations? (C2) Latent state-conditioned text generation: How can we adapt the LLM to generate accurate, temporally grounded textual forecasts conditioned on latent state trajectories? Figure 1 : An illustration of the LBS architecture under for a single-step forecasting scenario. Bottom: To enable variational inference with text, the LLM is tuned to summarize the context into a set of summarization tokens, which are then used together with the target value to obtain the posterior state distribution. Top: Conditioned on the state forecast, the shared LLM is trained to generate its corresponding text. When using instruction-tuned LLMs, both steps are accompanied by prompt templates in order to preserve its capabilities. To address these challenges, we propose the LLM-integrated Bayesian State Space Model (LBS) , a novel architecture that unifies a probabilistic SSM with a pretrained LLM for joint numeric and textual forecasting (see Figure 1 ). For (C1) , we adapt the LLM to summarize and compress textual inputs into a sequence of summary tokens, which are then together projected into the low-dimensional latent state space for deep Bayesian filtering. For (C2) , we leverage the LLM’s in-context generation capabilities by conditioning it on latent state trajectories—treated as non-textual context, akin to images or videos—enabling the model to generate temporally coherent, state-grounded textual forecasts. To our knowledge, LBS is the first framework to tightly couple probabilistic SSMs and LLMs to generate temporally coherent forecasts across both numeric and textual modalities. We evaluate LBS on the TimeTextCorpus (TTC) , a multimodal benchmark spanning climate and clinical domains. Our model consistently outperforms both unimodal and multimodal baselines, improving numeric forecasting accuracy by an average of 13.20% while generating coherent and contextually relevant textual predictions. Qualitative analyses show further benefits of the Bayesian SSM backbone, including its ability to encode trends and seasonality in low-dimensional latent states and provide principled uncertainty estimates . Finally, we explore LLM scaling by substituting various models from the Qwen2.5 yang2024qwen2 and LLaMA3 grattafiori2024llama families, finding that larger LLMs do not always improve performance, but incorporating textual context enhances robustness especially for long-horizon forecasts . The key contributions of our work are the following: • We present LBS, a novel architecture for multimodal time-series forecasting that integrates a Bayesian state space model with pretrained LLMs to enable conditioning posterior estimates on textual inputs as well as text generation. • We incorporate techniques from the LLM-based context compression and multimodal instruction tuning literature towards enabling LBS to perform posterior state estimation conditioned on text and state-conditioned text generation. • We verify the empirical performance of LBS on the multimodal forecasting benchmark, TimeTextCorpus (TTC) , achieving excellent performance on climate and medical domains. 2 Related Work We briefly review work related to LBS, specifically those in multimodal time-series forecasting and Bayesian state-space models. An extended discussion on related work can be found in Appendix A . Multimodal Time-Series Forecasting. Motivated by recent advancements in LLMs, recent studies have explored their potential to enhance time-series forecasting by incorporating textual context. Some approaches use LLMs directly merrill2024language ; gruver2023large , while others finetune them with auxiliary modules. xu2024beyond introduce a cross-attention architecture to fuse news and time-series data. kim2024multi propose HybridMMF, a joint model trained on the TimeTextCorpus using shared embeddings for text and time series. wang2024chattime develop ChatTime, a foundation model fine-tuned for diverse zero-shot forecasting tasks. li2025language present TaTS, which augments standard models with concurrent textual inputs as auxiliary variables. These works do not explore the integration of LLMs into state space models, which is the key contribution of our work . By incorporating LLMs into state space models, we not only improve time-series forecasting performance but also exploit the benefits of state space models, such as uncertainty estimation. Bayesian State Space Models. Bayesian filtering has long been central to time-series modeling, with the classical Kalman filter kalman1960new offering optimal estimation in linear Gaussian systems. To address nonlinear and non-Gaussian dynamics, extensions such as the Extended Kalman Filter, Unscented Kalman Filter julier2004unscented , Ensemble Kalman Filter houtekamer1998data , and Sequential Monte Carlo doucet2001sequential have been proposed to approximate posterior distributions using linearization or sampling techniques. More recent work combines state space structure with deep neural networks. KVAE (kvae)fraccaro2017disentangled and rangapuram2018deep apply neural parameterizations to learn latent dynamics for sequential and multivariate time-series forecasting. In reinforcement learning, models like PlaNet hafner2019learning , Dreamer hafner2020mastering , and KalmanNet revach2022kalmannet extend these ideas to partially observable, high-dimensional control tasks. Despite this progress, most prior work focuses on unimodal data such as images or numerical sequences. Our work is the first to integrate pretrained large language models (LLMs) with a probabilistic state space model for joint numeric and textual forecasting , extending Bayesian state estimation into the realm of multimodal generative modeling. 3 Preliminaries We provide background information on multimodal time-series forecasting (§ 3.1 ), followed by a discussion on the latent dynamical

Read the complete text