From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space
Lehui Li · 2026
Read From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space by Lehui Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12664.txt.
Introduction
Lehui Li and co-authors bridge text and numbers for time-series forecasting with event-driven shifts. Text states impacts in vague, qualitative terms, while forecast models need explicit signals. Tests showed older methods over-attend to redundant tokens and fail to turn meaning into cues. Their TESS builds a middle space from four temporal primitives: lag, shape, volatility, and mean shift. A language model extracts them with confidence gating. Across four real datasets, error fell by up to 29 percent.
Catalog record
| CREATOR | Lehui Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51037 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12664
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space Lehui Li Abstract Abstract. Incorporating textual information into time-series forecasting holds promise for addressing event-driven non-stationarity; however, a fundamental modality gap hinders effective fusion: textual descriptions express temporal impacts implicitly and qualitatively, whereas forecasting models rely on explicit and quantitative signals. Through controlled semi-synthetic experiments, we show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues. To bridge this gap, we propose TESS , which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities. This space consists of interpretable, numerically grounded temporal primitives—mean shift, volatility, shape, and lag—extracted from text by an LLM via structured prompting and filtered through confidence-aware gating. Experiments on four real-world datasets demonstrate up to a 29% reduction in forecasting error compared to state-of-the-art uni-modal and multimodal baselines. The code will be released after acceptance. 1 Introduction Figure 1 : Illustration of the cross-modal transformation via the Temporal Semantic Space. Verbose textual narratives (left) are distilled into structured temporal primitives (middle), which then guide numerical forecasting (right). Time-series forecasting plays a critical role in diverse domains such as transportation, energy, and finance ( 38 ; 28 ) . In recent years, unimodal forecasting models that rely solely on historical numerical observations have achieved substantial progress by modeling temporal dependencies ( 33 ) . However, real-world time series often exhibit significant non-stationarity ( 7 ; 11 ) , where the underlying data-generating mechanism evolves over time. In particular, exogenous events such as accidents, extreme weather, or public sentiment shocks can trigger regime shifts ( 22 ; 35 ) , causing statistical properties—including trends, volatility, mean, and variance—to change abruptly within short time windows ( 1 ) . Under such conditions, models struggle to extract reliable predictive signals from historical numerical observations alone, leading to severe performance degradation ( 37 ) . To mitigate event-driven non-stationarity, recent work has begun to incorporate textual data (e.g. news articles, social media posts) ( 24 ) , as an exogenous information source. The typical approach leverages pretrained language models to encode text into semantic embeddings ( 21 ) , which are then fused with numerical time-series features via concatenation, gating, or cross-modal attention ( 24 ; 14 ) . These multimodal methods have demonstrated notable gains on benchmarks with significant non-stationarity. Despite recent progress, the substantial modality gap ( 15 ) between time series data and textual data remains largely overlooked ( 14 ) , fundamentally limiting multimodal forecasting. Time series data is chronologically ordered and quantitative, offering precise measurements of temporal dynamics, yet it lacks explicit semantic abstraction. In contrast, textual data is semantically rich but unstructured and qualitative, with the impacts of events on temporal dynamics often implicit, diffuse, and weakly grounded in time. As a result, the predictive relevance of textual information is sparsely distributed across tokens and rarely aligned with the compact numerical structure required by time-series models. Through controlled semi-synthetic experiments, we analyze time-series predictors that directly fuse numerical observations with raw textual embeddings. We find that: (1) predictors fail to focus on prediction-relevant tokens, instead being distracted by noisy and redundant textual content; (2) even after removing redundant information, models still struggle to correctly decode temporal evolution signals, indicating a fundamental representational mismatch between the two modalities. To bridge this modality gap, As illustrated in Figure 1 , we propose to construct an intermediate representation between the textual space and the numerical series space, termed the Temporal Evolution Semantic Space . This space serves as an information bottleneck that explicitly filters, structures, and organizes textual information relevant to temporal evolution. Inspired by expert-driven paradigms in temporal pattern analysis, we predefine a set of temporal evolution primitives corresponding to critical attributes governing temporal dynamics, such as mean shift, volatility shift, evolution shape, and lag. Crucially, these primitives are expressed as verbal specifications, enabling large language models (LLMs) to emulate human-like judgments when interpreting textual descriptions of events. We decompose the fusion process into two stages: 1 Text \to Semantic Space : We prompt an LLM to reason over the input text and explicitly label latent temporal evolution signals using the predefined primitives. Since such labels may be unreliable due to information insufficiency or limitations of the LLM, we introduce a learnable gating network to estimate and weight the reliability of the generated semantic labels. 2 Semantic Space \to Numerical Space : The inferred temporal evolution primitives are then injected as exogenous conditions into the time-series model, allowing it to leverage its numerical modeling and pattern recognition capabilities to ground semantic signals in observed temporal dynamics. Our contributions are summarized as follows: • Through semi-synthetic experiments, we identify two bottlenecks in text-time-series fusion: attention is distracted by redundant context, and qualitative expressions resist decoding into predictive gains. • We propose TESS , an intermediate space that distills text into temporal primitives via LLM extraction with confidence-aware gating, guiding forecasters as exogenous conditions. • Experiments on four datasets show up to 29% MSE reduction; analyses validate primitive complementarity and gating effectiveness across non-stationary patterns. 2 Related Work Uni-Modal Time-Series Forecasting. Early time series forecasting relies on classical statistical methods that extrapolate trends under stationarity assumptions 3 ; 8 ; 10 , performing well on smaller, simpler datasets. Deep learning models 28 ; 5 ; 39 have since enabled end-to-end learning of nonlinear patterns and temporal dependencies in more complex settings. Nevertheless, these methods remain series-centric and inadequately account for external drivers and task context 12 ; 30 . Multimodal Time-Series Forecasting. Multimodal time series forecasting integrates external text (e.g., news and announcements) 16 ; 6 with historical numerical sequences to better capture distribution drift and structural breaks in event-driven scenarios 30 . Early approaches relied on shallow semantics such as sentiment scores or keyword, which often fail to reflect finer-grained temporal variations like mean shifts and volatility clustering. Recent advances in pretrained language models 27 have motivated direct text-series fusion via cross-attention 26 ; 16 or LLMs 30 ; 2 . However, such methods suffer from narrative redundancy and the difficulty of translating implicit, non-quantitative expressions into stable predictive signals, leading to optimization instability and performance degradation. 3 Preliminaries 3.1 Problem Formulation We consider the multimodal time-series forecasting task. Let \mathbf{x}_{t}\in\mathbb{R}^{d} denote the numerical observation at time step t , and let s_{t} represent the temporally aligned textual information (e.g., news articles, announcements). Given a historical window of length L , the numerical input is defined as \mathbf{X}_{\text{time}}=[\mathbf{x}_{t-L+1},\ldots,\mathbf{x}_{t}]\in\mathbb{R}^{L\times d}, (1) and the textual input is \mathbf{X}_{\text{text}}=s_{t} . The objective is to predict the numerical sequence over the next H steps: \hat{\mathbf{Y}}_{t}=[\hat{\mathbf{x}}_{t+1},\ldots,\hat{\mathbf{x}}_{t+H}]=f_{\theta}(\mathbf{X}_{\text{time}},\mathbf{X}_{\text{text}}), (2) where f_{\theta} denotes a multimodal forecasting model. The prevalent fusion paradigm employs modality-specific encoders each modality into latent representations \mathbf{H}_{\text{time}}=\phi_{\text{time}}(\mathbf{X}_{\text{time}})\in\mathbb{R}^{N\times d_{1}} and \mathbf{H}_{\text{text}}=\phi_{\text{text}}(\mathbf{X}_{\text{text}})\in\mathbb{R}^{M\times d_{2}} , where M is the number of text tokens, which are integrated via cross-modal attention or other fusion mechanisms. 3.2 Diagnosing the Modality Gap Semi-Synthetic Benchmark Construction Figure 2 : Analysis of attention misalignment. Left : Distribution of focus ratio R_{t} on test samples. Right : Relationship between redundant token count and predictive performance. Figure 3 : Comparison of three input variants. Left : Prediction performance (MSE) across