What If TSF : A Benchmark for Reframing Forecasting as Scenario-Guided Multimodal Forecasting
Jinkwan Jang, Hyunbin Jin, Hyungjin Park, Kyubyung Chae, Taesup KimGraduate School of Data Science, Seoul National University {jkjang22, hyunbin.jin, robin5310, kyubyung.chae, taesup.kim}@snu.ac.kr · 2026
Read What If TSF : A Benchmark for Reframing Forecasting as Scenario-Guided Multimodal Forecasting by Jinkwan Jang, Hyunbin Jin, Hyungjin Park, Kyubyung Chae, Taesup KimGraduate School of Data Science, Seoul National University {jkjang22, hyunbin.jin, robin5310, kyubyung.chae, taesup.kim}@snu.ac.kr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08509.txt.
Introduction
Jinkwan Jang and co-authors present What If TSF, a benchmark for scenario-guided multimodal forecasting. Their 2026 arXiv paper notes that time series forecasting drives real decisions, yet most methods stay unimodal. They extrapolate historical patterns without outside context. Language models suggest multimodal forecasting could use text as well, but benchmarks lag behind. Existing ones give retrospective or misaligned raw context, hiding whether models truly use text. The new benchmark reframes forecasting around what-if scenarios, testing if textual guidance changes predictions in the right way.
Catalog record
| CREATOR | Jinkwan Jang, Hyunbin Jin, Hyungjin Park, Kyubyung Chae, Taesup KimGraduate School of Data Science, Seoul National University {jkjang22, hyunbin.jin, robin5310, kyubyung.chae, taesup.kim}@snu.ac.kr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What If TSF : A Benchmark for Reframing Forecasting as Scenario-Guided Multimodal Forecasting Jinkwan Jang, Hyunbin Jin, Hyungjin Park, Kyubyung Chae, Taesup KimGraduate School of Data Science, Seoul National University {jkjang22, hyunbin.jin, robin5310, kyubyung.chae, taesup.kim}@snu.ac.kr Abstract Time series forecasting is critical to real-world decision making, yet most existing approaches remain unimodal and rely on extrapolating historical patterns. While recent progress in large language models (LLMs) highlights the potential for multimodal forecasting, existing benchmarks largely provide retrospective or misaligned raw context, making it unclear whether such models meaningfully leverage textual inputs. In practice, human experts incorporate what-if scenarios with historical evidence, often producing distinct forecasts from the same observations under different scenarios. Inspired by this, we introduce What If TSF (WIT) , a multimodal forecasting benchmark designed to evaluate whether models can condition their forecasts on contextual text, especially future scenarios. By providing expert-crafted plausible or counterfactual scenarios, WIT offers a rigorous testbed for scenario-guided multimodal forecasting. The benchmark is available at https://github.com/jinkwan1115/WhatIfTSF . 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author. 1 Introduction Forecasting plays a critical role across society: businesses estimate consumer demand to guide investment, governments predict economic or energy indicators to shape policy ( Goodwin et al. 2023 ; Coroneo 2025 ) , and fields from climate science ( Kent et al. 2025 ) to epidemiology ( George et al. 2019 ) use forecasting to transform past observations into actionable foresight. Most forecasting methods, whether statistical or learning-based, have conventionally focused on numerical time series alone. Recent time series foundation models (TSFMs) such as Chronos ( Ansari et al. 2024 ; Ansari et al. 2025 ) , TimesFM ( Das et al. 2024 ) , and Moirai ( Woo et al. 2024 ; Liu et al. 2025b ) extend this paradigm by scaling up model size and data coverage, and have gained prominence for their cross-domain zero-shot forecasting capability. Yet, they still primarily extrapolate historical patterns, implicitly assuming that future dynamics will resemble the past. Consequently, these models are limited to pattern matching, failing to leverage the external contexts that human experts integrate into their forecasts. Recent advances in large language models (LLMs) and multimodal alignment offer a promising path toward multimodal forecasting ( Kong et al. 2025b ) . Unlike conventional models confined to time series, LLMs can leverage unstructured text to capture external context that numerical observations alone cannot convey. Representation-fusion and prompting-based strategies ( Jin et al. 2023 ; Liu et al. 2024b ; Requeima et al. 2024 ) have emerged, showing that integrating text with time series can improve multimodal forecasting. Figure 1: Overview of scenario-guided multimodal forecasting with our WIT benchmark. The figure illustrates how textual information about plausible future scenarios and counterfactual scenarios can influence the directional outlook of target future time points, highlighting the role of scenario-guided context in shaping forecasts. However, recent evidence reveals that multimodal methods are highly sensitive to the quality of textual context. A comprehensive study ( Zhang et al. 2025b ) finds that they often fail to surpass strong unimodal baselines. This underperformance is primarily attributed to existing benchmarks where the text largely duplicates information already present in the historical observations. Specifically, retrospective descriptions of past events tend to provide limited predictive cues for future outcomes and can even hinder performance by introducing noise. Furthermore, we find that existing benchmarks often incorporate raw context alongside time series without sufficient preprocessing or verification, leading to empty or uninformative text and occasional temporal misalignment that can even leak future information into the context. These limitations highlight the need for multimodal forecasting benchmarks that go beyond history-bound text and poorly aligned raw context. Rather than relying on information that merely restates past observations, a good benchmark should further provide plausible future scenarios grounded in expert knowledge. Crucially, such future information must be carefully aligned with the forecasting horizon and expressed as conditional possibilities rather than definitive, fully specified scenarios. Without such signals, models may end up exploiting spurious correlations, making it unclear whether they truly leverage external information. Human experts, by contrast, anticipate contingencies by forming what-if scenarios grounded in past evidence and domain knowledge. This expert practice is especially important in high-stakes settings, where uncertainty and potential external shocks are never captured by time series alone. Motivated by this, we present What If TSF (WIT) , a new benchmark that provides plausible scenarios as future guidance, thereby enabling a more faithful evaluation than retrospective text alone. Our main contributions are: • Scenario-guided forecasting evaluation. WIT benchmark provides expert-crafted future scenarios grounded in real-world data, and evaluates whether models can condition forecasts on these scenarios using directional accuracy. • Counterfactual forecasting task for guidance adherence. WIT includes a counterfactual forecasting task to test whether models’ predicted direction flips in response to what-if guidance. • Complete and well-aligned benchmark. WIT ensures non-empty, coherent textual context and reduces noisy or misaligned cases through LLM filtering and expert verification. WIT de-identifies entities while preserving informative cues to mitigate memorization. 2 What If? Time Series Forecasting (WIT) Benchmark Datasets Numerical History ( x_{1:t} ) Static Context ( S ) Dynamic Context ( D_{t} ) Notes Domain/Variable Description TS-Aligned History ( H ) Plausible Future ( F_{\text{pl}} ) Counterfactual Future ( F_{\text{cf}} ) TS-Insights ✓ \blacktriangle \blacktriangle ✗ ✗ redundant with series patterns ChatTS ✓ \blacktriangle \blacktriangle ✗ ✗ redundant with series patterns MoTime ✓ ✓ ✗ ✗ ✗ variable-only MTBench ✓ ✓ \blacktriangle \blacktriangle ✗ raw, inconsistent TSQA ✓ ✓ \blacktriangle \blacktriangle ✗ raw, inconsistent, variable-only Time-MMD ✓ ✓ \blacktriangle \blacktriangle ✗ incomplete, inconsistent, redundant, misaligned CiK ✓ ✓ ✓ \blacktriangle \blacktriangle raw, overly specified futures WIT (ours) ✓ ✓ ✓ ✓ ✓ de-identified, complete, scenario-based futures Table 1: Comparison of time series datasets and benchmarks for multimodal forecasting. For textual context, ✓ : available; \blacktriangle : partially available or redundant with time series; ✗ : not available. ‘variable-only’ indicates cases where only variable descriptions are provided without input time series related contextual information. Figure 2: Comparison with Time-MMD and CiK. WIT resolves the issues in existing datasets and benchmarks. 2.1 Limitations of Existing Multimodal Forecasting Benchmarks Recent progress in multimodal forecasting has led to the development of datasets and benchmarks that combine observed time series with other modalities, with text as the primary modality. In Table 1 , we categorize the context used in prior multimodal forecasting datasets and benchmarks, identifying several common structural limitations, extending prior analyses ( Zhang et al. 2025b ) . Most existing benchmarks remain largely history-bound and are not designed to evaluate conditional forecasting under future information over target horizons. These benchmarks also exhibit limitations that stem from the raw context itself. Redundant context In many benchmarks, textual inputs primarily summarize historical patterns or restate numerical information already in the time series. For example, TS-Insights ( Zhang et al. 2023 ) , and ChatTS ( Xie et al. 2024 ) both contain textual information that largely overlaps with numerical patterns which provide limited signal for anticipating future changes. Some textual descriptions in Time-MMD ( Liu et al. 2024a ) contain similar redundancy. Context limited to static information Variable-only context refers to settings where the text provides only static information about the target variable, such