TimeSeek: Temporal Reliability of Agentic Forecasters
Dennis Lee Automorphic Labs [email protected] Hamza Mostafa Cheriton School of Computer Science University of Waterloo [email protected] Om Shastri The Wharton School University of Pennsylvania [email protected] · 2026
Read TimeSeek: Temporal Reliability of Agentic Forecasters by Dennis Lee Automorphic Labs [email protected] Hamza Mostafa Cheriton School of Computer Science University of Waterloo [email protected] Om Shastri The Wharton School University of Pennsylvania [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04220.txt.
Introduction
Hamza Mostafa, Om Shastri, and Dennis Lee introduced TimeSeek, a benchmark tracking how agentic language-model forecasters hold up across a prediction market's life. They tested 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five time checkpoints, with and without web search, for 15,000 forecasts total. Models competed best early and on high-uncertainty markets, but faded near resolution and on strong-consensus markets. Web search lifted pooled Brier Skill Score for every model overall yet hurt in 12 percent of model-checkpoint pairs, favoring time-aware evaluation over single snapshots.
Catalog record
| CREATOR | Dennis Lee Automorphic Labs [email protected] Hamza Mostafa Cheriton School of Computer Science University of Waterloo [email protected] Om Shastri The Wharton School University of Pennsylvania [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27161 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04220
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TimeSeek: Temporal Reliability of Agentic Forecasters Dennis Lee Automorphic Labs [email protected] Hamza Mostafa Cheriton School of Computer Science University of Waterloo [email protected] Om Shastri The Wharton School University of Pennsylvania [email protected] Abstract We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market’s lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market’s life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting. 1 Introduction When should an autonomous AI agent’s forecast be trusted relative to a prediction market? Existing LLM forecasting benchmarks usually evaluate a single snapshot per question or market ( 31 ; 22 ; 49 ) . That setup obscures an important temporal issue: prediction markets aggregate information continuously, so early prices can reflect sparse participation while late prices often incorporate much richer information. A model that is competitive early but weak late has a very different reliability profile from one that underperforms uniformly. We study this question with TimeSeek, a benchmark built from 150 CFTC-regulated Kalshi binary markets evaluated at five lifecycle checkpoints under two conditions: with and without web search. This yields 15,000 model forecasts across 10 frontier models. Rather than treating “LLMs vs. markets” as a single aggregate comparison, our goal is to characterize when models are relatively competitive, when retrieval helps, and when deferring to the market remains the stronger default. Throughout, we treat these comparisons as descriptive: the paper reports point estimates, while uncertainty quantification for subgroup analyses is left to future work. Our main contributions are: 1. Time-aware evaluation of agentic forecasters. We introduce a temporal benchmark and with/without-search ablation that measures model performance at five points in each market’s lifecycle rather than at a single snapshot. 2. Descriptive evidence of temporal and difficulty-dependent performance. Models are most competitive early in a market’s life and on high-uncertainty markets, but substantially less competitive late in the lifecycle and on strong-consensus markets. 3. Evidence for selective rather than uniform tool use. Web search improves pooled BSS for all 10 models overall, but degrades performance in 12% of model–checkpoint conditions and shows strong category-specific heterogeneity. 4. A simple ensemble baseline. Model–market error correlations are modest ( r=0.19 – 0.41 ), and simple two-model averaging reduces loss relative to the best single model, though no pair surpasses the market overall. 2 Related Work LLM forecasting and prediction benchmarks. KalshiBench ( 31 ) evaluates LLMs on thousands of Kalshi markets at fixed snapshots and highlights systematic miscalibration. ForecastBench ( 22 ) and Prophet Arena ( 48 ) move toward dynamic evaluation on live markets, while Mirai ( 49 ) emphasizes event-driven time sensitivity. TruthTensor ( 41 ) evaluates LLMs under market drift with emphasis on human imitation, but does not control for tool access or analyze within-market lifecycle dynamics. Our work complements this literature with a controlled tool-use ablation across temporal checkpoints and difficulty regimes. Adjacent work studies forecasting-specific reliability via consistency checks ( 34 ) and uncertainty-aware training ( 11 ) . Agentic AI for financial decision-making. Beyond forecasting, benchmarks increasingly measure end-to-end agent performance in market interaction, including multi-market trading ( 38 ; 50 ) and financial decision-making ( 26 ) . These stress execution and feedback loops; we focus on the upstream question of forecast reliability—whether an agent’s probabilistic prediction should be trusted before any trading decision is made. Tool use and temporal drift. Tool access can improve task performance ( 39 ) , but tool-augmented workflows can distort confidence ( 46 ) and deeper reasoning can degrade calibration ( 25 ) . Broader concerns about temporal drift in LLM behavior motivate evaluation protocols that are explicitly time-aware ( 23 ; 51 ) . Our temporal checkpoint design directly tests these stability and reliability issues in a high-stakes forecasting setting. Prediction markets as baselines. Prediction markets are generally well-calibrated across many domains ( 43 ; 3 ) , and crowd forecasting systems span markets, polls, and elite forecasters ( 4 ) . We use contemporaneous market prices as the baseline, consistent with the view that market-implied probabilities are strong default forecasts ( 43 ; 15 ) . 3 Method 3.1 Dataset: CFTC-Regulated Binary Markets We collect 150 binary markets from Kalshi, a CFTC-regulated exchange where real capital is at risk. Each market resolves to YES/NO based on objective criteria (election results, economic releases, sports scores). We enforce $5,000+ trading volume per market and require post-October 2025 resolution to ensure temporal separation from model training cutoffs. Concretely, we set the cutoff date to the latest publicly stated knowledge cutoff among the models we evaluate (October 2025), and keep only markets whose resolution occurs after that date to minimize outcome contamination (a KalshiBench-style filtering rule) ( 31 ) . The dataset spans five categories (30 markets each): Politics, Sports, Macro-Economics, Science/Tech, and Financial markets. Table 1 provides statistics. Table 1: TimeSeek dataset characteristics. Characteristic Value Markets / Checkpoints / Predictions 150 / 5 / 750 per condition Categories 5 (30 markets each) Timeline Oct 2025 – Jan 2026 Trading volume $5.6k – $41.3M (med. $35k) Duration (days) 2 – 337 (med. 89) Outcomes 73% NO, 27% YES 3.2 Temporal Sampling Protocol For each market, we define lifecycle progress p(t)=\frac{t-t_{\mathrm{open}}}{t_{\mathrm{close}}-t_{\mathrm{open}}} and sample at 5 checkpoints: Open+1 ( p=0.01 ), 25%, 50%, 75%, and Close-1 ( p=0.99 ). For each sample, we record sample date, market price, and ground truth outcome, yielding 750 samples per model per condition (with/without search), totaling 15,000 predictions. 3.3 Agentic Forecasting Architecture We implement agentic forecasters as a 4-node state machine: research (initialize context) \rightarrow agent (decide to search or predict) \rightarrow tools (execute searches) \rightarrow forecast (extract structured prediction). The agent loops between agent and tools until it produces a final forecast. When web search is enabled (agentic condition), agents access web search with date filtering (results restricted to before the sample date) to prevent future information leakage. When disabled (baseline condition), the model receives the identical prompt but cannot invoke any tools, isolating parametric knowledge. Market prices are withheld during prediction to prevent anchoring. 3.4 Models Evaluated We evaluate 10 frontier LLMs: Claude Opus 4.5 ( 1 ) , GPT-5.2 ( 32 ) , Gemini 3 Pro ( 19 ) , Grok 4.1-fast ( 44 ) , Kimi-k2 ( 45 ) , Kimi-k2.5 ( 29 ) , DeepSeek v3.2 ( 14 ) , Intellect-3 ( 37 ) , Trinity Large ( 2 ) , and Qwen3-235B ( 47 ) . Each model predicts on 150 markets \times 5 checkpoints =750 samples per condition, yielding 750\times 10 models \times 2 conditions =15{,}000 total predictions. 3.5 Evaluation Metrics Brier Score (BS): \mathrm{BS}=\frac{1}{N}\sum_{i=1}^{N}(\hat{p}_{i}-y_{i})^{2}, where \hat{p}_{i} is predicted probability and y_{i}\in\{0,1\} ( 6 ) . Brier Skill Score