Benchmark Datasets for Lead-Lag Forecasting on Social Platforms
Kimia Kazemian 1 , Zhenzhen Liu ∗ 1 , Yangfanyu Yang 2 , Katie Z Luo 1 , Shuhan Gu 1 , Audrey Du 1 , Xinyu Yang 2 , Jack Jansons 1 , Kilian Q Weinberger 1 , John Thickstun 1 , Yian Yin 2 , Sarah Dean 1 1 Department of Computer Science and 2 Department of Information Science at Cornell University · 2025
Read Benchmark Datasets for Lead-Lag Forecasting on Social Platforms by Kimia Kazemian 1 , Zhenzhen Liu ∗ 1 , Yangfanyu Yang 2 , Katie Z Luo 1 , Shuhan Gu 1 , Audrey Du 1 , Xinyu Yang 2 , Jack Jansons 1 , Kilian Q Weinberger 1 , John Thickstun 1 , Yian Yin 2 , Sarah Dean 1 1 Department of Computer Science and 2 Department of Information Science at Cornell University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03877.txt.
Introduction
Kimia Kazemian is the lead author among twelve authors introducing lead-lag forecasting as a named problem and releasing two large benchmark datasets for it. On social and collaborative platforms, early signals such as views, likes, pushes, or stars often precede later outcomes such as citations, sales, reviews, or forks by months or years.
One dataset links arXiv accesses to citations for about two point three million papers; the other links GitHub activity to forks for about three million repositories. Curation details are documented, lead-lag structure is verified statistically, and parametric and non-parametric baselines are benchmarked. The release gives forecasting researchers a shared testbed that avoids survivorship bias.
Catalog record
| CREATOR | Kimia Kazemian 1 , Zhenzhen Liu ∗ 1 , Yangfanyu Yang 2 , Katie Z Luo 1 , Shuhan Gu 1 , Audrey Du 1 , Xinyu Yang 2 , Jack Jansons 1 , Kilian Q Weinberger 1 , John Thickstun 1 , Yian Yin 2 , Sarah Dean 1 1 Department of Computer Science and 2 Department of Information Science at Cornell University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-03877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmark Datasets for Lead-Lag Forecasting on Social Platforms Kimia Kazemian 1 , Zhenzhen Liu ∗ 1 , Yangfanyu Yang 2 , Katie Z Luo 1 , Shuhan Gu 1 , Audrey Du 1 , Xinyu Yang 2 , Jack Jansons 1 , Kilian Q Weinberger 1 , John Thickstun 1 , Yian Yin 2 , Sarah Dean 1 1 Department of Computer Science and 2 Department of Information Science at Cornell University Abstract Social and collaborative platforms emit multivariate time-series traces in which early interactions—such as views, likes, or downloads—are followed, sometimes months or years later, by higher impact like citations, sales, or reviews. We formalize this setting as Lead-Lag Forecasting (LLF): given an early usage channel (the lead), predict a correlated but temporally shifted outcome channel (the lag). Despite the ubiquity of such patterns, LLF has not been treated as a unified forecasting problem within the time-series community, largely due to the absence of standardized datasets. To anchor research in LLF, here we present two high-volume benchmark datasets—arXiv (accesses → citations of 2.3M papers) and GitHub (pushes/stars → forks of 3M repositories)—and outline additional domains with analogous lead–lag dynamics, including Wikipedia (page-views → edits), Spotify (streams → concert attendance), e-commerce (click-throughs → purchases), and LinkedIn profile (views → messages). Our datasets provide ideal testbeds for lead–lag forecasting, by capturing long-horizon dynamics across years, spanning the full spectrum of outcomes, and avoiding survivorship bias in sampling. We documented all technical details of data curation and cleaning, verified the presence of lead-lag dynamics through statistical and classification tests, and benchmarked parametric and non-parametric baselines for regression. Our study establishes LLF as a novel forecasting paradigm and lays an empirical foundation for its systematic exploration in social and usage data. Our data portal with downloads and documentation is available at https://lead-lag-forecasting.github.io/. 1 Introduction The success of human activities is often measured by their collective impact, ranging from music streams and movie box office revenues to product sales and social media popularity. These impact metrics typically follow heavy-tailed distributions ( 5 ) and slow decay patterns across timescales ( 3 ) , making early identification of future hits fundamentally challenging ( 4 ; 15 ) . At the same time, digital platforms increasingly log online user interactions—searches, views, downloads, likes, and shares—that often precede these long-term dynamics. These temporal lead-lag dynamics are remarkably ubiquitous, spanning domains as diverse as science ( 9 ) , economics ( 28 ) , arts ( 6 ) , culture ( 8 ) , and social movements ( 12 ) . A systematic understanding of such lead-lag dynamics is not only crucial for anticipating and optimizing impact in digital ecosystems, but also essential for designing effective strategies that identify and promote emerging innovations and products. In this paper, we introduce Lead-Lag Forecasting (LLF) defined as follows: given a “lead” usage channel, predict a correlated but temporally shifted “lag” outcome channel. LLF aims to build upon the framework of early-signal prediction and further clarify evaluation goals that differ from generic long-horizon forecasting—it addresses a complementary regime to traditional forecasting where the predictor must transfer information across channels (e.g., access → citations) while generalizing to new series. LLF problems exhibit three key “dual process” properties: (i) observable early-phase interactions —e.g., views, likes, github repository pushes, (ii) meaningful outcome signal —e.g., paper citations, purchases, github repository forks, and (iii) predictive dependency —the early dynamics often cause or forecast later outcomes through complex, non-deterministic mechanisms. Despite the ubiquity of such dual process dynamics across digital ecosystems, our quantitative understanding of their predictability remains very limited, largely due to the lack of standardized datasets for systematic LLF research. Though frequently recorded in internal web logs, early “lead” signals are not often made publicly available. Popular forecasting benchmarks—traffic ( 2 ) , electricity load ( 25 ) , taxi demand ( 21 ) , ETTh ( 33 ) , M4 ( 10 ) , and related corpora—have spurred substantial progress on short-horizon and seasonal forecasting within a single measurement channel. While these benchmarks serve their intended applications well, they focus primarily on same-channel prediction rather than cross-channel dynamics with series generalization. To address this gap, here we build and release 1 1 1 Dataset portal: https://lead-lag-forecasting.github.io/ For the responsible usage terms rationale, see Appendix A . two high-volume datasets in science (accesses and citations of about 2.3M arXiv papers) and technology (pushes, stars, and forks of about 3 million GitHub repositories). These two examples represent ideal testbeds for LLF research: (1) The broad impact of papers and software often extends beyond their original communities, making them significant, real-world settings rather than artificial test cases ( 30 ) . (2) Both systems have observable effects that unfold over years, presenting complex real-world challenges that traditional methods struggle to model ( 13 ; 11 ) . (3) As our records include both the high volume of low-impact items and the small number of high-impact breakthroughs, the datasets provide unbiased coverage across the entire impact spectrum without survivorship bias. To summarize, our contributions are the following: (1) we provide two datasets, one of which has never before been publicly available, and document technical details of data curation and cleaning; (2) we analyze the lead-lag dynamics in both datasets and demonstrate predictive signals in the paired channels; (3) we benchmark deep learning, parametric, and non-parametric prediction baselines. Together, this study establishes LLF as a novel forecasting paradigm and lays an empirical foundation for its systematic exploration in social data. 2 Related Work Time Series Prediction Tasks. Time series prediction is a longstanding and active research area in machine learning. Two prominent settings for this modality include rolling forecasting ( 33 ; 27 ; 31 ) and temporal point process estimation ( 17 ; 22 ; 34 ; 29 ) . Rolling forecasting is commonly used for applications like weather forecasting or load monitoring. In this setting, observations are recorded at regular time intervals. At each time step t , one uses a lookback window of the past L observations \{\mathbf{x}_{t-L},\cdots\mathbf{x}_{t-1}\} to predict the next H future steps \{\mathbf{x}_{t},\cdots\mathbf{x}_{t+H-1}\} ( 33 ) . Temporal point processes are often used for tasks like behavior modeling and critical event forecasting. In this setting, one is interested in modeling discrete events that occur at potentially irregular intervals in continuous time. Given past events \{(t_{1},c_{1}),\cdots(t_{n-1},c_{n-1})\} , where t_{i} denotes the time and c_{i} the type of event i , the goal is to predict the arrival time and type of the next event ( 29 ) . Our new arXiv and GitHub datasets are relevant for such settings: the dense lead channels can be used for rolling forecasting, while the sparse lag channels can be reframed as temporal point processes. Beyond this contribution to the broader time-series community, though, our proposed lead-lag forecasting task differs from both settings. Unlike rolling forecasting, which often predicts near-future outcomes immediately following the context window, we introduce a gap between the lead window and the prediction horizon. Unlike temporal point processes, the lead channels in lead-lag prediction can consist of dense, regularly sampled measurements available at each time step. Our datasets introduce exciting challenges for time series prediction: models must generalize to unseen series, transfer information across