ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers
Gavin Hull, Alex Bihlo · 2025
Read ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers by Gavin Hull, Alex Bihlo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08941.txt.
Introduction
Gavin Hull and Alex Bihlo teach language models to forecast how often a paper will be cited, in a 2025 study. Their framework, ForeCite, bolts a simple linear head onto a pre-trained model so it predicts a paper's average monthly citation rate. The training set holds over 900,000 biomedical papers from 2000 to 2024.
ForeCite reached a correlation of 0.826 with real citation rates, beating the prior best by twenty-seven points. Gains held as models and data grew, and time-based holdouts stayed strong. Saliency maps warn that the model leans hard on titles and abstracts. Funders and editors could use such forecasts to spot rising work early.
Catalog record
| CREATOR | Gavin Hull, Alex Bihlo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08941
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers Gavin Hull, Alex Bihlo Abstract Predicting the future citation rates of academic papers is an important step toward the automation of research evaluation and the acceleration of scientific progress. We present ForeCite , a simple but powerful framework to append pre-trained causal language models with a linear head for average monthly citation rate prediction. Adapting transformers for regression tasks, ForeCite achieves a test correlation of ρ = 0.826 \rho=0.826 on a curated dataset of 900K+ biomedical papers published between 2000 and 2024, a 27-point improvement over the previous state-of-the-art. Comprehensive scaling-law analysis reveals consistent gains across model sizes and data volumes, while temporal holdout experiments confirm practical robustness. Gradient-based saliency heatmaps suggest a potentially undue reliance on titles and abstract texts. These results establish a new state-of-the-art in forecasting the long-term influence of academic research and lay the groundwork for the automated, high-fidelity evaluation of scientific contributions. 1 1 1 Relevant code will be made available upon publication. 1 Introduction This paper explores the potential of pre-trained language models to predict future citation rates directly from the textual content of scientific manuscripts. Though citation rates serve only as a weak proxy for true scholarly impact, they offer a tractable signal through which to examine how contemporary models might begin to interpret and anticipate patterns of scientific attention at scale. By framing citation prediction as a text-to-signal learning problem, this work establishes a foundation for more context-aware, adaptive systems that could assist in analyzing—and eventually guiding—the evolving landscape of scientific research. Traditional citation-prediction methods depend heavily on external metadata—author profiles, citation graphs, or peer-review excerpts—to anticipate future citations ( Pobiedina and Ichise, 2016 ; Davletov et al., 2014 ; Li et al., 2019 ) . While these features can yield reasonable forecasts in narrow domains, they inherently overlook the semantic information and methodological quality embedded in the manuscript itself. In contrast, modern language models have been shown to capture the deep contextual and conceptual relationships in text ( Brown et al., 2020 ) , although their application to citation forecasting has largely been limited to small text fragments or as embeddings for separate downstream learners ( Hirako et al., 2024 ; Hirako et al., 2023 ; van Dongen et al., 2020 ; Wenniger et al., 2023 ) . To bridge this gap, we introduce ForeCite, a plug-and-play framework for adapting pre-trained causal transformers for regression tasks, enabling end-to-end prediction of average monthly citation rates. We evaluate ForeCite on a curated corpus of 900K+ medical articles publish between 2000 and 2024, demonstrating superior predictive performance over state-of-the-art baselines. Our extensive suite of experiments (including scaling-law analysis across model sizes and data volumes, temporal holdout studies to probe concept drift, gradient-based saliency visualizations for interpretability, and targeted ablations) evidence the potential of large language models (LLMs) in scientific impact forecasting. We make four main contributions: 1. Framework for regression. A simple general method for adapting pre-trained causal language models for regression tasks while preserving semantic knowledge. 2. New state-of-the-art. A Spearman rank correlation of \rho=0.826 between predicted and true citation rates, marking a 27-point improvement over the previous state-of-the-art. 3. Comprehensive testing. In-depth analysis of scaling behavior, temporal robustness, interpretability, and targeted ablations. 4. Open science commitment. Public release of code and model checkpoints of various sizes to spur further advancements in this field. 2 Related work Citation prediction. Early work framed citation forecasting as a classification or simple regression problem, using handcrafted features derived from author and venue metadata ( Ibáñez et al., 2009 ; Chakraborty et al., 2014 ; Pobiedina and Ichise, 2016 ) . Alternative graph-based methods exploited network structure, both temporal and topological, to extend forecasting horizons ( Davletov et al., 2014 ) . Some efforts also employed manual mathematical modeling to capture citation dynamics more explicitly ( Wang et al., 2013 ) . More recent neural models incorporate textual and early-citation signals via multilayer perceptrons or recurrent neural networks, yielding improved but still limited performance ( Ruan et al., 2020 ; Jamal et al., 2024 ) . The latest transformer-based predictors leverage pre-trained language embeddings over abstracts or document chunks, but typically only use them as fixed features for downstream regressors rather than in an end-to-end predictive framework ( Hirako et al., 2024 ; van Dongen et al., 2020 ; Hirako et al., 2023 ; Wenniger et al., 2023 ) . A more in-depth analysis of techniques used in the field can be found in Aiza et al. (2024) . Large language models. The transformer architecture, built solely on self-attention and forgoing recurrence, has become a foundational tool in natural language processing (NLP), demonstrating superior performance and parallelizability across tasks ( Vaswani et al., 2023 ) . Scaling transformers to billions of parameters has yielded emergent capabilities, including few-shot learning, that have redefined model generality and adaptability ( Brown et al., 2020 ) . Despite these advances, fine-tuning causal transformers for regression tasks such as citation rate prediction remains largely underexplored in the literature. Scaling-laws. Empirical studies reveal that language models’ performance follows predictable power-law relationships with respect to both model size and training data volume ( Kaplan et al., 2020 ) . While these scaling-laws extend to many downstream tasks, the precise improvement achievable through fine-tuning is task dependent. Our work is the first to systematically characterize these trends for citation rate prediction. Interpretability. Understanding which textual elements drive model predictions is critical for transparency and interpretability. Interpretability techniques for transformer-based architectures fall into several categories. Attention-based visualization remains a popular means of attributing model decisions, highlighting tokens which receive high self-attention scores ( Hao et al., 2021 ) . Gradient-based saliency methods, originally developed for vision models and later adapted for NLP, assign importance scores to input tokens via back-propagated gradients ( Li et al., 2016 ) . We apply these techniques here to highlight the impact of individual tokens on predicted citation rates. 3 Methods 3.1 Data collection & preprocessing We constructed our dataset by querying the Elsevier API for seven biomedical keywords across every month from January 2000 to December 2024 with a maximum of 6K resulting articles per query. After de-duplication, our dataset underwent rigorous filtering of corrupted entries, non-English and restricted-access items, and outliers via several criteria (e.g. article length, completeness, etc.). This process yielded a high-quality corpus comprising 900K+ articles (see Figure 1 ), each annotated with its publication date and total number of citations as of December 2024. cardiovascular respiratory renal pulmonary neurology oncology hepatic 0 1 2 3 \cdot 10^{5} 3.36\cdot 10^{5} 3.33\cdot 10^{5} 3.1\cdot 10^{5} 2.94\cdot 10^{5} 2.17\cdot 10^{5} 1.95\cdot 10^{5} 1.8\cdot 10^{5} Number of articles Keyword counts Figure 1 : Distribution of articles in the final dataset containing each domain-specific keyword. Aligning with standard practice in citation prediction ( Hirako et al., 2024 ; van Dongen et al., 2020 ; Hirako et al., 2023 ; Wenniger et al., 2023 ) , we applied a log-transform to the average monthly citation rates. This produced a near-Gaussian distribution (Figure 2 ), motivating the standardization of our data prior to training. The mean and standard deviation were calculated using the training split to prevent leakage. Given the