NoLBERT: A No Lookahead(back) Foundational Language Model for Empirical Research

Ali Kakhbod, Peiyao Li · 2025

Read NoLBERT: A No Lookahead(back) Foundational Language Model for Empirical Research by Ali Kakhbod, Peiyao Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01110.txt.

Introduction

Language models trained on recent text can leak future knowledge into studies of the past, skewing results in economics and finance. Ali Kakhbod and Peiyao Li present NoLBERT, a small foundation model trained only on text from 1976 to 1995. By freezing its knowledge in that window, the model avoids both lookahead and lookback bias.

NoLBERT beat domain baselines on language tests while staying true to its time window. Kakhbod and Li then applied it to patent texts, building firm-level networks that show how companies link through new ideas. Firms that gained a more central place in those networks tended to see higher long-run profit growth. The work gives social scientists a tool that respects the flow of time.

Catalog record

CREATORAli Kakhbod, Peiyao Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01110

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NoLBERT: A No Lookahead(back) Foundational Language Model for Empirical Research Ali Kakhbod, Peiyao Li Abstract We present NoLBERT, a lightweight, timestamped foundational language model for empirical research in social sciences, particularly in economics and finance. By pre-training exclusively on 1976–1995 text, NoLBERT avoids both lookback and lookahead biases that can undermine econometric inference. It exceeds domain-specific baselines on NLP benchmarks while maintaining temporal consistency. Applied to patent texts, NoLBERT enables the construction of firm-level innovation networks and shows that gains in innovation centrality predict higher long-run profit growth. 1 Introduction Text-based methods have become increasingly central to empirical research in social sciences. In econometric settings that require time-consistent inference and backtesting, however, temporal biases can arise when language models are trained on corpora spanning long horizons: lookahead bias (implicit learning from future information) and lookback bias (conflation of meanings across eras), both of which can undermine temporal consistency. We introduce NoLBERT, a timestamped, domain-ready encoder. Built on the DeBERTa architecture introduced in He et al. 2020 , NoLBERT is pre-trained exclusively on 1976–1995 data, with validation on 1996. This narrow training horizon eliminates both lookback and lookahead biases while keeping the model compact (109M parameters), enabling large-scale text analysis on modest academic infrastructure. NoLBERT achieves higher performance on GLUE tasks relative to domain models such as FinBERT, while offering temporal constraints essential for empirical research. To illustrate its applicability in economic research, we apply NoLBERT to the study of firm innovation and performance. We fine-tune the model on patent texts to construct firm-level innovation networks, compute centrality measures, and link these to firm profit growth. Our econometric analysis shows that increases in innovation centrality are significantly associated with higher medium- and long-run profit growth. Together, these results highlight NoLBERT’s potential as a foundational tool for economic research: it combines strong language modeling performance with the temporal discipline necessary for credible econometric inference. 2 Pre-training NoLBERT 2.1 Base architecture and pre-training data For economists, a language encoder should (i) produce embeddings that are informative for downstream econometrics and (ii) be portable enough to run at scale on modest GPUs. We therefore adopt DeBERTa v3 base and pre-train it on text from 1976–1995, balancing compact size with strong language-modeling performance. Our pre-training corpus is curated to satisfy this time window while covering diverse, timestamped domains. We combine (a) popular-culture sources (movie scripts, TV dialogues, magazines, novels, blogs), (b) formal prose (parliamentary debates, campaign materials, news, academic papers), and (c) economics-relevant materials (FOMC transcripts, patents). See Appendix A.2 for dataset construction details. 2.2 Pre-training procedure and benchmark performance We pre-train our base model using data from 1976 to 1995 and use 1996 as validation data to track performance. To strictly prevent temporal biases, we first train a custom ByteLevelBPE tokenizer from scratch with a vocabulary size of 30,000 tokens and a minimum frequency threshold of 2, incorporating standard special tokens for masked language modeling. We then train our model with mixed precision over 15 epochs. We evaluate the performance of our model on the GLUE benchmark, including CoLA, SST2, QQP, MNLI, and QNLI. As shown in Table 1 , NoLBERT achieves the highest average performance when compared with similar models, FinBERT and StoriesLM, and it dominates the other two models in tasks other than CoLA ( Araci 2019 ; Sarkar 2025 ). Table 1: Performance comparison of financial language models across benchmark tasks. Model Vocabulary (K) #Params (M) CoLA SST2 QQP MNLI QNLI FinBERT 30 110 0.29 0.89 0.87 0.79 0.86 StoriesLM 30 110 0.49 0.90 0.87 0.80 0.87 NoLBERT 30 109 0.43 0.91 0.91 0.82 0.89 3 Lookback and lookahead biases 3.1 Avoiding lookahead and lookback biases using timestamped pre-training Lookahead bias occurs when models trained on future information contaminate predictions. For instance, when predicting stock returns from news articles, a model trained on data including future outcomes may simply retrieve memorized return patterns rather than genuinely inferring from text content. This renders predictions invalid for periods beyond the training data. We eliminate this bias by pre-training/validation exclusively on 1976-1996 data, ensuring no lookahead when backtesting strategies from 1997 onward. Lookback bias arises when models trained across long time periods produce representations that conflate different historical contexts. For example, the phrase " running a program " meant organizing an event in the early 1900s, but executing computer code by the late 20th century. Models trained on century-spanning data create ambiguous representations that may mischaracterize texts from specific periods. We avoid this by restricting training to the narrow 1976-1995 window, with validation strictly from 1996, ensuring temporal consistency in our text representations. 3.2 Validity check We empirically evaluate whether NoLBERT’s knowledge is temporally bounded to its pre-training window (1976–1995). To do so, we design two paired t -tests: one for lookahead bias and another for lookback bias . For lookahead bias, we construct 20 test words whose meanings shifted between 1976–1995 (old era) and 2020–present (new era). Using GPT-5, we generate sentence pairs in which the focal word is masked—one reflecting the old sense and the other the new. For example, the word “ token ” appears as “ He offered a <mask> of gratitude in the form of flowers. ” (old) versus “ She invested in a governance <mask> for the new DAO. ” (new). If the model is free from lookahead bias, it should predict the masked word more accurately in the context of its training era. For lookback bias, we repeat the procedure with another 20 words, where the “new” era is 1976–1995 and the “old” era is the 19th century. This tests whether the model’s understanding of language leans towards older meaning (lookback bias). Formally, let P_{\text{old}}(w_{i}) and P_{\text{new}}(w_{i}) denote the probabilities of predicting the masked word w_{i} in old versus new contexts. We conduct one-sided paired t -tests on \frac{1}{n}\sum_{i=1}^{n}\big[\log P_{\text{old}}(w_{i})-\log P_{\text{new}}(w_{i})\big]=0, with the alternative hypothesis positive for lookahead (favoring old meanings) and negative for lookback (favoring newer meanings). Table 2: Validity checks against lookahead and lookback biases. New era Old era Difference t -stat p -value 2020–present 1976–1995 -2.78 -5.56 <0.01 1976–1995 19th century 2.53 4.60 <0.01 As shown in Table 2 , the tests strongly reject the null of equal predictive accuracy across eras, confirming that NoLBERT is temporally localized: it favors contexts aligned with its training horizon and underperforms outside it. 4 Applications In this section, we present an end-to-end application where we use NoLBERT to form text representations and then conduct econometric analysis of the relationship between a firm’s centrality in the innovation network and its profit growth. 4.1 Fine-tuning NoLBERT via contrastive learning and patent text data We adopt a bottom-up approach to estimate firm-pair innovation similarities by fine-tuning NoLBERT with a contrastive learning objective. Our aim is to ensure that the document-level “[CLS]” embedding captures information relevant to innovation-specific textual similarity. More specifically, for each patent, we randomly split its text into two chunks, denoted A and B . We then construct a balanced dataset as follows: in 50% of the cases, chunk A is paired with chunk B from the same patent (labeled 1, representing 100% similarity), while in the remaining 50% of the cases, chunk A is paired with a chunk from a different patent (labeled 0). The model is

Read the complete text