How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes
2025
Read How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23358.txt.
Introduction
Sheri Osborn, Rohit Valecha, H Raghav Rao, Dan Sass, and Anthony Rios benchmark how well large language models forecast changes in job demand, especially in occupations touched by AI. Society lacks tools to predict AI effects on employment in a systematic way. Models can already extract sentiment and summarize economic reports, but forward-looking labor prediction was untested.
Their benchmark joins a high-frequency index of sector job postings with a second complementary data set. By scoring model forecasts against real posting shifts, the study shows where AI predictions of AI jobs can be trusted and where they mislead planners.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes Abstract Artificial intelligence is reshaping labor markets, yet we lack tools to systematically forecast its effects on employment. This paper introduces a benchmark for evaluating how well large language models (LLMs) can anticipate changes in job demand, especially in occupations affected by AI. Existing research has shown that LLMs can extract sentiment, summarize economic reports, and emulate forecaster behavior, but little work has assessed their use for forward-looking labor prediction. Our benchmark combines two complementary datasets: a high-frequency index of sector-level job postings in the United States, and a global dataset of projected occupational changes due to AI adoption. We format these data into forecasting tasks with clear temporal splits, minimizing the risk of information leakage. We then evaluate LLMs using multiple prompting strategies, comparing task-scaffolded, persona-driven, and hybrid approaches across model families. We assess both quantitative accuracy and qualitative consistency over time. Results show that structured task prompts consistently improve forecast stability, while persona prompts offer advantages on short-term trends. However, performance varies significantly across sectors and horizons, highlighting the need for domain-aware prompting and rigorous evaluation protocols. By releasing our benchmark, we aim to support future research on labor forecasting, prompt design, and LLM-based economic reasoning. This work contributes to a growing body of research on how LLMs interact with real-world economic data, and provides a reproducible testbed for studying the limits and opportunities of AI as a forecasting tool in the context of labor markets. How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes Sheri Osborn, Rohit Valecha, H. Raghav Rao, Daniel A. Sass, Anthony Rios The University of Texas at San Antonio {sheri.osborn, anthony.rios}.utsa.edu Abstract content 1. Introduction Artificial intelligence is changing how people work. Large language models (LLMs) such as GPT-4 Achiam et al. 2023 and LLaMA 2 Touvron et al. 2023 are now employed for tasks ranging from writing and coding to analysis and decision support. These tools can help workers produce more output, learn new skills faster, and take on tasks that were previously beyond their scope. At the same time, LLMs may change how firms design jobs and what skills they prioritize in hiring. This raises a crucial question: How will AI reshape the labor market across both AI-centric occupations and the broader economy? Answering this question is important for workers planning their careers, for firms investing in human capital, and for policymakers designing training programs and safety nets. However, forecasting labor-market changes driven by AI is challenging. Traditional time-series forecasting methods assume stable historical patterns Hyndman and Athanasopoulos 2018 ; Tsay 2005 , and even modern transformer-based forecasters rely on well-behaved numeric sequences Zhou et al. 2021 ; Wu et al. 2021 ; Zhou et al. 2022 . AI adoption does not necessarily follow such stable patterns; it is influenced by irregular, unstructured signals from news events, research breakthroughs, product launches, policy changes, and shifts in job postings. LLMs offer a new approach: they can read and reason over text describing these developments, integrate those insights with quantitative data, and produce structured predictions. Early evidence suggests this approach is promising. LLMs have been shown to extract market sentiment Bond et al. 2024 , map news to stock returns Lopez-Lira and Tang 2023 ; Van Binsbergen et al. 2022 ; Tan et al. 2024 , and reason about macroeconomic outcomes Chen et al. 2023 ; Rahimikia and Drinkall 2024 . They can even emulate professional forecasters in surveys Hansen et al. 2024 and reveal economic reasoning heuristics or biases Ross et al. 2024 ; Ludwig et al. 2025 ; Manning et al. 2024 . At the same time, research on evaluation warns about pitfalls: if we do not handle timing carefully, lookahead leakage can inflate predictive performance Glasserman and Lin 2023 ; Levy 2024 ; He et al. 2025 ; Sarkar and Vafa 2024 , and the choice of prompting strategy can affect an LLM’s reliability as a forecaster Pham and Cunningham 2024 . These insights highlight both the potential of LLM-based forecasting and the need for careful, robust design. Recently, Natural Language Processing (NLP) researchers have begun examining LLMs as general-purpose forecasters. Some works adapt prompts or “reprogram” LLMs to handle purely numeric time-series data Jin et al. 2023 ; Cao et al. 2023 , while others develop sequence modeling architectures specialized for time series forecasting Rasul et al. 2023 ; Zhou et al. 2024 ; Liu et al. 2023 ; Jin et al. 2024 . With the right tokenization and prompting, LLMs have even demonstrated zero-shot ability on forecasting tasks Gruver et al. 2024 . This progress suggests a straightforward recipe for labor market prediction: we can combine official labor statistics with text streams about technology and firms, then prompt an LLM to generate interpretable forecasts of job trends and skill demand. Yet, most LLM-based economic forecasting to date has centered on financial assets or high-level macroeconomic indicators rather than employment outcomes Bond et al. 2024 ; Lopez-Lira and Tang 2023 ; Chen et al. 2023 ; Hansen et al. 2024 . In particular, the field lacks a dedicated benchmark that merges unstructured text and structured data to evaluate sector-specific and skill-specific employment predictions. We also have little evidence on how different prompt designs perform in this context, even though initial studies indicate prompt design can significantly impact forecasting performance Pham and Cunningham 2024 . Moreover, rigorous evaluation protocols that respect temporal order are often absent, despite their importance for avoiding false optimism in forecasting tasks Glasserman and Lin 2023 ; Levy 2024 ; He et al. 2025 . To address these gaps, we introduce a benchmark for AI and the labor market and use it to study how well LLMs can forecast employment trends. We focus on evaluating forecasts for both AI-intensive jobs (roles directly related to AI development or deployment) and general jobs across the broader economy, to capture both direct impacts of AI and indirect spillover effects. Our benchmark links each time period’s official labor statistics with contemporaneous textual signals: in one setting we use long-horizon reports on global AI adoption, and in another we use real-time streams of online job postings. We then prompt LLMs to produce sector-level predictions in a step-by-step, transparent manner. All experiments enforce strict time-based splits and leakage controls Glasserman and Lin 2023 ; Levy 2024 ; He et al. 2025 to ensure that models are never evaluated on future data that they could have implicitly seen. We compare a range of prompting strategies, from minimal persona-style prompts to structured, task-specific prompts, to assess their impact on forecast accuracy. In summary our contributions are: (1.) We present a reproducible benchmark to forecast employment changes that has been tested on several LLMs. It fuses structured labor data with real-world text streams, spanning both AI-intensive and general occupations to capture direct and systemic labor shifts. 1 1 1 All data and code will be released upon acceptance. (2.) We systematically compare prompting strategies—contrasting persona-driven and task-structured prompts across LLM families and forecasting horizons. The research is performed under time-consistent, leakage-controlled evaluation. Glasserman and Lin 2023 ; Levy 2024 ; He et al. 2025 to ensure realistic forecasting conditions. This