Large Language Models for Predictive Analysis: How Far Are They?

Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi · 2025

Read Large Language Models for Predictive Analysis: How Far Are They? by Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17149.txt.

Introduction

Qin Chen and three co-authors present a study from twenty twenty-five on how well large language models can forecast from data. Forecasting guides choices in fields from health to trade. Models shine at talk and code, yet skill at forecasts from real data was untested. The team built a test set, named PredictiQ, to fill that gap.

The set holds over a thousand queries drawn from dozens of real data sets in eight fields. Their checks weigh text insight, code craft, and how well the two line up. Twelve known models took the test. All faced steep hurdles in this kind of work. The team shares the test in public for later use.

Catalog record

CREATORQin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models for Predictive Analysis: How Far Are They? Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi Abstract Predictive analysis is a cornerstone of modern decision-making, with applications in various domains. Large Language Models (LLMs) have emerged as powerful tools in enabling nuanced, knowledge-intensive conversations, thus aiding in complex decision-making tasks. With the burgeoning expectation to harness LLMs for predictive analysis, there is an urgent need to systematically assess their capability in this domain. However, there is a lack of relevant evaluations in existing studies. To bridge this gap, we introduce the PredictiQ benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. We design an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis. Generally, we believe that existing LLMs still face considerable challenges in conducting predictive analysis. See Github . 1 Introduction Predictive analysis ( Kumar and Garg 2017 ) involves making predictions about future outcomes based on past data, using statistical modeling, data mining, and machine learning techniques. It is widely used in decision-making across various fields like business ( Che et al. 2024 ) and healthcare ( Dixon et al. 2024 ) . Figure 1: An example of users conducting predictive analysis via Large Language Models. To facilitate off-the-shelf predictive analysis for users without data analysis expertise, Large Language Models (LLMs) ( Brown et al. 2020 ; OpenAI 2023 ; Anil et al. 2023 ; Abdin et al. 2024 ) serve as powerful tools, supporting users with coherent and contextually relevant responses. Figure 1 shows how users engage with LLMs by submitting predictive queries based on specific datasets. Given the substantial potential of LLM-based predictive analysis, a comprehensive evaluation of leading LLMs is essential. While no studies specifically assess LLM-based predictive analysis, related studies Zhao et al. 2023b ; Chen 2022 ; Saeed et al. 2023 ; Gao et al. 2023 ; He et al. 2023 ; Abolghasemi et al. 2024 ; Pratt et al. 2024 ; Hong et al. 2024 in relevant fields primarily focus on evaluating either the model’s outcomes only (e.g., a total sales figure answering a database query) or the generated codes that, when executed, produce these outcomes. However, the former raises scalability issues as LLMs may struggle to process the entire lengthy dataset due to limited context length. The latter lacks textual explanations, such as justifications for algorithm selection, which constrains its applicability and undermines user trust. This makes existing evaluation efforts insufficient when directly adapted to predictive analysis, where tasks are inherently more complex. LLM-based predictive analysis typically requires handling tasks such as data preprocessing, algorithm selection, result interpretation, and so on. These tasks require both textual explanations and code implementations . In practical applications, textual analysis is crucial for explaining algorithm selection and enhancing reliability. Moreover, effective alignment between text and code improves user comprehension of LLM-generated solutions. Thus, textual analysis , code generation , and text-code alignment are integral to the functionality of AI-driven assistants. In this paper, we propose the PredictiQ (Q stands for query) benchmark to evaluate the potential of current LLMs in predictive analysis. We first collect datasets covering eight common fields, including economics, traffic, and more. Diverse datasets form the basis for PredictiQ to assess LLMs’ performance in predictive analysis. Based on the collected datasets, we collaborate with data analysis experts to formulate predictive queries tailored to the datasets. Each query is limited to a single, self-contained question that relies solely on the dataset, without requiring external knowledge. We then formulate the data-specific queries and corresponding datasets into prompts, which are provided to LLMs to generate responses. We propose an evaluation protocol encompassing three domains: textual analysis, code generation, and text-code alignment, focusing on seven key aspects. The responses from the LLMs are then evaluated by (i) data analysis experts and (ii) LLMs. The LLM whose responses align most closely with those of the human experts is selected as the primary evaluator. In this study, we adopt GPT4Turbo, which achieves an average alignment ratio of 90.5% with human experts. PredictiQ involves 1130 queries from 44 datasets across 8 fields, requiring 300 human hours for query generation and examination. Our evaluation on PredictiQ costs 900 human hours for response evaluation, 72.18 million input tokens, and 20.4 million output tokens. We conduct extensive experiments on PredictiQ with twelve renowned LLMs. We find that (i) Fine-tuning LLMs on code enhances both text analysis and code generation, occasionally pushing the model beyond its parameter limits. (ii) Code generation and text analysis are interconnected processes that shape LLMs’ overall predictive analysis proficiency. (iii) Several LLMs’ predictive analysis abilities vary significantly across different fields, with wide margins, and exhibit diversified score distributions. We believe existing LLMs are far from mastering predictive analysis, both in terms of performance and efficiency. Our contributions are summarized as follows. • We propose the PredictiQ benchmark—a comprehensive evaluation framework for LLMs in predictive analysis. It covers the entire analytic process by integrating 44 datasets across 8 real-world fields, 1,130 rigorously curated data-specific queries, and evaluation protocols. This approach goes beyond existing benchmarks that focus only on basic descriptive tasks. • We systematically evaluate the performance of twelve LLMs on the PredictiQ benchmark, highlighting insights that could improve the efficacy of LLMs in predictive analysis. 2 Preliminary Predictive Analysis refers to the use of statistical algorithms and machine learning techniques to analyze data patterns, predict future events, trends, or behaviors Siegel 2013 ; Kumar and Garg 2018 . It is widely used in various fields, including finance, marketing, healthcare, and risk management, to anticipate potential risks, identify opportunities, and inform the decision-making process. See Figure 1 for an illustration. Problem Definition. The predictive analysis process with LLMs is formalized as follows: (\text{query},\text{data})\xrightarrow{\text{LLM}}(\text{text},\text{code}). (1) The input consists of a predictive analysis query associated with corresponding data . A query is a data-specific predictive question that requests future predictions, identifies implicit data patterns, and so on. The data consists of multiple columns and rows. The output includes text , which explains how the prediction is made and justifies the chosen method, and code , which implements the analysis. Ideally, the text should provide clear and in-depth analysis based on the query and data . The code should provide related and functionally correct implementation that solves the problem raised by the query . 3 The PredictiQ Benchmark This section outlines the details of the PredictiQ benchmark, covering (i) data collection, (ii) query formulation, and (iii) response evaluation. 3.1 Data Collection The PredictiQ benchmark evaluates LLMs’ predictive analysis capabilities across diverse fields. It includes various publicly available datasets from real-world scenarios, ensuring the benchmark’s diversity and validity. The datasets are organized into eight fields that are prevalent in real-world predictive analysis applications, as presented in Table 1 . See Appendix A for detailed descriptions. 3.2 Query Collection Table 1: Statistics of datasets and generated queries. Field #Datasets #Queries Economics 12 270 Marketing and Sales 6 200 Industry Analysis 7 180 Traffic 5 130 Healthcare 4 130 Social Study 4 110 Human Resource 3 80 Education 3 70 Total 44 1130 In this subsection, our goal is to

Read the complete text