Are Foundation Models Useful for Bankruptcy Prediction?
Marcin Kostrzewa, Oleksii Furman, Roman Furman, Sebastian Tomczak, Maciej Zięba · 2025
Read Are Foundation Models Useful for Bankruptcy Prediction? by Marcin Kostrzewa, Oleksii Furman, Roman Furman, Sebastian Tomczak, Maciej Zięba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16375.txt.
Introduction
Marcin Kostrzewa and four co-authors run the first systematic test of foundation models for predicting corporate bankruptcy, a task where bad probability estimates can do real harm. They try Llama-3.3-70B-Instruct and TabPFN on highly imbalanced data covering more than a million company records from the Visegrád Group countries.
Classical gradient boosting methods, XGBoost and CatBoost, beat the foundation models at every prediction horizon. The language model approach gave unreliable probabilities, a serious flaw for risk use. TabPFN matched simpler baselines but cost far more computation than its results justified. The findings caution finance teams that general-purpose models, for all their breadth, still trail specialized methods on this forecasting task.
Catalog record
| CREATOR | Marcin Kostrzewa, Oleksii Furman, Roman Furman, Sebastian Tomczak, Maciej Zięba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are Foundation Models Useful for Bankruptcy Prediction? Marcin Kostrzewa, Oleksii Furman, Roman Furman, Sebastian Tomczak, Maciej Zięba Abstract Foundation models have shown promise across various financial applications, yet their effectiveness for corporate bankruptcy prediction remains systematically unevaluated against established methods. We study bankruptcy forecasting using Llama-3.3-70B-Instruct and TabPFN, evaluated on large, highly imbalanced datasets of over one million company records from the Visegrád Group. We provide the first systematic comparison of foundation models against classical machine learning baselines for this task. Our results show that models such as XGBoost and CatBoost consistently outperform foundation models across all prediction horizons. LLM-based approaches suffer from unreliable probability estimates, undermining their use in risk-sensitive financial settings. TabPFN shows mixed results — underperforming on ROC-AUC but competitive on F 1 F_{1} -score — while requiring substantial computational resources that cannot be justified by its performance. These findings suggest that, despite their generality, current foundation models remain less effective than specialized methods for bankruptcy forecasting. \workshoptitle Generative AI in Finance 1 Introduction Foundation models such as large language models (LLMs) are increasingly applied across various financial domains, as documented in recent survey papers [ 12 , 17 , 25 ] . Applications span sentiment analysis, algorithmic trading, fraud detection, credit rating, and other financial tasks. Their appeal lies mainly in the ability to provide solutions without extensive task-specific engineering. One of the most extensively studied financial prediction problems is corporate bankruptcy 1 1 1 Throughout the paper we will use terms default , financial distress , and bankruptcy interchangeably. forecasting [ 7 , 9 , 11 , 21 , 27 ] . Accurate default prediction is of practical relevance to investors, regulators, and policymakers, while the task’s structured nature and requirement for reliable probability estimates present unique challenges for general-purpose models. While foundation models have been applied to various financial risk tasks, systematic evaluation against classical methods for bankruptcy prediction, particularly on large-scale structured datasets, remains unexplored. In this work, we address this gap by applying two foundation models — Llama-3.3-70B-Instruct [ 1 ] for textual inputs and TabPFN [ 8 ] for tabular data — to corporate default forecasting. We evaluate them on highly imbalanced datasets of financial records from the Visegrád Group, containing hundreds of thousands of samples. While recent work has addressed TabPFN’s scalability limitations [ 14 , 23 ] , these solutions remain untested for financial prediction tasks against domain-optimized classical methods. We extensively compare these approaches against classical machine learning baselines and analyze the reliability of foundation model outputs for structured financial prediction. Our contributions can be summarized as follows: • We provide the first systematic comparison of foundation models (LLMs and TabPFN) against classical ML baselines for bankruptcy prediction on large-scale structured financial data. • We show that LLM self-reported probability estimates are poorly calibrated and discretized, undermining their reliability for risk assessment, and that TabPFN incurs substantial computational overhead without commensurate performance gains. 2 Related Work Foundation models in finance. Recent work has demonstrated foundation model capabilities across financial domains, from sentiment analysis and time series forecasting [ 5 , 13 , 17 ] to risk prediction. FinPT applies LLMs to personal financial risk prediction [ 24 ] , while time series foundation models like FinCast and Kronos show strong performance for market forecasting [ 19 , 26 ] . For bankruptcy prediction specifically, multimodal approaches have been explored using textual and numerical data from US markets [ 3 , 16 ] , but systematic evaluation against classical methods on structured data remains limited. Bankruptcy prediction methods. Traditional machine learning approaches dominate bankruptcy prediction, with gradient boosting methods (XGBoost, CatBoost) and neural networks consistently achieving strong performance on financial datasets [ 4 , 10 , 27 ] . Recent studies have incorporated more advanced deep learning techniques [ 9 , 22 ] and ensemble methods [ 15 ] . Despite foundation model advances in other financial areas, no systematic comparison exists evaluating their effectiveness against these established approaches for bankruptcy prediction on large-scale datasets. 3 Data and methodology 3.1 Data We construct five datasets based on over one million financial statement records from 2006-2021 of companies from the V4 group (Czech Republic, Hungary, Poland, Slovakia). The data is a part of our ongoing research and will be described in detail in a separate publication. Each dataset corresponds to a different prediction horizon: immediate bankruptcy risk (0 years ahead) up to four years ahead. The target variable is binary, indicating whether a company will experience financial distress as defined by a set of financial ratio thresholds. This multi-horizon approach enables systematic evaluation of model performance as prediction difficulty increases with longer time periods. Table 1 summarizes the five datasets, showing the expected decline in both sample size and bankruptcy cases as prediction horizons extend. The severe class imbalance (bankruptcy rates below 1%) reflects real-world conditions and necessitates careful evaluation using metrics appropriate for imbalanced classification. Detailed feature descriptions and target variable construction are provided in Appendix B . Table 1: Overview of datasets and prediction tasks. Prediction Horizon Total Instances Bankruptcy Non-bankruptcy 0 years (current) 1,000,087 3,587 996,500 1 year ahead 996,500 3,054 993,446 2 years ahead 898,692 2,374 896,318 3 years ahead 793,234 1,896 791,338 4 years ahead 700,041 1,485 698,556 Evaluation subsets. For each prediction horizon, all models are evaluated on the same stratified test subset of 20,000 samples to ensure computational feasibility and fair comparison across methods. The remaining data is split into training and validation sets, with validation data used for hyperparameter optimization and probability threshold calibration. 3.2 Methods To evaluate foundation models against classical approaches for bankruptcy prediction, we employ both a LLM and a tabular foundation model alongside established baselines. For each dataset, we report ROC-AUC and F_{1} -score, with the latter being particularly important for highly imbalanced data. The decision threshold is calibrated on the validation dataset by choosing the value maximizing F_{1} -score. Detailed results for additional metrics are provided in Appendix D . Foundation models We evaluate two representative foundation models, each addressing different data modalities. For textual input processing, we use Llama-3.3-70B-Instruct [ 1 ] , serializing company features into natural-language prompts that request binary labels and self-reported probabilities. We test both zero-shot prediction and in-context learning (ICL) with 20 examples (10 positive, 10 negative). We access Llama-3.3 via API, effectively treating it as a closed model since we cannot examine weights or internal representations. For direct tabular processing, we utilize TabPFN [ 8 ] , a transformer-based model designed for small tabular datasets. Since TabPFN is optimized for datasets under 10,000 samples while our datasets contain hundreds of thousands of records, we implement a partition-then-predict approach where a decision tree partitions the feature space and TabPFN processes smaller leaf subsets. Detailed prompt design and scaling approaches are described in Appendices C.1 and C.2 . Classical baselines. We compare foundation models against five established methods: logistic regression, multi-layer perceptron, XGBoost, LightGBM, and CatBoost. We follow a standard training protocol with hyperparameter optimization via grid search on validation data. Hyperparameters are selected to maximize F_{1} -score, prioritizing performance on the minority class. Complete hyperparameter grids and training specifications are provided in Appendix C.3 . 4 Experiments and results The results