ForestLLM : Large Language Models Make Random Forest Great on Few-shot Tabular Learning
Zhihan Yang, Jiaqi Wei, Xiang Zhang, Haoyu Dong, Yiwen Wang, Xiaoke Guo, Pengkun Zhang, Yiwei Xu, Chenyu You · 2026
Read ForestLLM : Large Language Models Make Random Forest Great on Few-shot Tabular Learning by Zhihan Yang, Jiaqi Wei, Xiang Zhang, Haoyu Dong, Yiwen Wang, Xiaoke Guo, Pengkun Zhang, Yiwei Xu, Chenyu You (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11311.txt.
Introduction
Zhihan Yang and eight coauthors revive random forests with language models. Their 2026 preprint is named ForestLLM, on few-shot tabular learning. Tables drive high-stakes choices in finance, health, and science. Learning from tables with few labeled rows is still hard.
Classic tree methods lean on purity scores to split data. With few labels, those scores turn shaky and prone to overfit. Random forests, a strong classic tool, falter in that regime. ForestLLM brings large language model knowledge to the trees. The aim is splits guided by meaning, not just counts. That help could steady tree learning when labels are scarce.
Catalog record
| CREATOR | Zhihan Yang, Jiaqi Wei, Xiang Zhang, Haoyu Dong, Yiwen Wang, Xiaoke Guo, Pengkun Zhang, Yiwei Xu, Chenyu You |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11311
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ForestLLM : Large Language Models Make Random Forest Great on Few-shot Tabular Learning Zhihan Yang, Jiaqi Wei, Xiang Zhang, Haoyu Dong, Yiwen Wang, Xiaoke Guo, Pengkun Zhang, Yiwei Xu, Chenyu You Abstract Tabular data high-stakes critical decision-making in domains such as finance, healthcare, and scientific discovery. Yet, learning effectively from tabular data in few-shot settings, where labeled examples are scarce, remains a fundamental challenge. Traditional tree-based methods often falter in these regimes due to their reliance on statistical purity metrics, which become unstable and prone to overfitting with limited supervision. At the same time, direct applications of large language models (LLMs) often overlook its inherent structure, leading to suboptimal performance. To overcome these limitations, we propose ForestLLM , a novel framework that unifies the structural inductive biases of decision forests with the semantic reasoning capabilities of LLMs. Crucially, ForestLLM leverages the LLM only during training, treating it as an offline model designer that encodes rich, contextual knowledge into a lightweight, interpretable forest model, eliminating the need for LLM inference at test time. Our method is two-fold. First, we introduce a semantic splitting criterion in which the LLM evaluates candidate partitions based on their coherence over both labeled and unlabeled data , enabling the induction of more robust and generalizable tree structures under few-shot supervision. Second, we propose a one-time in-context inference mechanism for leaf node stabilization , where the LLM distills the decision path and its supporting examples into a concise, deterministic prediction, replacing noisy empirical estimates with semantically informed outputs. Across a diverse suite of few-shot classification and regression benchmarks, ForestLLM achieves state-of-the-art performance. These results highlight the promise of using LLMs as offline model designers, rather than online predictors, for scalable and efficient tabular learning under data scarcity. Our code is available at https://github.com/kelvin715/ForestLLM . Introduction Tabular data, structured as collections of heterogeneous, feature-label pairs, is central to decision-making in critical domains such as healthcare, finance, and public policy ( 27 ; 21 ; 2 ; 30 ; 51 ; 1 ; 11 ; 62 ) . However, building reliable models from such data in the few-shot regime remains a longstanding challenge. Unlike images or text, tabular data lacks spatial or sequential structure, and its highly combinatorial, non-Euclidean nature undermines the geometric assumptions underlying most modern few-shot learning techniques ( 12 ; 36 ; 42 ) . As a result, these methods often perform poorly on tabular tasks with limited supervision. Tree-based models such as Random Forests and Gradient Boosted Trees ( 10 ; 32 ; 44 ) remain strong performers on tabular benchmarks, thanks to their inductive bias toward hierarchical data partitioning. Yet under low-data conditions, their reliability degrades. Conventional split selection heuristics, like information gain or Gini impurity, rely entirely on labeled data and become unstable when estimates are noisy. Moreover, predictions at the leaves are often based on only a handful of examples, resulting in high variance and poor calibration. These issues are further exacerbated by the fact that unlabeled data, though often available, is ignored entirely during training. Table 1: Comparison of LLM-based Tabular Methods. Method No LLM Fine-tuning Needed No LLM Needed at Inference Task Support Classification Regression LIFT (NeurIPS'22) ✗ ✗ ✓ ✓ TabLLM (AISTATS'23) ✗ ✗ ✓ ✗ FeatLLM (ICML'24) ✓ ✓ ✓ ✗ TP-BERTa (ICLR'24) ✓ ✗ ✓ ✓ P2T (COLM'24) ✓ ✗ ✓ ✓ ForestLLM (Ours) ✓ ✓ ✓ ✓ In parallel, the generalization and reasoning capabilities of Large Language Models (LLMs) have spurred interest in applying them to tabular learning ( 16 ; 26 ; 25 ; 58 ; 40 ; 55 ; 63 ; 57 ) . But most existing approaches serialize rows into text and rely on prompting or fine-tuning, discarding the structural regularities that classical models are designed to exploit. Despite their flexibility, such methods frequently underperform tree ensembles on core supervised tasks, particularly in few-shot settings ( 24 ; 8 ; 38 ; 61 ) . In this paper, we explore a new direction: rather than replacing classical tabular models with foundation models, can we instead enlist LLMs as offline architects, guiding model design during training but removed entirely at test time? Can we use LLMs to imbue decision trees with semantic inductive priors, while preserving their efficiency, robustness, and interpretability? To this end, we propose ForestLLM , a novel decision forest framework that leverages a LLM as an offline model designer , guiding model construction without participating in inference. Rather than using the LLM for inference, ForestLLM engages it once during training to guide both split selection and leaf prediction. Our ForestLLM is fast, interpretable, and entirely LLM-free at test time. At its core, ForestLLM rethinks decision tree construction under few-shot supervision through a new framework for splitting and prediction. Our approach is two-fold. First, we introduce semi-supervised semantic tree induction, a split selection strategy that replaces conventional label-driven heuristics with LLM-guided semantic evaluation. Each candidate split is summarized into a prompt that captures labeled and unlabeled feature distributions along with local tree context. The LLM then ranks these candidates based on their semantic plausibility, effectively leveraging unlabeled data to guide generalization under limited supervision. Second, we propose in-context leaf label inference, a one-time procedure for producing stable, static predictions at leaf nodes. Each decision path is translated into a natural language rule, which, together with its supporting examples, is used to prompt the LLM for a final prediction. This decouples inference from sparse empirical estimates, reducing variance while preserving efficiency. ForestLLM bakes in the structural bias of decision forests with the semantic abstraction capabilities of LLMs, without requiring fine-tuning or inference time access (Table 1 ). It operates entirely in a few-shot, in-context regime and supports both classification and regression tasks. By repositioning the LLM as an offline architect rather than an online oracle, ForestLLM enables scalable, low-resource learning from tabular data. We evaluate ForestLLM across a broad set of few-shot tabular benchmarks and find that it consistently matches or outperforms strong baselines, often achieving state-of-the-art or second-best performance with as few as 4 to 48 labeled examples. Notably, it retains strong performance on datasets that were unseen during LLM pretraining, underscoring its generalization ability beyond memorized priors. Our key contributions are as follows: • We introduce the concept of using a LLM as an offline model designer , guiding training-time structure induction without relying on the LLM at test time. • We propose a novel semantic splitting strategy that replaces statistical heuristics with LLM-evaluated prompts over labeled and unlabeled data. • We develop a single-pass, in-context leaf prediction mechanism that produces deterministic, low-variance outputs without empirical averaging or LLM inference. • We demonstrate that ForestLLM achieves state-of-the-art few-shot performance on a wide range of tabular classification and regression benchmarks, without fine-tuning or inference-time prompting. Figure 1: An overview of our proposed ForestLLM framework for few-shot tabular learning. Related Work Few-Shot and Semi-Supervised Tabular Learning. Few-shot tabular learning poses a core challenge: enabling generalization from limited labeled data ( 12 ; 36 ) . A common line of work addresses this by leveraging unlabeled data to learn expressive representations via semi- or self-supervised pretraining ( 48 ; 60 ; 50 ; 5 ; 54 ) . In contrast, ForestLLM takes a fundamentally