Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data
Renat Sergazinov ∗, Shao-An Yin ∗ · 2025
Read Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data by Renat Sergazinov ∗, Shao-An Yin ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00326.txt.
Introduction
Renat Sergazinov and Shao-An Yin present Chunked TabPFN in 2025, a way to run a strong tabular model on long data sets with no training and no pre-processing. TabPFN v2 beats tree-based models on many table benchmarks, which is notable since trees usually lead on tables. Its limit is ten thousand context tokens, set by the quadratic cost of transformer attention.
Instead of compressing context by picking sample rows, their tiled-block strategy computes attention in chunks inside the TabPFN frame. The design fits standard GPU setups. Tests on the TabArena benchmark show the method works. Long tables in medicine and finance could now use in-context learning as-is.
Catalog record
| CREATOR | Renat Sergazinov ∗, Shao-An Yin ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data Thanks: ∗ Equal contribution. Renat Sergazinov ∗, Shao-An Yin ∗ Abstract . TabPFN v2 achieves better results than tree-based models on several tabular benchmarks, which is notable since tree-based models are usually the strongest choice for tabular data. However, it cannot handle more than 10K context tokens because transformers have quadratic computation and memory costs. Unlike existing approaches that rely on context compression, such as selecting representative samples via K-nearest neighbors (KNN), we introduce a tiled-block strategy to compute attention within the TabPFN framework. This design is compatible with standard GPU setups and, to the best of our knowledge, is the first to enable TabPFN to process long contexts without any pre-processing . We demonstrate the effectiveness of our approach on the standard TabArena benchmark, with code available at chunk_tabpfn . 1. Introduction Even though deep learning foundation models that capture the underlying data manifold have shown promising results in language and vision by enabling broad transfer without fine-tuning, applying the same techniques to tabular data remains highly challenging. A key difficulty is that deep learning methods often underperform compared to tree-based models on tabular benchmarks. Recent work in top venues has advanced deep tabular learning through architectures and priors specifically tailored for tabular data. Representative milestones include TabNet [ 1 ] , FT-Transformer [ 5 ] , NODE [ 12 ] , efficient MLP models [ 7 ] , retrieval-based architectures such as TabR [ 6 ] and ModernNCA [ 15 ] , as well as zero-shot foundation models such as TabPFN [ 8 , 9 ] and TabDPT [ 11 ] . Still, most deep tabular models need to be trained for each specific dataset and task, which means they are not true foundation models that can serve as plug-and-play solutions. Recent advances in in-context learning and foundation models for large language models (LLMs) suggest promising directions for developing tabular foundation models. Inspired by in-context learning, the family of Prior-Data Fitted Networks (PFNs) eliminates the need for per-dataset training, thereby offering a unified foundation model for tabular data. PFNs achieve this by training a single transformer once to approximate the posterior predictive distribution p(y\mid x,D_{\text{train}}) on synthetic tasks sampled from a prior. At test time, predictions on a new tabular dataset can then be obtained in a single forward pass without any gradient updates. In this work, we build on the TabPFN v2 implementation [ 8 , 9 ] . However, the limited context window remains a practical limitation of PFN models. The original TabPFN [ 8 ] could handle datasets with at most 3,000 samples, while the later version by Hollmann et al. 2025 extended this to 10,000 samples. Nevertheless, this scale is still restrictive in many real-world scenarios. To mitigate this issue, several works have proposed pre-processing strategies to reduce the effective context size, which can be broadly categorized into two approaches: clustering the training set [ 9 , 14 , 13 ] and data compression [ 4 ] . Within the category of clustering-based approaches, Xu et al. 2025 proposed Mixture of In-Context Prompters (MICP), which forms a fixed set of in-context prompts, learns a routing mechanism, and further fine-tunes TabPFN using bootstrapping within each cluster. Similarly, Hollmann et al. 2025 introduced an ensemble-like method that trains a random forest to partition the data into subsets, on top of which TabPFN is applied. In parallel, Thomas et al. 2024 proposed a k NN-based strategy, where the “clusters” are not predefined but instead sampled per test point, at the cost of increased computational overhead. On the other hand, within the category of data-compression methods, Feuer et al. 2024 proposed TuneTables , which learns compact, dataset-specific embeddings to replace or augment the context. TuneTables optimizes a small table of embeddings that serves as a surrogate for the full dataset, thereby improving both accuracy and latency through parameter-efficient updates. In addition, it explores multi-objective tuning (e.g., accuracy–fairness trade-offs) and provides analyses of different context optimization strategies. Even though the above approaches have demonstrated strong performance, two fundamental issues remain with respect to addressing the long-context challenge: 1. They require per-dataset hyperparameter tuning and fine-tuning, which undermines the principle of in-context learning and the vision of foundation models, where predictions should be made in a single forward pass without task-specific adaptation. 2. From a theoretical perspective, the core principle of TabPFN—fitting the posterior through the provided context—is violated when the original long context is replaced by clustered or compressed representations. Given the above fundamental limitations, directly using all training examples as context is preferable from both practical and theoretical perspectives. In this paper, we address the long-context challenge without any pre-processing by posing the following research questions: 1. How can the memory complexity of long contexts be resolved without pre-processing the training examples? 2. How well does TabPFN perform in a direct, one-shot in-context learning setting on long contexts without any pre-processing? Our contributions are summarized as follows: 1. Training-free long-context extension. We introduce a simple, pure-PyTorch modification that computes attention in chunks, supporting both FlashAttention [ 2 ] and native PyTorch scaled dot-product attention. This enables TabPFN to operate on datasets with over 100K samples without any retraining. 2. Empirical analysis. We systematically evaluate TabPFN v2 on the TabArena benchmark. On a curated subset of 17 long-context tasks, we observe consistent performance gains from additional in-context examples, even beyond the original pre-training limits. This establishes a strong training-free baseline for future research and challenges the prevailing assumption that TabPFN models cannot scale beyond their pre-training context window. 2. Methodology In this section, we introduce our chunked TabPFN method. We begin by reviewing the standard attention computation and its role in PFN models. Next, we identify the key memory complexity bottleneck that prevents standard TabPFN from scaling to long contexts. Finally, we present our approach, which partitions the input into chunks and performs attention operations in a batch-wise manner, thereby mitigating out-of-memory (OOM) issues. Notation. We denote the training set (context) as D_{\text{train}}=\{(x_{i},y_{i})\}_{i=1}^{n} , where each feature vector is x\in\mathbb{R}^{p} with p features and the corresponding label is y\in\mathcal{Y} . A test input is denoted by x_{\ast} , and the test set is given by D_{\text{test}}=\{(x_{j},y_{j})\}_{j=1}^{m} . Attention shapes. For the attention block, let L denote the total sequence length consumed by the transformer (i.e., context tokens plus query), B the batch size (number of test points processed in parallel), H the number of attention heads, and d_{k} the per-head key/query dimension. We adopt the standard notation of Query (Q), Key (K), and Value (V), where the attention operation is applied on (1) \begin{split}Q&\in\mathbb{R}^{B\times H\times L_{q}\times d_{k}},\\ K&\in\mathbb{R}^{B\times H\times L_{k}\times d_{k}},\\ V&\in\mathbb{R}^{B\times H\times L_{k}\times d_{k}},\end{split} with Q , K , and V being linear projections of the input data sequence. In the common case where L_{q}=L_{k}=L_{v} , we simply write L for brevity. PFN model sketch. Prior-Data Fitted Networks (PFNs) train a transformer once on synthetic tasks sampled from a prior, such that at inference time a new tabular dataset can be processed in a single forward pass without gradient updates (i.e., the model parameters remain fixed). PFNs approximate the posterior predictive distribution (2) p(y_{\ast}\mid x_{\ast},D_{\text{train}}). The model ingests