Automatic Demonstration Selection for LLM-based Tabular Data Classification

Shuchu Han, Wolfgang Bruckner · 2025

Read Automatic Demonstration Selection for LLM-based Tabular Data Classification by Shuchu Han, Wolfgang Bruckner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20451.txt.

Introduction

Shuchu Han and Wolfgang Bruckner give a method to pick how many worked examples a language model needs to sort table data. Their paper from 2025 deals with in context learning, where examples sit in the prompt. Too few can starve the model, and too many waste space, yet the right count was a guess.

Their method counts not just the data spread but also the prompt template and the model in use. A new score measures how alike examples are, built on spectral graph theory. A graph of those scores is built, and its spectrum gives the least count that can stand for the data in the model own space. Tests across data sets and models beat random example picking.

Catalog record

CREATORShuchu Han, Wolfgang Bruckner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20451

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Demonstration Selection for LLM-based Tabular Data Classification Shuchu Han, Wolfgang Bruckner Abstract A fundamental question in applying In-Context Learning (ICL) for tabular data classification is how to determine the ideal number of demonstrations in the prompt. This work addresses this challenge by presenting an algorithm to automatically select a reasonable number of required demonstrations. Our method distinguishes itself by integrating not only the tabular data’s distribution but also the user’s selected prompt template and the specific Large Language Model (LLM) into its estimation. Rooted in Spectral Graph Theory, our proposed algorithm defines a novel metric to quantify the similarities between different demonstrations. We then construct a similarity graph and analyze the eigenvalues of its Laplacian to derive the minimum number of demonstrations capable of representing the data within the LLM’s intrinsic representation space. We validate the efficacy of our approach through experiments comparing its performance against conventional random selection algorithms on diverse datasets and LLMs. 1 Introduction Within enterprise settings, tabular data is the most prevalent form of information. The emergence of Generative AI has prompted researchers and organizations to investigate methods for leveraging Large Language Models (LLMs) to derive insights from data stored in Relational Database Management Systems (RDBMS) Codd 1970 . Initial efforts in this domain involved the fine-tuning of pre-trained LLMs for domain-specific learning objectives Dinh et al. 2022 Hu et al. 2022 Houlsby et al. 2019 Rebuffi et al. 2017 Wang et al. 2020 . Subsequently, the research community proposed In-Context Learning (ICL) Brown et al. 2020 Nam et al. 2023 Hegselmann et al. 2023 Lu et al. 2021 Min et al. 2023 , a paradigm wherein an LLM acquires new task capabilities solely through the processing of examples and instructions embedded within the input prompt (i.e., the "context"), thereby obviating the need for modifications to the LLM’s internal weights or neural network architecture. This inherent simplicity and cost-efficiency render ICL a highly accessible approach for executing classical machine learning tasks, such as classification and prediction, utilizing LLMs, particularly for users without prior specialized machine learning training. The core philosophy of applying LLMs to tabular data involves representing the table in a natural language format, effectively "teaching" the LLM to interpret it as a human would. This mimicry has led to observed improvements in classification and prediction tasks on tabular data Borisov et al. 2022 Hegselmann et al. 2023 Yan et al. 2024 . However, a fundamental question remains for enhancing user experience with ICL: How many demonstrations should be included in the prompt? The common practice, as noted in many papers, is to randomly select a small number of samples (e.g., two, three, or five) to construct these demonstrations. While seemingly straightforward, this question presents several challenges. For instance, the specific learning task (classification or prediction) may be unknown, and the nature of the target variable (discrete category or continuous numerical value) varies. Furthermore, considerations such as data quality, missing values, and the possibility of partial columns in end-user input when utilizing LLM services add further complexity. In this work, we try to tackle a small portion of above mentioned questions. To be specific, We formally define our problem as: Problem 1.1 . Given a pre-trained LLM and a tabular dataset, can we select few demonstrations automatically for the users with quality classification performance in general. While our ultimate aim is to address a general problem, we prioritize refining it into a demonstration selection problem . There are few challenges behind our defined problem: The impact of labels The research work Min et al. 2022 shows that the golden labels actually do not have significant impact to the classification performance. Unlike the classical machine learning, if the given dataset is separable in the feature vector space, a subset of randomly selected samples from each category can have the better performance than random samples after the machine learning model is trained, for example, support vector machine (SVM). Nevertheless, for the learning task we are discussing in this work, we don’t limit the user to only select one column as the label column. The users have the freedom to choose arbitrary column as the label column. Lack of theory support To our knowledge, there is no well proved theory about how to select good demonstrations from a given sample set for a given LLM. Most existing works Wang et al. 2023 Qin et al. 2023 Zebaze et al. 2024 are treating the LLMs as machine learning models, and employing the well established theory framework like Information theory Shannon 1948 , Solomonoff Induction Solomonoff 1964 , Minimum Description Length Grünwald 2007 or etc. With above mentioned problems and challenges, we presented a heuristic algorithm to select demonstrations automatically for the ICL over tabular data. We treat each row of the tabular data as a demonstration candidate, and estimate the data distribution of the candidates in the representation space of the LLM. To be specific, we use the token IDs as the representation of each demonstration, and use a sparse graph to represent the data distribution. With that, we use the Spectral Gap from the Spectral graph theory to estimate the number of clusters and use it as the minimum number of demonstrations. Our contribuions are summarized as follows: 1. We proposed a heuristic algorithm to select demonstrations automatically for ICL with tabular data, Especially for classification tasks. 2. Our algorithm is almost a parametric-free algorithm, except the construction of the sparse graph. 3. Our empirical studies show that our proposed algorithm can have resonable classification performance stablely on different LLMs and datasets. Figure 1 : An overview of our proposed algorithm. The prompt template and the LLM are chosen by the user. 2 Algorithm In this section, we present the details of our proposed algorithm. We first present the demonstration design for each row of the tabular data. Secondly, we calculate the similarity among different demonstrations and build a similarity graph. Thirdly, we use the spectral graph theory to estimate the number of clusters in the similarity graph, which is the number of demonstrations we need. Finally, we design a prompt template with the selected demonstrations for the final inference. 2.1 Demonstration Representation For in-context learning over the tabular data, we need to package the information held by each row of the table into the language that can be understood by the LLM. Without loss of generality, we follow the design from Borisov et al. 2022 Dinh et al. 2022 , and design our demonstration as follows: "input: given {column name} is {column value}, \dots , and {column name} is {column value}, class: {label}." An example illustration for the Iris dataset Fisher 1936 is: "input: given sepallength is 6.8 , sepalwidth is 3.0 , petallength is 5.5 , and petalwidth is 2.1 , class: Iris-virginica ." As we can see from the above example, we don’t explicit add modification to the original table data like improve the column names from "sepallength" to "sepal length" which is more human readable and meaningful. Similarity, we don’t add extra description to the table values like use "6.8 cm" instead of "6.8". In our proposed algorithm, the end-users always have the freedom to choose a better demonstration template based on their

Read the complete text