MarginSel : Max-Margin Demonstration Selection for LLMs

Rajeev Bhatt Ambati, James Lester, Shashank Srivastava, Snigdha Chaturvedi · 2025

Read MarginSel : Max-Margin Demonstration Selection for LLMs by Rajeev Bhatt Ambati, James Lester, Shashank Srivastava, Snigdha Chaturvedi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06699.txt.

Introduction

Rajeev Ambati and co-authors present MarginSel, a method for picking the example answers shown to a large language model when it learns a task from a prompt, in a 2025 paper. Few-shot prompting works well, but its quality swings with which examples are chosen and in what order. The team aims to pick the examples that teach the most.

Their two-step method chooses hard examples, tuned to each new question. Hard cases sit near the model's decision edge, much like the support points that define a margin in classic machine learning. By stressing those cases, the method widens the model's effective margin. Across classification tasks, scores rose by two to seven points over random example choice. The authors back the idea with both theory and tests.

Catalog record

CREATORRajeev Bhatt Ambati, James Lester, Shashank Srivastava, Snigdha Chaturvedi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06699

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MarginSel : Max-Margin Demonstration Selection for LLMs Rajeev Bhatt Ambati, James Lester, Shashank Srivastava, Snigdha Chaturvedi Abstract Large Language Models (LLMs) excel at few-shot learning via in-context learning (ICL). However, the effectiveness of ICL is often sensitive to the selection and ordering of demonstration examples Lu et al. 2022 ; Liu et al. 2022 . To address this, we present MarginSel : Max- Margin Demonstration Sel ection for LLMs, a two-step method that selects hard demonstration examples for the ICL prompt, adapting to each test instance. Our approach achieves 2-7% absolute improvement in F 1 F_{1} -score across classification tasks, compared to a random selection of examples. We also provide theoretical insights and empirical evidence showing that MarginSel induces max-margin behavior in LLMs by effectively increasing the margin for hard examples—analogous to support vectors—thereby shifting the decision boundary in a beneficial direction. 1 Introduction Large Language Models (LLMs) Brown et al. 2020 perform well on a wide variety of tasks through few-shot learning via in-context learning (ICL), where task demonstrations are provided within the prompt. However, ICL’s effectiveness is highly sensitive to the selection of demonstrations Lu et al. 2022 ; Liu et al. 2022 . Prior work addresses this by training explicit retrievers Lu et al. 2023 ; Wang et al. 2024 , which optimize demonstration selection using labeled data. However, these approaches are computationally costly and task-specific, requiring retraining for each new task or LLM. Some methods avoid training by relying on semantic similarity, yet they neglect the LLM’s inherent uncertainty near the decision boundary. This is a critical factor for tasks with fine-grained distinctions between classes. Figure 1: MarginSel identifies misclassified examples near the decision boundary (circled) for a two-class classification task (yellow circles and green stars). By including these high-margin examples in the ICL prompt, MarginSel shifts the decision boundary, improving predictions and inducing max-margin behavior in LLMs. To address these gaps, we propose MarginSel : Max-Margin Demonstration Selection , a novel method that dynamically selects demonstrations near the decision boundary to maximize the classifier margin and adapt to each test instance’s uncertainty. MarginSel operates in two steps: (1) an LLM is prompted to generate relevant candidate labels for all training and test instances, identifying regions of ambiguity; and (2) training examples with the same candidate labels as the test instance are selected as demonstrations. Drawing inspiration from Support Vector Machines, we posit that these carefully chosen examples act as analogous to support vectors , inducing max-margin behavior within the LLM. As illustrated in Figure 1 , incorporating ambiguous examples(dotted), which are likely to be boundary-proximal, drives a beneficial shift in the decision boundary. Empirical evaluations across four LLMs and three tasks (e.g., cognitive distortion detection, sentiment classification) show that MarginSel consistently outperforms standard prompting and kNN-ICL Liu et al. 2022 , achieving absolute F1-score gains of 2–7%. Furthermore, our theoretical analysis demonstrates that MarginSel examples induce a max-margin behavior within LLMs, a finding corroborated by empirical analysis showing significantly increased class separation compared to random selection. Our key contributions are: • MarginSel : Max-Margin Demonstration Selection , a novel method that self-guides demonstration selection using the LLM’s uncertainty signals. • Theoretical Foundation: A formal connection between MarginSel and max-margin optimization, showing its equivalence to SVM-like boundary adjustment in LLMs. • Empirical Validation: Robust improvements across model scales and tasks, with ablations highlighting the critical role of boundary-aware examples. 2 Related Work While in-context learning (ICL) has enabled LLMs to solve complex tasks using only a few examples Brown et al. 2020 , the choice and ordering of demonstration examples critically affect performance. For instance, Lu et al. 2022 showed that permuting the order of demonstrations can lead to significant output variations, underscoring the need for robust demonstration selection. Existing approaches for demonstration selection broadly fall into two categories: retriever-trained methods that optimize selection via learned models and retriever-free methods. Retriever-trained methods optimize example selection using task-specific labels. For example, Rubin et al. 2022 trains a retriever using contrastive learning to score input-output pairs, while Wang et al. 2024 uses a deep Q-network to balance label diversity and relevance. Similarly, Lu et al. 2023 trains a reinforcement learning (RL) agent to optimize demonstration ordering for downstream task accuracy. Although effective, these methods incur high computational costs and require retraining for each task or LLM, limiting their scalability. Retriever-free methods avoid retriever training. A widely adopted approach, kNN-ICL Liu et al. 2022 , retrieves examples based on semantic similarity in an embedding space (e.g., RoBERTa Liu et al. 2020 ). Other works, such as Levy et al. 2023 , prioritize diversity through clustering or determinantal point processes. While efficient, these methods fail to adapt to task-specific ambiguities. For instance, kNN-ICL often selects redundant or easily classified examples, providing limited signal for refining decision boundaries near regions of uncertainty. Our work MarginSel is also a retriever-free method that dynamically adapts demonstrations to each test instance’s ambiguity pattern. Unlike retriever-trained methods, MarginSel requires no task-specific optimization, and unlike existing retriever-free methods like kNN-ICL, it self-guides selection using the LLM’s uncertainty signals. It selects demonstration examples near the decision boundary that act as support vectors (Section 4.1 ). MarginSel thus bridges the gap between these two categories, offering both efficiency and adaptability. In our experiments, we compare with kNN-ICL, as it is a widely adopted retriever-free method and the most relevant baseline for our approach. Theoretically, we extend Dai et al. 2023 ’s duality between ICL and gradient descent to show that MarginSel induces max-margin behavior (Section 4.2 ), a property unaddressed by prior approaches. 3 Max-Margin Demonstration Selection For a classification task, given a test example 1 1 1 (.)^{t} denotes a variable whose datatype is token(s) x^{t}_{\text{test}} (a sequence of tokens) and a label set \{y^{t}_{c}\}_{c=1}^{C} , where C is the number of classes, the goal is to predict a label \hat{y}^{t} . In zero-shot learning, we predict \hat{y}^{t}=\arg\max_{y^{t}_{c}}P(y^{t}_{c}\mid x^{t}_{\text{test}},\mathcal{M}), for a given model \mathcal{M} . In in-context learning (ICL) using a set of demonstration examples \mathcal{D}=\{(x^{t}_{\text{demo},k},y^{t}_{\text{demo},k})\}_{k=1}^{N} where each x^{t}_{\text{demo},k} is the demonstration input and y^{t}_{\text{demo},k} is the corresponding demonstration label, we predict \hat{y}^{t}=\arg\max_{y^{t}_{c}}P(y^{t}_{c}\mid x^{t}_{\text{test}},\mathcal{D},\mathcal{M}). The goal of MarginSel is to identify the demonstration set \mathcal{D} such that it leads to the most accurate prediction \hat{y}^{t} for a test example x^{t}_{\text{test}} . Figure 2: MarginSel : In Step 1 (Candidate Label Assignment), the LLM is prompted to assign all relevant candidate labels, which are displayed in binary form (e.g., 10010 indicates that labels 1 and 4 are assigned). In Step 2 (Hard Example Selection), training examples with matching candidate labels (10010) are selected for ICL—these examples may belong to multiple classes (depicted by stars and diamonds are 2 out of 5 classes). MarginSel draws motivation from hard mining Shrivastava et al. 2016 , where misclassified examples are identified and incorporated into the training set. In hard mining, the gradients computed from examples near the decision boundary help update the model’s weights to correct the boundary. Inspired by this idea, MarginSel first identifies hard examples by prompting the LLM to assign all relevant candidate labels in a zero-shot manner (see Figure 2 ) and then selects training examples that have the same candidate labels. Our approach consists of two

Read the complete text