In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

Yifei DUAN, Liu LI, Zirui ZHAI, Jinxia YAO · 2024

Read In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning by Yifei DUAN, Liu LI, Zirui ZHAI, Jinxia YAO (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13243.txt.

Introduction

Yifei Duan, Liu Li, Zirui Zhai, and Jinxia Yao distill in context learning into a small model for few shot tasks in 2024. Prompt based learning keeps big models costly to run. Their method trains a small model to absorb what the big one does with examples in the prompt. The test task was language inference.

Model size fell from over a billion weights to a tenth of that. Storage shrank ten fold as well. Accuracy on new domains rose near half over prompt use on like sized models. Memory use fell by up to sixty percent. Gains also beat standard pattern tuning. The authors pitch the trick for cheap few shot deployment.

Catalog record

CREATORYifei DUAN, Liu LI, Zirui ZHAI, Jinxia YAO
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning Yifei DUAN, Liu LI, Zirui ZHAI, Jinxia YAO Abstract We applied few-shot in-context learning on the OPT-1.3B model for the natural language inference task and employed knowledge distillation to internalize the context information, reducing model parameter from 1.3B to 125M and achieving a size reduction from 2.5GB to 0.25GB. Compared to using in-context learning alone on similarly sized models, this context distillation approach achieved a nearly 50% improvement in out-of-domain accuracy, demonstrating superior knowledge transfer capabilities over prompt-based methods. Furthermore, this approach reduced memory consumption by up to 60% while delivering a 20% improvement in out-of-domain accuracy compared to conventional pattern-based fine-tuning. 1 Introduction/Background/Motivation Large Language Models (LLMs) have revolutionized natural language processing, demonstrating remarkable capabilities in various tasks. However, their deployment faces significant challenges, particularly with regard to memory requirements during training and inference. In-context learning (ICL) [ 3 ] , while effective, requires substantial computational resources due to the need to maintain long context windows. This limitation becomes particularly severe when dealing with tasks that require extensive context processing, such as long-form dialogue or document analysis. The core challenge lies in the trade-off between model performance and resource efficiency. Current approaches require extensive computational resources (full fine-tuning) [ 2 ] or significant memory overhead (ICL). This creates a barrier to training, inference and deployment in resource-constrained environments and limits the practical applicability of LLMs in real-world scenarios. Few-shot learning is used for task adaptation by enabling the model to generalize to specific tasks with minimal labeled data. This approach leverages a small number of training examples, which makes it particularly valuable for scenarios where obtaining large-scale annotated datasets is impractical. Conventional solutions to few-shot learning generally fall into two categories: weights-updating fine-tuning and prompt-based context learning. Each approach has significant limitations, particularly when scaling to larger models or deploying in resource-constrained environments. Fine-tuning requires updating some or all model parameters, leading to high computational costs and potential catastrophic forgetting. In-Context Learning demands large memory allocations for context windows, limiting scalability. In this paper, we implemented Context Distillation (CD) [ 4 ] , a novel approach that combines the benefits of fine-tuning and in-context learning while minimizing their respective drawbacks. Our key contributions include: (1) A novel context distillation methodology that efficiently captures prompt information in model parameters. (2) Integration with parameter-efficient fine-tuning techniques (BitFit [ 6 ] and LoRA [ 1 ] ). (3) Comprehensive experiment designs to study the impact of critical hyper-parameters and model sizes horizontally and vertically. This approach provides a resource-efficient solution that balances performance with hardware constraints, enabling the deployment of advanced LLM capabilities, particularly for on-device applications. The Multi-Genre Natural Language Inference (MNLI) corpus [ 5 ] is used for all training and evaluation in this paper. It contains sentence pairs labeled as entailment, contradiction, or neutral, designed to train and assess models on natural language inference tasks. The dataset includes separate training, development, and testing sets, divided into matched (in-domain) and mismatched (cross-domain) subsets to evaluate generalization. Each data point consists of a ”premise” sentence, a ”hypothesis” sentence, and a label indicating their relationship. 2 Approach 2.1 implementation The hardware setup for this experiment includes NVIDIA T4, A100, and RTX 3090 GPUs. The T4 GPU, with at least 16 GB of RAM, is sufficient for performing inference on models up to 1.3 billion parameters in half precision, supporting up to 12 support examples per evaluation and a training size of 20 in this study. The implementation is developed using PyTorch and Python 3.9, primarily on the RTX 3090 GPU. Most evaluations and analyses are conducted on cloud-based A100 GPUs to ensure consistency in training time and memory usage, eliminating hardware variability as a factor in the results. The MNLI dataset is first refined into a binary classification task by removing instances labeled as neutral. For efficient processing, the dataset is reduced to a pool of 3,000 examples, from which support cases are randomly selected while ensuring the query case is excluded from the support set. Experiments show that varying the pool size to 1,000 or 10,000 examples has negligible impact on the results, demonstrating the robustness of the approach to pool size variations. This is expected as the few-shot training data is limited to 20 examples, regardless of the size of the pool they are drawn from. The methods ICL, PBFT, and CD require prompts structured with three components: prefix, support, and query. The support component includes 1 to 12 examples. A sample prompt structure is provided in Table 1 . Few-shot fine-tuning leverages contextual embeddings during the model training process, employing the AdamW optimizer with a learning rate of 1e-5 and a 40-epoch schedule. Evaluations are performed using in-domain and out-of-domain accuracy, along with peak memory usage. GPU memory utilization is tracked with utility tools such as ‘torch_peak_allocated_memory’, and computational efficiency is measured using ‘timer’ over 100 cases per evaluation. While ‘timer’ may not fully account for asynchronous GPU operations, its error margin of less than 1 second is deemed acceptable for this study. Results are benchmarked against the pre-trained model without fine-tuning, providing insights into the effectiveness and efficiency of context distillation in enhancing model performance. To adapt the large language model (LLM) for binary classification, a verbalizer was utilized to map the model’s output logits to interpretable labels, specifically using ”yes” for entailment and ”no” for contradiction. This simplified label space enhanced decision-making while leveraging the pre-trained model’s ability to assign meaningful probabilities to natural language tokens. During evaluation, the verbalizer mapping was consistently applied across both in-domain (MNLI-matched) and out-of-domain (MNLI-mismatched) validation splits, enabling a comprehensive assessment of model performance. Label Content Prefix Determine if the premise entails the hypothesis. Answer with yes or no. Support Premise: It was a steep learning curve for me, she said. Hypothesis: She faced no difficulty with the task. Answer: No Query Premise: I’ll listen and agree with what I think sounds right. Hypothesis: I won’t even bother listening. Answer: Prediction Yes (based on the logit outputs of verbalizers) Table 1: Sample prompt with one support and one query for pattern-based fine-tuning and context distillation. The table reproduces the structure of the prompt used in the experiment. To ensure efficient and consistent fine-tuning, we independently implemented two parameter-efficient techniques: LoRA and BitFit. LoRA used a low-rank matrix rank ( r ) of 8, a scaling factor ( lora_alpha ) of 32, and targeted the q_proj and v_proj layers with a dropout rate ( lora_dropout ) of 0.05. This setup was optimized for sequence classification tasks ( task_type="SEQ_CLS" ) and allowed approximately 23% of the model’s parameters to remain trainable. BitFit, on the other hand, fine-tuned only the bias parameters, freezing all other weights. In the investigated cases, less than 0.1% of the parameters are trainable for BitFit. 2.2 Few-shot fine-tuning overfitting Few-shot fine-tuning is particularly prone to overfitting, as observed in our experiments, where increasing the number of epochs leads to a decline in out-of-domain accuracy. To counter this, hyperparameters such as learning rate and batch size were carefully optimized to ensure controlled parameter updates, reducing the model’s susceptibility to noise in the limited data. Learning rates were tested systematically within the range of

Read the complete text