FEET: A Framework for Evaluating Embedding Techniques
Simon A. Lee, John Lee, Jeffrey N. Chiang · 2024
Read FEET: A Framework for Evaluating Embedding Techniques by Simon A. Lee, John Lee, Jeffrey N. Chiang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01322.txt.
Description
Abstract In this study, we introduce FEET, a standardized protocol designed to guide the development and benchmarking of foundation models. While numerous benchmark datasets exist for evaluating these models, we propose a structured evaluation protocol across three distinct scenarios to gain a comprehensive understanding of their practical performance.
Catalog record
| CREATOR | Simon A. Lee, John Lee, Jeffrey N. Chiang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01322
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FEET: A Framework for Evaluating Embedding Techniques Simon A. Lee, John Lee, Jeffrey N. Chiang Abstract In this study, we introduce FEET, a standardized protocol designed to guide the development and benchmarking of foundation models. While numerous benchmark datasets exist for evaluating these models, we propose a structured evaluation protocol across three distinct scenarios to gain a comprehensive understanding of their practical performance. We define three primary use cases: frozen embeddings, few-shot embeddings, and fully fine-tuned embeddings. Each scenario is detailed and illustrated through two case studies: one in sentiment analysis and another in the medical domain, demonstrating how these evaluations provide a thorough assessment of foundation models’ effectiveness in research applications. We recommend this protocol as a standard for future research aimed at advancing representation learning models. 1 Introduction Foundation models, such as BERT ( Devlin, 2018 ) , GPT ( Radford, 2018 ) , and CLIP ( Radford et al., 2021 ) , have revolutionized the field of artificial intelligence. These models are trained on vast datasets using self-supervised methods ( Ericsson et al., 2022 ) , with little to no explicit supervision, enabling them to capture a wide range of knowledge. Once pre-trained, they can be fine-tuned for specific applications ( Bommasani et al., 2021 ) , making them highly versatile. Their success has led to widespread adoption across various domains, including biomedicine ( Wornow et al., 2023 ) , astronomy ( Lanusse et al., 2023 ) , and particle physics ( Birk et al., 2024 ) . The rapid proliferation of foundation models has spurred continuous development, with frequent updates shared on platforms like Hugging Face ( Wolf et al., 2019 ) . While existing benchmark datasets, such as MMLU ( Hendrycks et al., 2020 ) and GSM8k ( Cobbe et al., 2021 ) , measure the performance of these models on specific tasks, standardized protocols for evaluating and reporting their performance metrics remain underdeveloped. To address this gap, we introduce the Framework for Evaluating Embedding Techniques (FEET) , a comprehensive protocol that categorizes foundation model use cases into three types: frozen embeddings, few-shot embeddings, and fully fine-tuned embeddings. By systematically evaluating these categories, we aim to assess the adaptability and effectiveness of foundation models, providing a holistic view of their practical utility. We demonstrate the application of this evaluation protocol through two case studies: the first is a canonical example in natural language processing, focusing on sentiment analysis using the SST-2 (Stanford Sentiment Treebank 2) dataset ( Socher et al., 2013 ) . The second is a medical case study that builds on the work of Lee et al. (2024) ( Lee et al., 2024a ) , highlighting the relevance and importance of this approach based on our interesting findings. Why is this an important problem? Figure 1: A comparative analysis of the Claude Model, GPT, and Gemini across varying shot counts. The selection of shot numbers (0, 25, 4, 3, 10) appears arbitrary and inconsistent, raising concerns about potential cherry-picking to emphasize Claude as the state-of-the-art (SOTA) model. Rigorous benchmarking protocols are crucial as they establish performance standards and promote competition, which in turn drives advancements in machine learning and more broadly in science. A well-defined protocol facilitates reproducibility and provides a standardized framework for evaluating diverse models and approaches, allowing for consistent progress from one study to the next. Currently, benchmarks in foundation model research suffer from inadequate evaluation practices, creating uncertainties about what truly constitutes state-of-the-art (SOTA). These benchmarks often rely on numerous datasets, lack reproducibility ( Theis et al., 2015 ) , and span multiple tasks, which complicates comprehensive assessments of their generalizability and potential for future improvements. We propose that this protocol will address these gaps, enabling more thorough and standardized evaluations of model capabilities within the broader scientific community. To further motivate the problem, we examine an example in the wild (Figure 1 ), which compares Large Language Models (LLMs) on various benchmarking datasets. However, several key issues arise from this comparison. When the LLMs were evaluated on Grade School Math (GSM8K), the number of shots measured across models was inconsistent. There were subtle variations between 0-shot Chain of Thought (CoT) and 5-shot CoT, making it difficult to draw accurate comparisons. Another notable issue is the misalignment of shot counts across different datasets. Some datasets present state-of-the-art performance at zero shots, while others go as high as 25 shots. These variable shot numbers are not well-motivated, and we argue that such inconsistent reporting practices are detrimental to scientific progress. In a field already overwhelmed by the sheer volume of AI research, we urge researchers to avoid cherry-picking examples and to adopt standardized, transparent evaluation methodologies. 2 Background Foundation model embeddings can be categorized into three primary use cases: Frozen Embeddings, Few-shot Embeddings, and Fine-tuned Embeddings. While many studies introducing new foundation models explore different permutations of these embeddings, we argue that it is crucial to present all three types for a comprehensive evaluation. Doing so ensures a more thorough assessment of a model’s performance and is necessary to substantiate claims of achieving state-of-the-art results. 2.1 Frozen Embeddings Frozen embeddings refer to pre-trained feature embeddings that remain unchanged during the training of a new model. The term ‘frozen’ indicates that these embeddings retain their pre-trained parameter values without any further modification: \mathbf{E}_{\text{frozen}}=\mathbf{E}_{\text{initial}} . Prior to the rise of transformer-based foundation models, embedding techniques such as Word2Vec ( Church, 2017 ) , GloVe ( Pennington et al., 2014 ) , and fastText ( Joulin et al., 2016 ) were widely used as frozen embeddings in various applications. Frozen embeddings are particularly valuable in machine learning, especially in scenarios where the generality and robustness of pre-trained models are crucial. Studies such as ( Lee et al., 2024b ) demonstrate how frozen embeddings from open-source models can be applied across a wide range of clinical tasks without requiring any fine-tuning. By leveraging the extensive knowledge embedded in models trained on large, diverse datasets, researchers and practitioners can evaluate how well these pre-trained representations transfer to new tasks or domains without the need for additional training. 2.2 Few-Shot Embeddings Few-shot learning is a machine learning technique that enables models to learn from a limited number of labeled examples, which is particularly important in scenarios where extensive data collection is impractical, such as specialized fields or rare event detection ( Brown, 2020 ; Parnami and Lee, 2022 ; Lai et al., 2020 ) . It operates by distinguishing between a ‘support set’ S=\{(x_{i},y_{i})\}_{i=1}^{k} , which contains a few labeled examples for training, and a ‘query set’ Q=\{x^{\prime}_{j}\}_{j=1}^{m} , which consists of samples for classification. The goal is to train a model, represented by f and parameterized by \theta , to predict the labels of the query set by optimizing a similarity function: y^{\prime}_{j}=\arg\max_{y}\text{sim}(f_{\theta}(x^{\prime}_{j}),f_{\theta}(x_{i}))\quad\forall(x_{i},y_{i})\in S Foundation models significantly enhance the effectiveness of few-shot learning by leveraging their deep architectures and self-attention mechanisms ( Vaswani, 2017 ) to transfer learned features with minimal data ( Wang et al., 2020 ) . These transferable embeddings closely align with the concept of transfer learning ( Weiss et al., 2016 ; Torrey and Shavlik, 2010 ) . However, the current literature on few-shot learning often highlights inconsistencies in evaluation practices,