Smoothie : Label Free Language Model Routing

Neel Guha ∗, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré · 2024

Read Smoothie : Label Free Language Model Routing by Neel Guha ∗, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04692.txt.

Introduction

Neel Guha and colleagues present Smoothie in a 2024 paper on how to pick the best large language model for each new input. Different models do well on different tasks, so sending each item to the right model can lift results. Older ways to make that choice need labeled training data.

Smoothie needs no labels at all. It looks at the outputs from several models, builds a model of how those outputs relate, and scores each model for each input. In tests it often picked the best model and beat other routing methods by up to ten points in accuracy. The work offers a low-cost way to use many models together.

Catalog record

CREATORNeel Guha ∗, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Smoothie : Label Free Language Model Routing Neel Guha ∗, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré Abstract Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing ). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie , a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown “true” outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie ’s LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy. 1 Introduction Large language models (LLMs) are increasingly being deployed in multi-capability regimes where data inputs may span a diverse range of tasks, each of which requires different capabilities [ 8 ] . For instance, an LLM-powered chatbot may be asked to write code, answer questions about different domains, summarize documents, perform extraction, and more [ 3 , 14 , 30 , 8 ] . One challenge is that while engineers often have access to numerous pre-trained LLMs (i.e., through Huggingface or various APIs), they do not know which LLM is optimal for each possible user input [ 86 ] . Because the quality of generations can vary significantly between LLMs, choosing the right LLM for each input sample is important to ensure high task performance [ 41 ] . Recent work has explored various ways to utilize ensembles of pretrained LLMs in multi-capability settings, by (1) collecting a diverse pool of LLMs and (2) identifying which LLM to route each sample to [ 86 , 55 ] . However, most existing approaches require labeled data; Engineers typically either (1) train an auxiliary model using labeled data to rank or predict the LLM to which each sample should be routed [ 41 , 79 ] , or (2) directly use labeled data to determine which LLM is the best on average [ 86 ] . As a result, engineers designing routing protocols face the practical difficulty of constructing labeled datasets. Given a candidate pool of LLMs and an unlabeled test dataset, this paper explores how to best select LLM outputs for each sample in an entirely unsupervised manner—without labeled data, or models trained on labeled data. To make progress in addressing this question, we face two technical challenges: • Unknown LLM quality : The first challenge is estimating the quality of each LLM. Access to labeled data allows engineers to identify higher performing LLMs by measuring the accuracy/quality of LLM outputs. In this paper, we study the question of how to estimate quality without labeled validation data. • Sample-conditional generator performance : The second challenge is determining how to select the best LLM for each individual test sample. LLM outputs can vary in quality over different samples, which could render population-level estimates of LLM quality misleading. In this work, we propose Smoothie , a method for routing samples to LLMs in a label-free manner (Figure 1 ). Below, we describe how Smoothie addresses the two challenges described above. • Quality estimation : Using the LLM outputs for each test sample as “voters,” Smoothie estimates the quality of each generator using methods from Weak Supervision (WS). Concretely, Smoothie constructs a latent variable graphical model over observable LLM outputs and an unknown true output. By modeling the embedding vector difference between each LLM output and the true output as a multivariate Gaussian, we can derive a closed-form estimator adapted from [ 85 ] for learning LLM quality scores efficiently. • Conditioning : We condition these quality estimates to be particular to a given test sample by only using the nearest neighbors of a test sample in the training data as inputs to the estimator (i.e., kernel smoothing). We then route each test sample to the LLM with the highest quality score estimate on that sample. We call the version of Smoothie that produces quality estimates using all available test data Smoothie-Global , and we call the version that uses a sample’s nearest neighbors Smoothie-Local . Figure 1 : For a given input x , Smoothie estimates the quality of every LLM ensemble’s generation, and uses this quality weight to route x to a single LLM. We empirically evaluate Smoothie in three stages. • LLM selection: First, we assess Smoothie-Global ’s ability to identify—from an ensemble of mixed quality LLMs—the optimal LLM for a given task overall. On traditional generation tasks such as summarization, reading comprehension, and data-to-text generation, we find that Smoothie-Global ’s learned LLM quality-weights correlate with actual LLM performance ( \rho=0.72) ), and on the AlpacaEval benchmark, Smoothie-Global identifies the best-performing instruction model 70% of the time [ 50 ] . The highest quality LLM identified by Smoothie-Global —all computed without labeled data—can beat random-selection by up to 15 points win-rate on AlpacaEval, and by up to 8 points on SQuAD. • Routing: Second, we study whether Smoothie-Local ’s sample-conditional scoring mechanism allows it to route samples in mixed-task datasets to higher-performing LLMs (i.e., the multi-capability regime). We find that Smoothie-Local can improve the quality of produced generations by up to 7 points accuracy over Smoothie-Global , and that Smoothie-Local outperforms baseline unsupervised routing methods by up to 10 points accuracy and supervised routing methods by up to 5.0 points accuracy. • Prompt selection: Finally, we assess whether Smoothie ’s quality-estimation mechanism can be applied to select the optimal prompt template in a candidate pool while using a fixed LLM. We find that Smoothie-Global can outperform other prompt selection approaches by up to 18 points, allowing a 410M parameter model to match the performance of 6.9B parameter model. 2 Related Work We provide an abbreviated related work, with a full treatment in Appendix C . Routing Routing has been classically utilized in Mixture-of-Experts models [ 42 , 37 , 82 , 25 ] , which involve jointly training a set of models as well as a router. Recently, routing mechanisms have been used at inference time to decide which pre-trained LLM to use for a given sample [ 79 ] . Some approaches involve training an auxiliary model using labeled training data to either score or rank the performance of each LLM on each test sample [ 38 , 74 ] . Others do not involve training a model but instead use nearest neighbor methods, selecting the LLM that does the best on a test sample’s labeled neighbors [ 48 , 86 ] . In contrast, Smoothie does not require any labels. Ensembling Ensembling is another way of utilizing a pool of LLMs. Existing work has primarily focused on ensembling outputs for

Read the complete text