Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks

Marios Kokkodis, Richard Demsyn-Jones, Vijay Raghavan · 2025

Read Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks by Marios Kokkodis, Richard Demsyn-Jones, Vijay Raghavan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04277.txt.

Introduction

Marios Kokkodis, Richard Demsyn-Jones, and Vijay Raghavan compared two ways to sort home-service requests in 2025. Customers on Thumbtack describe a job in words and photos, and the system must pick the right trade, such as handyman or bathroom work. The team built classic models from text and image embeddings and tested them against large language model prompts.

The classic models won on accuracy, balance of scores, speed, and cost. Their accuracy lead over the best prompts was large, and it held for text alone, photos alone, and both together. The scores from the classic models were also better matched to real odds. Readers get a sober check on hype: for a fixed sorting task with good training data, a smaller trained model can beat a big general model.

Catalog record

CREATORMarios Kokkodis, Richard Demsyn-Jones, Vijay Raghavan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE7 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks
Marios Kokkodis, Richard Demsyn-Jones, Vijay Raghavan

Abstract
. Are traditional classification approaches irrelevant in this era of AI hype? We show that there are multiclass classification problems where predictive models holistically outperform LLM prompt-based frameworks. Given text and images from home-service project descriptions provided by Thumbtack customers, we build embeddings-based softmax models that predict the professional category (e.g., handyman, bathroom remodeling) associated with each problem description. We then compare against prompts that ask state-of-the-art LLM models to solve the same problem. We find that the embeddings approach outperforms the best LLM prompts in terms of accuracy, calibration, latency, and financial cost. In particular, the embeddings approach has 49.5% higher accuracy than the prompting approach, and its superiority is consistent across text-only, image-only, and text-image problem descriptions. Furthermore, it yields well-calibrated probabilities, which we later use as confidence signals to provide contextualized user experience during deployment. On the contrary, prompting scores are overly uninformative. Finally, the embeddings approach is 14 and 81 times faster than prompting in processing images and text respectively, while under realistic deployment assumptions, it can be up to 10 times cheaper. Based on these results, we deployed a variation of the embeddings approach, and through A/B testing we observed performance consistent with our offline analysis. Our study shows that for multiclass classification problems that can leverage proprietary datasets, an embeddings-based approach may yield unequivocally better results. Hence, scientists, practitioners, engineers, and business leaders can use our study to go beyond the hype and consider appropriate predictive models for their classification use cases.



1.
Introduction

AI hype is taking over the world: since the first version of OpenAI’s ChatGPT there has been an increased focus and excitement on the advances of generative AI technology. AI companies’ valuations have skyrocketed,
1
1

1

https://www.cnbc.com/2024/12/27/how-five-top-ceos-described-the-ai-boom-in-2024.html
accompanied by bold investments to push the boundaries towards Artificial General Intelligence. Big tech US companies are expected to invest a quarter of a trillion in AI innovation in 2025,
2
2

2

https://www.forbes.com/sites/bethkindig/2024/11/14/ai-spending-to-exceed-a-quarter-trillion-next-year/
more than ever before.

In this environment, many tech companies are constantly striving to come up with AI use cases that would allow them to maintain relevant investment targets but also increase revenue, improve user experience, and, in general, provide better products and services. Naturally, this creates a working environment where leadership, product managers, software engineers, and designers propose AI-based solutions, primarily inspired by their individual experiences with prompt engineering. Paraphrasing the ‘‘everyone is a programmer with AI’’ quote,
3
3

3

https://www.cnbc.com/2023/05/30/everyone-is-a-programmer-with-generative-ai-nvidia-ceo-.html
everyone can propose machine learning (ML) solutions after chatting with an LLM.

Classification problems (e.g., churn prediction
(
Pamina et al. 2019
)
, fraud prediction
(
Lokanan and Sharma 2022
)
, and spam prediction
(
Roy et al. 2020
)
) are some of the most common AI use-cases that can come up in business settings. Traditionally, these problems are solved with classifiers such as logistic regression, gradient boosting, and multilayer perceptrons. However, recently, with LLMs becoming more and more ubiquitous problem solvers, it is natural for many to think of and propose solutions that rely on prompt engineering
(
Wang et al. 2023
;
Zhu et al. 2023
)
.

2.
Background

A significant portion of relevant work has primarily focused on LLM applications and use cases such as API-calls
(
Sheng et al. 2024
)
, spelling correction
(
Dutta et al. 2024
)
, commentary on live sport and music events
(
Baughman et al. 2024
)
, building agents for real-world tasks
(
Fischer et al. 2024
)
, and detecting whether text was generated by AI
(
Abburi et al. 2023
)
.

In terms of prompting, some work focuses on label generation at scale
(
Wan et al. 2024
)
. For instance,
Xu et al. 2024
showed that LLMs have clear limitations identifying the correct label when none of the correct answers is provided as a label option.
Other prompting studies focus on providing frameworks for text classification. For example,
Wang et al. 2023
propose a prompt framework for improving zero-shot text classification, while
Zhu et al. 2023
focus on optimizing prompting techniques to increase performance in short-text classification tasks. These studies, however, do not directly compare how training traditional ML models would perform for these tasks, which is the main focus of our work.

Closely related to our work,
Al Faraby et al. 2024
showed that a random forest built on embeddings outperformed prompting by 22% at classifying educational questions, while
Gholamian et al. 2024
found the opposite result, showing that few-shot prompting in fact outperforms supervised approaches. In contrast to our work, both studies use public datasets. Even if the exact data was not used to train the evaluated LLMs, those are datasets that align with publicly held conventions and definitions (e.g., the conventional definition of a camera, a ball, etc.). On the contrary, most company classification problems rely on proprietary data that often diverge significantly from publicly defined concepts (e.g., what does it mean for a customer to churn, what are the thresholds that fraud becomes a problem, what kind of problems can a handyman solve, etc.). For these types of problems, a supervised approach can clearly learn the patterns of proprietary data that might not generalize to patterns found in public information used to train LLMs.

In addition, both studies are overly focused on accuracy. However, when deploying predictive classification models, accuracy is only one of the important dimensions–and arguably often not the most important one. Latency is of utmost importance since high latency can drive away customers
(
Basalla et al. 2021
)
, while cost is also part of the decision process. Finally, calibration is another factor that companies care about in settings where prediction confidence affects decision-making
(
Wang 2023
)

Read the complete text