When Do We Need LLMs? A Diagnostic for Language-Driven Bandits

Uljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi, Michael Cashmore, Manuela Veloso · 2026

Read When Do We Need LLMs? A Diagnostic for Language-Driven Bandits by Uljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi, Michael Cashmore, Manuela Veloso (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05859.txt.

Introduction

Berdica and five colleagues ask when language models are truly needed for contextual bandits mixing text and numbers. Their LLMP-UCB draws uncertainty from repeated language-model inference, yet light number bandits on text embeddings match or beat it at far lower cost. Embedding size tunes search versus payoff, and a shape-based diagnostic guides the choice between approaches.

Catalog record

CREATORUljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi, Michael Cashmore, Manuela Veloso
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54983
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05859

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Do We Need LLMs? A Diagnostic for Language-Driven Bandits Uljad Berdica, Fernando Acero, Anton Ipsen, Parisa Zehtabi, Michael Cashmore, Manuela Veloso Abstract We study Contextual Multi-Armed Bandits (CMABs) for non-episodic sequential decision making problems where the context includes both textual and numerical information (e.g., recommendation systems, dynamic portfolio adjustments, offer selection; all frequent problems in finance). While Large Language Models (LLMs) are increasingly applied to these settings, utilizing LLMs for reasoning at every decision step is computationally expensive and uncertainty estimates are difficult to obtain. To address this, we introduce LLMP-UCB, a bandit algorithm that derives uncertainty estimates from LLMs via repeated inference. However, our experiments demonstrate that lightweight numerical bandits operating on text embeddings (dense or Matryoshka) match or exceed the accuracy of LLM-based solutions at a fraction of their cost. We further show that embedding dimensionality is a practical lever on the exploration–exploitation balance, enabling cost–performance tradeoffs without prompt complexity. Finally, to guide practitioners, we propose a geometric diagnostic based on the arms’ embedding to decide when to use LLM-driven reasoning versus a lightweight numerical bandit. Our results provide a principled deployment framework for cost-effective, uncertainty-aware decision systems with broad applicability across AI use cases in financial services. 1 Introduction Contextual Multi-Armed Bandits (CMABs) learn over multiple rounds of independently sampled environment interactions where the learning agent picks an action given a context from the environment and receives a reward ( Lattimore & Szepesvári 2020 ; Sutton & Barto 2018 ) . These properties make them very useful for a large family of problems that do not follow state-based transitions dynamics and cannot be modeled as a Markov Decision Process (MDP). Such flexibility has allowed widespread application in numerous fields like recommendation systems ( Li et al. 2012 ) , hyperparameter tuning ( Li et al. 2018 ; Jackson et al. 2025 ) , portfolio management ( Ni et al. 2023 ; Miao 2025 ) . However, CMABs are confined to numerical settings and do not allow for the direct introduction of natural language into the learning process without using embedding models that map language into numerical vectors. This fundamental limitation restricts their application to problems where natural language or textual information is crucial like commercial banking services, market analysis and investment reports. Large Language Models (LLMs) have emerged as promising solution to directly applying language to decision-making problems without the numerical CMABs limitations ( Monea et al. 2025 ) . A bandit algorithm can be obtained with an LLM-based setup by appending past rounds of interaction --- consisting of a context, action and reward tuple --- to the language model’s prompt 1 1 1 We will henceforth use the term prompt to refer to the any information that is used an input to the decoder-based LLM, and exclusively use the term context for the contextual variables in the bandit setting. This is to disambiguate from the common use of the term context in the language modelling literature. . The number of rounds in the prompt is limited by the model’s inference capacity, a common feature of state-of-the-art LLMs. In these LLM-based CMABs, exploration is induced by randomly dropping out past rounds from the prompt and increasing the sampling temperature as demonstrated in recent works by Monea et al. 2025 and Sun et al. 2025 . Such methods require additional hyperparameter tuning and underutilize established CMAB literature ( Srinivas et al. 2009 ; Lu et al. 2010 ; Lattimore & Szepesvári 2020 ; Sutton & Barto 2018 ) . Our work seeks to reconcile the differences between recent LLM-based and well-established numerical approaches. Crucially, using LLMs in problems where a CMAB solution is expected to perform well requires reliable uncertainty estimates that LLMs are largely limited in providing ( Hager et al. 2025 ; Hobelsberger et al. 2025 ; Tomov et al. 2025 ; Ulmer et al. 2025 ) . This is in contrast to the large number of numerical bandit algorithms that directly handle uncertainty like Gaussian Process Upper Confidence (GP-UCB) and Thompson Sampling (TS) ( Lattimore & Szepesvári 2020 ) . To address this, we turn to the concepts behind LLM Processes ( Requeima et al. 2024 ) where the model is queried multiple times on the same target to obtain a distribution rather than a single point prediction. The intersection of robust CMABs ( Ni et al. 2023 ; Miao 2025 ) and LLM agents ( Batra et al. 2025 ) is particularly critical within the financial sector. CMABs are increasingly used to dynamically optimize portfolios ( Ni et al. 2023 ) or to adapt strategies based on sentiments extracted from news and social media ( Miao 2025 ) . While LLMs can enhance these solutions by directly processing natural language, they incur substantial deployment costs ( Li et al. 2023 ) and risk propagating inherent training biases if not carefully calibrated ( Zhi et al. 2025 ) . We directly address the trade-off between costly LLM-based methods and efficient numerical CMABs through our experiments. We evaluate recent approaches to incorporating language into decision making. We include datasets of both numerical and textual features, compare LLM-based approaches to the numerical ones and propose a new competitive method that enables uncertainty estimation for bandits using LLMs. Our motivation is to combine the language understanding of LLMs with the uncertainty estimates benefits of traditional bandit algorithms to ensure more reliable and flexible decision-making in challenging setting where more complex formalisms fail and both language and numerical information are required for optimal performance. To summarize, our contributions are: • A comprehensive evaluation of LLM and non-LLM based CMAB methods for decision-making problems where the context information contains both textual and numerical features; • Introducing LLM Process UCB bandits combining the benefits of uncertainty estimation used by numerical bandits and the useful semantic priors of LLMs; • Showing that numerical bandits maintain competitive performance when trained directly on language embeddings; • Empirically demonstrating that the dimensionality of the embeddings space directly affects exploration and can be used to predict the performance gains resulting from using LLMs. 2 Preliminaries In this section, we formally define the action space, different compositions of the context space and different types of reward functions. 2.1 Contextual Multi-Armed Bandits Context Space We combine numerical and textual information. At each round t\in[T] , the environment presents a context c_{t}=[x_{t},z_{t}]_{d+n} that is a stacking of a d -dimensional numerical context vector x_{t}\in\mathbb{R}^{d} and the n -dimensional textual information z_{t} (e.g., natural language description, categorical features); n can be arbitrarily large, as is the case for vocabulary size in natural language. We may use the embedding function \psi:\mathcal{Z}\rightarrow\mathbb{R}^{m} that maps textual information z_{t} from the text space \mathcal{Z} to a m -dimensional vector representation where m is the embedding size. This would lead to c_{t}=[x_{t},z_{t}]_{d+m} where m<<n . Action Space We use discrete action space \mathcal{A} with |\mathcal{A}|=K arms. Our objective is to learn a policy \pi:\mathcal{C}\rightarrow\mathcal{A} that maximizes the reward over T rounds R_{T}=\sum_{t=1}^{T}r_{t} . The reward is a mapping R:\mathcal{C}\times\mathcal{A}\rightarrow\mathbb{R} . For every pull t , r_{t}=f(c_{t},a_{t})+\epsilon_{t} where \epsilon_{t} is sampled from a \mathcal{N}(0,\sigma_{t}^{2}) . We use constant \sigma unless stated otherwise. 2.2 Embeddings Models Matryoshka Embeddings To use numerical bandits on traditionally language-based settings,

Read the complete text