CARGO: A Framework for Confidence-Aware Routing of Large Language Models
Amine Barrak 1 , Yosr Fourati 1 3 , Michael Olchawa 1 , Emna Ksontini 2 , Khalil Zoghlami 1 · 2025
Read CARGO: A Framework for Confidence-Aware Routing of Large Language Models by Amine Barrak 1 , Yosr Fourati 1 3 , Michael Olchawa 1 , Emna Ksontini 2 , Khalil Zoghlami 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14899.txt.
Introduction
Amine Barrak and four co-authors present CARGO, a framework for sending each user prompt to the most suitable large language model, in a 2025 paper. Models now differ widely in skill, speed, and cost, so choosing well matters. CARGO aims to balance answer quality against price without human-labeled training data.
A single embedding-based regressor, trained on pairwise comparisons judged by a language model, predicts how each candidate model will perform on a prompt. When the prediction looks uncertain, an optional binary classifier steps in for a second opinion. Category-specific regressors handle five task groups, from mathematics and coding to creative writing. Tests across four leading models show strong routing accuracy and high win rates against fixed choices, at a lightweight running cost.
Catalog record
| CREATOR | Amine Barrak 1 , Yosr Fourati 1 3 , Michael Olchawa 1 , Emna Ksontini 2 , Khalil Zoghlami 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CARGO: A Framework for Confidence-Aware Routing of Large Language Models Amine Barrak 1 , Yosr Fourati 1 3 , Michael Olchawa 1 , Emna Ksontini 2 , Khalil Zoghlami 1 Abstract As large language models (LLMs) proliferate in scale, specialization, and latency profiles, the challenge of routing user prompts to the most appropriate model has become increasingly critical for balancing performance and cost. We introduce CARGO ( Category-Aware Routing with Gap-based Optimisation ), a lightweight, confidence-aware framework for dynamic LLM selection. CARGO employs a single embedding-based regressor trained on LLM-judged pairwise comparisons to predict model performance, with an optional binary classifier invoked when predictions are uncertain. This two-stage design enables precise, cost-aware routing without the need for human-annotated supervision. To capture domain-specific behavior, CARGO also supports category-specific regressors trained across five task groups: mathematics, coding, reasoning, summarization, and creative writing. Evaluated on four competitive LLMs (GPT-4o, Claude 3.5 Sonnet, DeepSeek V3, and Perplexity Sonar), CARGO achieves a top-1 routing accuracy of 76.4% and win rates ranging from 72% to 89% against individual experts. These results demonstrate that confidence-guided, lightweight routing can achieve expert-level performance with minimal overhead, offering a practical solution for real-world, multi-model LLM deployments. I Introduction Large language models (LLMs) have advanced significantly in recent years, driven by improvements in model architectures, training methodologies, and large-scale data availability [ 1 ] . These developments have expanded their capabilities beyond traditional natural language processing (NLP) tasks, allowing complex reasoning, code generation, knowledge retrieval, and multimodal understanding [ 2 ] . Modern LLMs incorporate billions of parameters, self-attention mechanisms, and reinforcement learning to enhance response quality based on human feedback [ 3 ] . However, LLMs are optimized for different objectives, some excel in efficiency and factual accuracy, while others prioritize creativity and depth of reasoning [ 4 ] . For instance, models tailored for coding tasks emphasize precise syntax generation and debugging assistance [ 5 ] , whereas those optimized for creative writing focus on coherence and stylistic variation [ 6 ] . These differences arise from variations in training data, alignment techniques, and external knowledge integration. As LLMs are deployed across diverse applications, users submit queries that span domains such as mathematics, coding, creative writing, and business analysis. Each domain imposes distinct requirements on reasoning abilities, factual accuracy, and language fluency [ 7 , 8 ] . No single model consistently outperforms others in all categories due to differences in optimization strategies and architectural design. A model adept at numerical reasoning may struggle to generate coherent narratives, while a model fine-tuned for conversational fluency may lack precision in coding-related tasks [ 4 , 9 ] . Moreover, within the same domain, task complexity varies significantly. A business-related query can involve simple data retrieval or complex financial forecasting, each requiring different levels of reasoning and contextual understanding [ 10 ] . Similarly, coding tasks range from syntax corrections to large-scale algorithm optimization, where different models excel in specific areas [ 11 ] . Figure 1 : Overview of CARGO. A single embedding regressor scores all candidate LLMs; if the top–two scores are within threshold \tau , a classifier selects the better one. Most deployments address this challenge by maintaining a portfolio of LLM experts and then routing each query to whichever model promises the best quality [ 12 ] . However, contemporary routing frameworks exhibit three limitations that undermine their practical utility. (1) Latency overhead . Some routing frameworks employ sequential cascade strategies: prompts are first processed by small models and, if confidence is insufficient, passed step-by-step to larger ones. This incremental procedure causes latency to accumulate, pushing the worst-case response time toward the sum of all model calls [ 13 ] . (2) Limited expert models diversity . Many cost-aware systems reduce routing to a binary choice between one “small” and one “large” model, overlooking the mid-sized, domain-specialist experts that are increasingly common in production environments [ 14 ] . (3) High supervision and scalability constraints Routers still rely on thousands of labeled prompt–response pairs or reward scores for calibration. Because the cost of label acquisition grows with both the breadth of domains and the size of the expert pool, every expansion or task shift triggers an expensive retraining cycle, hindering operational scalability [ 12 , 14 ] . Recent benchmarking, further show that existing routing mechanisms, have difficulty generalizing to complex tasks and up-to-date models [ 15 ] . Motivated by these shortcomings, this paper introduces a confidence-aware routing framework (CARGO) structured around three complementary steps, each carefully designed to resolve key gaps identified in existing approaches. In the first stage, we avoid the common dependence on costly human-annotated data by using candidate LLMs themselves to evaluate response quality. This idea is supported by recent studies demonstrating that LLM-generated assessments closely match human judgements, especially in complex reasoning and knowledge-rich contexts [ 16 , 17 ] . The second stage involves a lightweight embedding-based regression model. This model simultaneously ranks all candidate experts in a single inference pass. Finally, if the two best candidates receive close regression scores, a refinement step invokes a binary classifier trained on pair-wise preferences to break the tie. This overall process is illustrated in Figure 1 . We provide the source code as a fully reproducible framework to support future research 1 1 1 https://sites.google.com/view/cascon2025 . This work is guided by the following research questions: • RQ1 (Individual Model Performance): How do selected expert LLMs (GPT-4o, Claude 3.5 Sonnet, DeepSeek V3, Perplexity Sonar) individually perform on our curated five-domain dataset? • RQ2 (Comparative Routing Efficacy): How effectively does our confidence-aware routing framework select optimal LLMs compared to individual expert models? • RQ3 (Category-Specific Routing Behavior): How does routing performance vary across task categories, and does category-specific modeling further enhance accuracy? II Related work The rapid proliferation of large language models (LLMs), accelerated by Transformer architectures, has significantly expanded accessibility but introduced substantial complexity in model selection, with platforms like Hugging Face now hosting over 200,000 models [ 18 ] . As no single model is universally optimal for all tasks [ 4 ] , end-users face considerable difficulty identifying the best fit among diverse architectures, training data, and intended applications. The distinct strengths and weaknesses exhibited by each model necessitate intelligent selection strategies to maximize performance. Large Language Models (LLMs) often excel in specialized domains when adapted or fine-tuned for specific tasks. For instance, OpenAI’s Codex, fine-tuned for programming, solves 28.8% of HumanEval coding benchmark problems in a single attempt, whereas the base GPT-3 model solves nearly none [ 19 ] . Similarly, Google’s Minerva, trained on mathematical content, achieves state-of-the-art accuracy on STEM question-answering tasks [ 20 ] , surpassing general LLMs in quantitative reasoning. Instruction-tuned models like InstructGPT [ 21 ] further highlight the impact of domain-specific fine-tuning, with a 1.3-billion-parameter variant preferred by human evaluators over the original 175-billion-parameter GPT-3 across diverse prompts. In other words, no single model performs best across all tasks. Instead, performance depends on training strategies, reasoning capabilities, and task alignment, meaning a model’s effectiveness relies on how it was trained, its ability to reason, and how well it fits the requirements of a specific task. Given the diverse strengths of individual LLMs across tasks,