Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection

Menna Elgabry, Ali Hamdi · 2025

Read Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection by Menna Elgabry, Ali Hamdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17630.txt.

Introduction

Menna Elgabry and Ali Hamdi show that a jury of small models can out-read big ones at spotting emotion in text. Their panel joins five compact, differently built language models, each fine-tuned for emotion classes. An old voting idea, Condorcet's jury theorem, guides the panel: many decent judges, wisely weighted, beat one star judge.

Each vote counts twice over, weighted by the model's past record and by how sure it sounds on this very text. On the DAIR-AI emotion set, the panel scored a macro F1 of 93.5 percent. That beat giant models with up to 7 billion parameters, even after those giants got task-specific tuning. The whole jury totals just 595 million parameters.

Catalog record

CREATORMenna Elgabry, Ali Hamdi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17630

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detection Menna Elgabry, Ali Hamdi Abstract This paper introduces a confidence-weighted, credibility-aware ensemble framework for text-based emotion detection, inspired by Condorcet’s Jury Theorem (CJT). Unlike conventional ensembles that often rely on homogeneous architectures, our approach combines architecturally diverse small transformer-based large language models (sLLMs)—BERT, RoBERTa, DistilBERT, DeBERTa, and ELECTRA—each fully fine-tuned for emotion classification. To preserve error diversity, we minimize parameter convergence while taking advantage of the unique biases of each model. A dual-weighted voting mechanism integrates both global credibility (validation F1- score) and local confidence (instance-level probability) to dynamically weight model contributions. Experiments on the DAIR-AI dataset demonstrate that our credibility-confidence ensemble achieves a macro F1-score of 93.5%, surpassing state-of-the-art benchmarks and significantly outperforming large-scale LLMs, including Falcon, Mistral, Qwen, and Phi, even after task-specific Low-Rank Adaptation (LoRA). With only 595M parameters in total, our small LLMs ensemble proves more parameter-efficient and robust than models up to 7B parameters, establishing that carefully designed ensembles of small, fine-tuned models can outperform much larger LLMs in specialized natural language processing (NLP) tasks such as emotion detection. Keywords: Emotion Detection, Small LLMs, Ensemble Learning, Weighted Voting, Condorcet’s Jury Theorem. 0.1 Introduction Building intelligent interactive systems that can understand and respond to human emotions is a remarkable goal in computer science. As processing systems advance in their use of NLP technologies, correctly identifying and responding to the emotional complexities of a given text is essential for effective mental health applications [ 8 ] , personalized recommendations [ 1 ] , and human-computer interactions [ 30 ] . Understanding this task is very difficult. Emotions cannot be understood through a simple framework of predetermined binary choices. Instead, they can be communicated implicitly, and they exist in varying amounts and may be represented in different forms, including irony, cultural references, and figurative language [ 24 ] . Such complexities are recognized in the DAIR-AI benchmark dataset [ 29 ] that struggle to achieve reliable classification due to the inherent imbalance of the classification targets. In text-based emotion detection (TED), the current approach involves the direct fine-tuning of large language models (LLMs) that have been previously trained on specific tasks. Although several models from the BERT family have been developed and deployed with success, their effectiveness has come to a standstill, especially with poorly distributed data. This limitation arises because a single model, no matter how sophisticated its architecture is, will always be biased toward the majority [ 16 ] . Consequently, interest has shifted toward the use of ensemble models because of their ability to combine the outputs of several models to deliver a more integrated and accurate classifier. An example of ensemble methods is the Condorcet’s Jury Theorem (CJT), which states that a more accurate decision can be derived from a collection of independent and competent models than from any single member of the collection [ 5 ] . However, the use of CJT in TED may not be optimal when Several ensembles are made of homogeneous models(multiple copies of the same architecture) for example, BERT fine-tuned with different seed initializations. The result of this is that the models reinforce each other and often make correlated errors, violating the independence assumption crucial to CJT’s success [ 21 ] . Moreover, Traditional aggregation by majority voting assumes that all models contribute equally to every prediction, which is clearly not the case. As, this equal treatment misses two important signals: (1) the overall reliability of a model, expressed in its evaluation metrics (e.g., F1-score on validation data), and (2) the confidence level for a prediction on a given case. Not considering these factors is likely to produce suboptimal ensemble decisions in tasks that require subtle discrimination, such as emotion classification. In this work, we introduce a novel ensemble framework to mitigate these limitations using a dual-weighted voting mechanism. We perform a fine-grained six-class emotion classification setup where, unlike standard majority voting, our approach employs a confidence and credibility weighted voting system, where each model’s contribution is based on its voting confidence on the instance level and it’s voting credibility on the dataset level. With this design, the ensemble can make predictions that are both certain and reliable. This leads to better predictions for a range of emotions. We illustrate the overall effectiveness of our ensemble, as it consistently outperforms LLMs and achieves results that are within 0.2% of a benchmark from a Q1 journal [ 4 ] . These findings conclusively show that the improvements observed are due to the enhanced reliability of the suggested weighted voting system 0.2 Background 0.2.1 Transformers Architectures for Text Classification The transformer architecture introduced by Vaswani et al. [ 32 ] revolutionized modern Natural Language Processing (NLP) due to its self-attention mechanism, which evaluates the contextual relationships among all words in a sequence, resulting in a comprehensive understanding of the text. Transformer models are generally classified according to their pretraining objectives. For example, encoder models such as BERT [ 9 ] are designed for natural language understanding tasks, as they generate bidirectional contextual representations, while decoder models such as GPT focus on text generation through auto regressive language modeling. On the other hand, encoder–decoder hybrids are optimized for sequence-to-sequence tasks such as translation and summarization, as demonstrated by models like T5 and BART [ 33 ] . In the context of Text-based Emotion Detection (TED)—a classification-focused problem—encoder-based transformers tend to yield superior results due to their ability to identify and interpret intricate semantic and contextual nuances [ 18 ] . This study employs a varied ensemble of transformer models, integrating encoder based models, and comparing against modern decoder based Large Language Models (LLMs). Encoder-Based Models: We utilize five members of the BERT family, chosen because of their proven efficiency and diverse architectures: BERT (Bidirectional Encoder Representations from Transformers) [ 9 ] : The foundational model, selected for its robust understanding of the bidirectional context. RoBERTa (A Robustly Optimized BERT Pretraining Approach) [ 23 ] : An optimized variant that removes the next-sentence prediction objective and incorporates more dynamic masking, frequently resulting in stronger performance in the General Language Understanding Evaluation (GLUE) and comparable evaluations. DistilBERT [ 28 ] : A distilled variant of BERT that has a 40% reduction in size and 60% increase in speed while retaining 97% of its language understanding capabilities. Its modified layer structure introduces a unique ”reasoning style” to the ensemble. DeBERTa (Decoding-enhanced BERT with Disentangled Attention) [ 17 ] : This new version improves BERT by incorporating a disentangled attention mechanism and an advanced masking decoder, resulting in improved efficacy in numerous natural language understanding (NLU) tasks. ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) [ 6 ] : This model uses a novel pre-training task called replaced token detection, which is more sample-efficient than BERT’s masked language modeling. Decoder-Based Models (LLMs): we also compare our results against several decoder only LLMs finetuned for classification: Phi [ 22 ] : A high performance open-source model with a focus on common sense reasoning and logical understanding. Falcon [ 26 ] : A suite of state-of-the-art LLMs trained in large and diverse corpora, known for their strong general capabilities. Qwen (Qianwen) [

Read the complete text