Benchmarking PyCaret AutoML Against BiLSTM for Fine-Grained Emotion Classification: A Comparative Study on 20-Class Emotion Detection
Arya Muda Siregar Haikal Fransisko Simbolon Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Arielva Simon Siahaan Luluk Muthoharoh Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Ardika Satria Martin C.T. Manullang Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] · 2026
Read Benchmarking PyCaret AutoML Against BiLSTM for Fine-Grained Emotion Classification: A Comparative Study on 20-Class Emotion Detection by Arya Muda Siregar Haikal Fransisko Simbolon Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Arielva Simon Siahaan Luluk Muthoharoh Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Ardika Satria Martin C.T. Manullang Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26310.txt.
Introduction
Siregar and five colleagues benchmarked PyCaret AutoML tools against BiLSTM models for sorting text into 20 emotion classes. On 79,595 English sentences, logistic regression, naive Bayes, and support vector machines ran on TF-IDF features, while BiLSTM, GRU, and a light Transformer ran as deep rivals. BiLSTM won overall at 89 percent accuracy, just ahead of the best classic model at about 88 percent. Sequence models caught emotional context better, though classic models stayed cheap and competitive.
Catalog record
| CREATOR | Arya Muda Siregar Haikal Fransisko Simbolon Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Arielva Simon Siahaan Luluk Muthoharoh Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Ardika Satria Martin C.T. Manullang Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14560 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26310
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking PyCaret AutoML Against BiLSTM for Fine-Grained Emotion Classification: A Comparative Study on 20-Class Emotion Detection Arya Muda Siregar Haikal Fransisko Simbolon Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Arielva Simon Siahaan Luluk Muthoharoh Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Ardika Satria Martin C.T. Manullang Institut Teknologi Sumatera Institut Teknologi Sumatera Lampung Selatan, Indonesia Lampung Selatan, Indonesia [email protected] [email protected] Abstract Fine-grained emotion classification—identifying specific emotional states such as happiness, anger, sadness, and fear—is a challenging yet important task in Natural Language Processing (NLP). This paper presents a benchmark comparison between classical Machine Learning (ML) approaches and a Deep Learning (DL) approach for multi-class emotion classification over 20 emotion categories. On the ML side, we evaluate Logistic Regression, Multinomial Naive Bayes, and Support Vector Machine (LinearSVC) using TF-IDF feature representations. On the DL side, we compare Bidirectional Long Short-Term Memory (BiLSTM), Gated Recurrent Unit (GRU), and a lightweight Transformer model implemented in PyTorch. Experiments are conducted on the 20-Emotion Text Classification Dataset comprising 79,595 English sentences. Results show that the BiLSTM model achieves the best overall performance with an accuracy of 89% and a weighted F1-score of 0.89, outperforming both the best ML model (SVM: 88.11%) and other DL variants. Our findings demonstrate that while traditional ML models remain competitive and computationally efficient, sequence-based deep learning models better capture contextual emotional cues in text. Keywords Emotion Classification \cdot Sentiment Analysis \cdot BiLSTM \cdot Support Vector Machine \cdot PyCaret \cdot Natural Language Processing \cdot Deep Learning 1 Introduction Emotion recognition in text is a fundamental challenge in NLP with broad applications ranging from mental health monitoring and customer feedback analysis to social media sentiment tracking. Unlike binary sentiment analysis (positive/negative), fine-grained emotion classification requires distinguishing between a larger and more nuanced set of emotional states, significantly increasing the complexity of the task [ 3 ] . Recent progress in NLP has produced two dominant paradigms for tackling text classification: traditional Machine Learning (ML) methods relying on handcrafted features, and Deep Learning (DL) approaches that learn representations directly from raw text. Classical ML methods such as Support Vector Machines and Logistic Regression have demonstrated strong performance on many text classification benchmarks, especially when paired with effective feature extraction techniques such as TF-IDF [ 5 ] . Meanwhile, deep learning architectures like Long Short-Term Memory networks [ 4 ] , Gated Recurrent Units [ 2 ] , and Transformer models [ 13 ] have shown remarkable ability to model sequential dependencies in text. Despite extensive research, there is limited direct comparison between these two paradigms on fine-grained emotion classification tasks with large class sets (20 classes). This paper aims to bridge this gap by conducting a systematic benchmark study that evaluates both ML and DL approaches on the same dataset under identical experimental conditions. The contributions of this work are as follows: • A systematic benchmark of three ML classifiers (Logistic Regression, Naive Bayes, SVM) using TF-IDF features for 20-class emotion classification. • A comparative evaluation of three DL architectures (BiLSTM, GRU, Transformer) implemented in PyTorch on the same dataset. • An analysis of the trade-offs between model performance, training time, and computational complexity across all six models. • Deployment of the best ML and DL models as interactive web applications on Hugging Face Spaces. 2 Related Work Early approaches to emotion recognition relied heavily on lexicon-based methods and handcrafted features [ 12 ] . These methods, while interpretable, often fail to generalize across domains. The introduction of machine learning methods, particularly Support Vector Machines, provided significant improvements in classification accuracy by learning discriminative boundaries in high-dimensional feature spaces [ 7 ] . The advent of deep learning marked a paradigm shift in NLP. Hochreiter and Schmidhuber [ 4 ] proposed LSTM networks capable of capturing long-range dependencies in sequential data—a critical property for understanding the contextual nuances of emotion in text. The Bidirectional LSTM [ 11 ] further improved upon this by processing sequences in both forward and backward directions, allowing the model to incorporate both past and future context simultaneously. Cho et al. [ 2 ] introduced the Gated Recurrent Unit as a simplified variant of LSTM, reducing computational overhead while retaining strong sequential modeling capability. The Transformer architecture [ 13 ] , based entirely on self-attention mechanisms, has since become the dominant paradigm in NLP, enabling parallelization and scalability. For emotion classification specifically, Saravia et al. [ 10 ] demonstrated the effectiveness of LSTM-based models on Twitter emotion data. Mohammad and Turney [ 6 ] contributed foundational work on crowdsourcing emotion annotations for NLP benchmarks. More recently, AutoML frameworks such as PyCaret [ 1 ] have democratized access to ML pipelines by automating model selection, hyperparameter tuning, and evaluation, making it easier to compare multiple classifiers on the same task. 3 Dataset 3.1 Description We use the 20-Emotion Text Classification Dataset , an English-language benchmark comprising 79,595 sentences labeled with one of 20 distinct emotion categories. The emotion labels include: happiness, sadness, anger, fear, love, surprise, disgust, trust, anticipation, anxiety, boredom, confusion, curiosity, excitement, guilt, loneliness, nostalgia, pride, relief, and shame . 3.2 Statistics Table 1: Dataset Statistics Property Value Total Samples 79,595 Number of Classes 20 Language English Task Multi-class Text Classification Average Sentence Length ~12 tokens 3.3 Preprocessing Prior to feature extraction and model training, we apply the following text preprocessing steps: 1. Lowercasing : All text is converted to lowercase to reduce vocabulary size. 2. Punctuation and special character removal : Non-alphanumeric characters are removed. 3. Tokenization : Text is split into individual tokens. 4. Stopword removal : Common English stopwords are removed for ML feature extraction. 5. Train/Validation/Test Split : The dataset is split into 80% training, 10% validation, and 10% test sets with stratified sampling to maintain class distribution. For ML models, TF-IDF (Term Frequency-Inverse Document Frequency) vectors are computed with a maximum vocabulary size of 50,000 terms. For DL models, we use a word embedding layer with an embedding dimension of 128 and a vocabulary size of 30,000. 4 Methodology 4.1 Machine Learning Pipeline We employ TF-IDF feature extraction to transform raw text into numerical feature vectors, which are then fed into three ML classifiers. We adopt the AutoML benchmark approach to compare multiple models on the same feature representation: Logistic Regression (LR) : A linear classifier optimized using the L-BFGS solver with L2 regularization. Used as the baseline model due to its simplicity and interpretability. Multinomial Naive Bayes (MNB) : A probabilistic classifier based on Bayes’ theorem, well-suited for text classification with TF-IDF features. Applied with Laplace smoothing ( \alpha=1.0 ). Support Vector Machine (LinearSVC) : A linear SVM optimized using the squared hinge loss with L2 regularization. Known for strong performance on high-dimensional text classification tasks [ 5 ] . 4.2 Deep Learning Architecture All DL models are implemented in PyTorch [ 8 ] with the following shared configuration: • Embedding dimension: 128 • Vocabulary size: 30,000 • Maximum sequence length: 128 tokens • Dropout rate: 0.3 • Optimizer: Adam with learning rate 0.001 • Batch size: 64 • Training epochs: 10 BiLSTM (Bidirectional Long Short-Term Memory) : Two stacked LSTM layers (hidden size = 256) processing input