Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data

Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani, Martin C.T. Manullang · 2026

Read Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data by Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani, Martin C.T. Manullang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25392.txt.

Introduction

Mayzaroh and three colleagues rate PyCaret AutoML next to IndoBERT tuning for binary sentiment analysis of Indonesian Twitter notes on Ibu Kota Nusantara, or IKN. The set holds 1,472 samples: 780 negative and 692 positive. Logistic Regression, Naive Bayes, and Support Vector Machine were tried with 10-fold cross-checks, with Logistic Regression best. The indobert-base-p1 model was tuned for five epochs, hitting 89.59 percent accuracy. IndoBERT beats the plain machine learning baselines by far, showing the worth of Transformer-based reads for casual Indonesian social media text.

Catalog record

CREATORMutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani, Martin C.T. Manullang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14919
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data Mutia Alfi Mayzaroh, Dwi Fitria Ningsih, Nindi Destriani, Martin C.T. Manullang Abstract Sentiment analysis on issues surrounding the development of Indonesia’s new capital city, Ibu Kota Nusantara (IKN), on social media plays an important role in understanding public opinion in Indonesia. This study presents a benchmark comparison between a Machine Learning (ML) approach based on AutoML with PyCaret and a Deep Learning (DL) approach based on IndoBERT fine-tuning for binary sentiment classification (positive/negative) on an Indonesian-language Twitter comment dataset related to IKN. The dataset consists of 1,472 samples (780 negative and 692 positive). In the ML approach, three algorithms were evaluated through 10-fold cross-validation: Logistic Regression (LR), Naive Bayes (NB), and Support Vector Machine (SVM). Logistic Regression achieved the best performance among the ML models with an average accuracy of 77.57% and an F1-score of 77.17%. Meanwhile, in the DL approach, fine-tuning the indobenchmark/indobert-base-p1 model using PyTorch for 5 epochs yielded a test set accuracy of 89.59% and an F1-score of 89.37%. The results show that IndoBERT significantly outperforms all ML models by a margin of more than 12 percentage points in accuracy, confirming the superiority of Transformer-based contextual representations in handling informal Indonesian-language text. Keywords Sentiment Analysis \cdot PyCaret \cdot IndoBERT \cdot AutoML \cdot Indonesian NLP \cdot IKN \cdot Transformer 1 Introduction Ibu Kota Nusantara (IKN) is Indonesia’s capital relocation project, which has become one of the most widely discussed topics on social media since its announcement. Various segments of society have expressed their views through platforms such as Twitter/X, generating a large and diverse volume of comments. Understanding the distribution of public sentiment toward this project holds strategic value for policymakers and researchers alike. In the field of Natural Language Processing (NLP), sentiment analysis has evolved rapidly from rule-based approaches to machine learning, and now to deep learning [ 1 ] . The availability of Indonesian pre-trained language models such as IndoBERT [ 2 ] opens opportunities to apply cutting-edge techniques to Indonesian, a language that is considered low-resource . This study contributes: (1) an Indonesian-language IKN sentiment dataset of 1,472 manually labeled samples; (2) a comprehensive benchmark of three ML algorithms (LR, NB, SVM) via PyCaret AutoML; (3) an IndoBERT fine-tuning implementation using PyTorch; and (4) a comparative analysis answering the question: does a Transformer-based DL approach outperform classical ML for sentiment analysis of informal Indonesian text? 2 Related Work Sentiment analysis on Indonesian-language social media text has been widely studied using various approaches. Alfina et al. [ 6 ] applied SVM and Naive Bayes for hate speech detection in Indonesian, finding that SVM outperformed using n-gram features. Riadi et al. [ 7 ] used Naive Bayes and SVM on an Indonesian Twitter dataset for political sentiment analysis, with SVM achieving 83% accuracy using TF-IDF features. In the context of AutoML, PyCaret provides an integrated framework that simplifies ML experimentation through automated preprocessing, model comparison, and hyperparameter tuning [ 8 ] , and has become increasingly popular for text classification tasks [ 11 ] . In deep learning, the Transformer architecture introduced by Vaswani et al. [ 5 ] revolutionized NLP through the self-attention mechanism. BERT [ 4 ] became the foundation for domain-specific pre-trained models, including IndoBERT [ 2 ] , which was trained on a large Indonesian corpus and has been shown to outperform ML baselines on Indonesian NLP tasks [ 3 ] . A comparative study of ML vs. DL on Indonesian text by Salsabila et al. [ 9 ] showed that BERT-based models consistently outperform traditional ML on e-commerce sentiment (F1 improvement of 10–15 points). Similar findings were reported by Kurniawan and Louvan [ 10 ] on news sentiment (IndoBERT F1-macro 89.3% vs. SVM 76.8%). Research on public understanding of government policy through social media is growing [ 12 ] , with the need for accurate Indonesian sentiment analysis becoming increasingly urgent amid the surge in social media content [ 13 ] . 3 Dataset 3.1 Dataset Description The dataset used in this study was sourced from Indonesian-language Twitter/X comments on the topic of Ibu Kota Nusantara (IKN). Data were collected through keyword-based searches related to IKN and manually labeled into two classes: positive and negative . 3.2 Dataset Statistics The final dataset after preprocessing consists of 1,472 samples with the following class distribution shown in Table 1 . Table 1: Class Distribution of the IKN Sentiment Dataset Class Number of Samples Percentage Negative 780 53.0% Positive 692 47.0% Total 1,472 100% Text length statistics (in characters) show a mean of 50.3 characters (std=38.5), a minimum of 2 characters, a median of 39 characters, and a maximum of 205 characters. This reflects the characteristic brevity and informality of social media comments. 3.3 Preprocessing The preprocessing steps include: 1. Lowercasing : All text is converted to lowercase. 2. Removal of missing values : Rows with NaN values are dropped. 3. Text length computation : An additional text_length feature is calculated. The cleaned data is saved as clean_data.csv for use in the subsequent modeling stage. 4 Methodology 4.1 Machine Learning Pipeline with PyCaret PyCaret version 3.x was used as the AutoML framework for machine learning experiments. The setup() function was configured with the text column as the text feature and label as the target, with session_id=42 for reproducibility. PyCaret automatically performs feature extraction using TF-IDF from the text column. Three ML algorithms were compared: 1. Logistic Regression (LR) : A linear model with L2 regularization, C=1.0 , lbfgs solver, and max_iter=1000 . 2. Naive Bayes (NB) : A probabilistic classifier based on Bayes’ theorem, well-suited for high-frequency text data. 3. Support Vector Machine (SVM) : SVM with a linear kernel for high-dimensional text classification. Evaluation was performed using 10-fold stratified cross-validation. The best model was then tuned using tune_model() with Optuna-based hyperparameter search (10 candidates, 10 folds), and finalized with finalize_model() . 4.2 Deep Learning Pipeline: IndoBERT Fine-Tuning For the DL approach, the indobenchmark/indobert-base-p1 model [ 2 ] was fine-tuned using PyTorch. IndoBERT is a BERT-based model with 12 Transformer layers, 768 hidden dimensions, 12 attention heads, and approximately 111 million parameters. Classification was performed by adding a linear layer ( classifier head ) on top of the [CLS] token representation. Hyperparameter configuration: • MAX_LEN = 128 tokens • BATCH_SIZE = 16 • EPOCHS = 5 • Learning Rate = 2\times 10^{-5} (AdamW, weight_decay=0.01) • Scheduler: linear warmup (10% of total steps), followed by linear decay • Gradient clipping: max_norm = 1.0 Data split: • Train: 1,030 samples (70%) • Validation: 221 samples (15%) • Test: 221 samples (15%) A stratified split was used to preserve the class distribution in each partition. The best model (based on highest val_acc) was saved during training. 4.3 AI Prompts Used In developing this pipeline, an AI assistant (Claude) was used to assist with code writing. The following are the main prompts used: Listing 1: Prompt for EDA and Preprocessing ⬇ " Create a Python notebook for EDA and preprocessing of a sentiment CSV dataset with ’ text ’ and ’ label ’ columns . Include : load data , check label distribution , compute text length statistics , lowercase text , drop NaN values

Read the complete text