CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification VII Concurso de Teses, Dissertações e Trabalhos de Graduação em SI XXI Simpósio Brasileiro de Sistemas de Informação
Washington Cunha 1, Leonardo Rocha 2 (Co-advisor), Marcos Gonçalves 1 (Advisor) · 2025
Read CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification VII Concurso de Teses, Dissertações e Trabalhos de Graduação em SI XXI Simpósio Brasileiro de Sistemas de Informação by Washington Cunha 1, Leonardo Rocha 2 (Co-advisor), Marcos Gonçalves 1 (Advisor) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07169.txt.
Introduction
Washington Cunha presents a doctoral dissertation on shrinking training data for text classification, summarized in a paper from 2025. Modern language tech follows a rule of more: more data, more compute, more cost. Instance selection fights that trend by cutting training sets down to their useful core.
The method drops noisy or redundant examples while keeping model quality. Cunha compares many selection methods in a broad, careful way for text tasks. Smaller training sets mean cheaper, faster fine-tuning of large models. The dissertation gives the technique its first deep treatment for natural language work. It suits teams that lack giant compute budgets. The summary appeared at a Brazilian systems symposium.
Catalog record
| CREATOR | Washington Cunha 1, Leonardo Rocha 2 (Co-advisor), Marcos Gonçalves 1 (Advisor) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification VII Concurso de Teses, Dissertações e Trabalhos de Graduação em SI XXI Simpósio Brasileiro de Sistemas de Informação Washington Cunha 1, Leonardo Rocha 2 (Co-advisor), Marcos Gonçalves 1 (Advisor) Abstract Resumo Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. This Ph.D. dissertation focuses on an under-investigated NLP data engineering technique, whose potential is enormous in the current scenario known as Instance Selection (IS). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining the effectiveness of the trained models and reducing the training process cost. We provide a comprehensive and scientifically sound comparison of IS methods applied to an essential NLP task – Automatic Text Classification (ATC), considering several classification solutions and many datasets. Our findings reveal a significant untapped potential for IS solutions. We also propose two novel IS solutions that are noise-oriented and redundancy-aware, specifically designed for large datasets and transformer architectures. Our final solution achieved an average reduction of 41% in training sets, while maintaining the same levels of effectiveness in all datasets. Importantly, our solutions demonstrated speedup improvements of 1.67x (up to 2.46x), making them scalable for datasets with hundreds of thousands of documents. Student Level : Ph.D., concluded on August 26th, 2024 Board Members : Franco Maria Nardini (ISTI-CNR); Thierson Couto Rosa (UFG); Rodrygo Luis Teodoro Santos (DCC-UFMG); Anisio Mendes Lacerda (DCC-UFMG) Dissertation available at : http://hdl.handle.net/1843/76441 Publications available at : http://bit.ly/3WvX1T5 1 Introduction The exponential growth in data availability has made it challenging to organize and retrieve content effectively. In this context, Automatic Text Classification ( ATC ) offers a solution by mapping textual documents (e.g., web pages, emails, reviews, tweets, social media messages) into predefined categories of interest. Indeed, ATC models are crucial for emerging applications [ Cunha et al. 2023a ] such as fake news detection, hate speech identification, relevance feedback, sentiment analysis, product review analysis, election vote inference, and assessing public agency satisfaction. As a supervised task, ATC benefits from applications generating large volumes of labeled data, such as social networks like Twitter. Moreover, crowdsourcing [ Sindagi et al. 2020 ] and soft labeling [ Roy and Cambria 2022 ] further reduce the costs of acquiring labeled datasets. Transformer-based architectures, including small and large language models (SLMs and LLMs) such as RoBERTa, GPT-4, and LLama3, represent the state-of-the-art (SOTA) in ATC, achieving outstanding results through pre-training on large datasets and fine-tuning on domain-specific tasks. While zero-shot capabilities are feasible, fine-tuning remains essential for optimal performance, leveraging pre-trained weights for task-specific adaptations [ de Andrade et al. 2023 ] . According to Andrew Ng, the success of these (language) models is due to extensive pre-training on massive datasets (e.g., 45TB for GPT-3) and the adaptability of pre-trained models via fine-tuning. This approach enables faster task-specific training compared to starting from scratch [ Uppaal et al. 2023 ] . However, fine-tuning remains resource-intensive despite being faster than full training, requiring significant computational power and time. For instance, fine-tuning XLNet (one of the datasets considered in our dissertation) on the MEDLINE dataset took 80 GPU hours. Therefore, practical constraints often delay re-training, impacting model performance in scenarios requiring continuous updates, such as fraud detection and recommendation systems. Indeed, resource limitations in companies and research groups restrict experimentation with deep learning models, which require extensive computational resources. For example, in this dissertation, we ran 4,000 experiments that took approximately 5,600 hours. Reducing financial, computational, and environmental costs is crucial, as shown by the significant energy consumption and carbon emissions associated with large models. Given increasing data volumes, re-training demands, and environmental concerns, proposing scalable and cost-effective strategies is essential. These include creating efficient deep learning algorithms, using advanced hardware, or improving data preprocessing techniques. This dissertation focuses on the latter, aiming to enhance model performance while reducing costs. In [ Cunha et al. 2020 ] , we proposed exploiting a set of preprocessing techniques in a data transformation pipeline for building cost-effective models. That solution improved effectiveness (e.g., models with higher accuracy) at a much lower cost (e.g., shorter time for ATC model construction). One of the main contributions of this work was the explicit incorporation into the pipeline of an Instance Selection (IS) stage, a promising set of techniques and growing research area that helps to deal with many of the aforementioned issues. Specifically, in contrast to traditional Feature Selection approaches [ Lu et al. 2017 ] , in which the main objective is to select the most informative terms (words), Instance Selection methods are focused on selecting the most representative instances (documents) for the training set [ Garcia et al. 2012 ] . The intuition behind this type of algorithm is to remove potentially noisy or redundant instances from the original training set and improve performance in terms of total time training time while keeping or even improving effectiveness. More specifically, IS methods have three main goals: (i) to reduce the number of instances by selecting the most representative ones; (ii) to maintain (or even improve) effectiveness by removing noise and redundancy; and (iii) to reduce the total time for applying an end-to-end model (which includes from traditional preprocessing steps to the model training step). By selecting the most representative instances, IS methods can also potentially remove noise from erroneous annotations. According to these objectives, IS methods must respect three fundamental constraints consisting of reducing the amount of training without loss of effectiveness and with efficiency gains . IS methods seek to optimize these three constraints simultaneously. Despite their potential, IS approaches have been understudied in ATC, particularly in deep learning [ Barigou 2018 , Tsai et al. 2014 ] . Traditional IS methods developed for other domains include CNN [ Hart 1968 ] , ENN [ Wilson 1972 ] , and Drop3 [ Wilson and Martinez 2000 ] . Recent methods, such as LSSm , LSBo [ Leyva et al. 2015 ] , and PSDSP [ Carbonera and Abel 2018 ] , have mainly been tested on small tabular datasets with weak classifiers such as KNN. In contrast, text classification datasets are unstructured, larger, and more complex, featuring high dimensionality and skewness. Given the high computational costs of deep learning transformer models with large training datasets, presenting an ideal scenario for applying IS techniques. 2 Hypothesis and Research Questions The main hypothesis (H1) of this Ph.D. dissertation is: H1: It is possible to simultaneously reduce data, maintain model quality, and improve time for fine-tuning ATC models through IS methods. In order to confirm this hypothesis, we propose three research questions for our Ph.D. dissertation. In sum, RQ1 aims at evaluating the traditional IS approaches shall be present in Section 4 in the context of the ATC task concerning the posed IS methods tripod constraints: reduction, effectiveness, and efficiency. Next, considering the literature gap, RQ2 aims at demonstrating the feasibility of proposing a novel IS framework and by showing how it can accommodate different requirements