BrightCookies at SemEval-2025 Task 9: Exploring Data Augmentation for Food Hazard Classification
Foteini Papadopoulou, Osman Mutlu, Neris Özen, Bas H.M. van der Velden, Iris Hendrickx, Ali Hürriyetoğlu · 2025
Read BrightCookies at SemEval-2025 Task 9: Exploring Data Augmentation for Food Hazard Classification by Foteini Papadopoulou, Osman Mutlu, Neris Özen, Bas H.M. van der Velden, Iris Hendrickx, Ali Hürriyetoğlu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20703.txt.
Introduction
Foteini Papadopoulou and five co-authors tested text tricks to lift food hazard sorting. Their paper, from twenty twenty-five, tells of a shared task system for a SemEval challenge. The task asks for clear sorting of hazards and products in food recall reports, at two levels of detail. Rare classes drag scores down. The team tried to pad the thin classes with made up text.
Three word level tricks were tried: swap in like words, swap word order, and drop in context fit words. Transformer models beat classic learners on the whole. None of the three tricks won across the board. Readers in food safety data learn where plain padding helps and where it fails.
Catalog record
| CREATOR | Foteini Papadopoulou, Osman Mutlu, Neris Özen, Bas H.M. van der Velden, Iris Hendrickx, Ali Hürriyetoğlu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20703
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BrightCookies at SemEval-2025 Task 9: Exploring Data Augmentation for Food Hazard Classification Foteini Papadopoulou, Osman Mutlu, Neris Özen, Bas H.M. van der Velden, Iris Hendrickx, Ali Hürriyetoğlu Abstract This paper presents our system developed for the SemEval-2025 Task 9: The Food Hazard Detection Challenge. The shared task’s objective is to evaluate explainable classification systems for classifying hazards and products in two levels of granularity from food recall incident reports. In this work, we propose text augmentation techniques as a way to improve poor performance on minority classes and compare their effect for each category on various transformer and machine learning models. We explore three word-level data augmentation techniques, namely synonym replacement, random word swapping, and contextual word insertion. The results show that transformer models tend to have a better overall performance. None of the three augmentation techniques consistently improved overall performance for classifying hazards and products. We observed a statistically significant improvement (P < 0.05) in the fine-grained categories when using the BERT model to compare the baseline with each augmented model. Compared to the baseline, the contextual words insertion augmentation improved the accuracy of predictions for the minority hazard classes by 6%. This suggests that targeted augmentation of minority classes can improve the performance of transformer models. 1 Introduction Foodborne diseases affect millions of people every year. The World Health Organization highlights that food contamination leads to more than 200 diseases, resulting in severe health complications and affecting the socioeconomic stability of communities and nations World Health Organization (2024) . There is a vast amount of publicly available information on food safety-related websites. Given the importance of early detection of food hazards, there is a need to timely and accurately analyze all this publicly available information to detect food hazards. Figure 1: An overview of our developed system’s architecture. The SemEval-2025 Task 9: Food Hazard Detection Challenge Randl et al. (2025) was proposed to facilitate automated classification of food hazards in food safety-related documents. It stimulates research that combines food safety and natural language processing (NLP) for explainable multi-class classification of food recall incident reports. SemEval-2025 Task 9 includes two sub-tasks: classifying coarse food hazard and product categories (ST1) ( hazard-category, product-category ), and fine-grained hazard and product categories (ST2) ( hazard, product ). A significant challenge with the SemEval-2025 Task 9 dataset is its substantial class imbalance. There is a long-tailed distribution across classes, especially in the fine-grained categories. This imbalance can give poor performance of classifiers, especially for deep learning (DL) models Henning et al. (2023) . Text augmentation techniques have been shown to mitigate the effects of imbalanced data to an extent Khan and Venugopal (2024) . Text augmentation can range from simple string manipulations, such as those used in Easy Data Augmentation (EDA) Wei and Zou (2019) , to more advanced methods involving transformer-based text generation Henning et al. (2023) . This helps to boost the representation of minority classes, which can result in a more balanced dataset and robust models. We investigated three basic text augmentation techniques (synonym replacement, contextual word insertion, and random word swapping) to boost the representation of under-represented classes in multi-class classifications of food recall incident reports. Our main research question is: Can text augmentation techniques on under-represented classes enhance a food hazard multi-class classifier’s performance? We evaluated the performance of various machine learning (ML) algorithms and encoder-only transformer models, both in their baseline form and after applying each augmentation technique. To participate in the task, only one submission was allowed. We submitted our predictions in the official test set after we evaluated our models in the development set, selecting the best-performing ones for each category. In ST1, our system ranked 15 th out of 27 participants, with an F_{1} -macro score difference of 0.0613 from the first, and in ST2, it ranked 11 th out of 26, with a 0.0944 score gap from the top (see subsection 6.1 for the exact scores). Our work provides valuable insights into the efficacy of text augmentation in this field. 1 1 1 Our code is available at https://github.com/WFSRDataScience/SemEval2025Task9 2 Related Work 2.1 Research on food hazard classification Little work has been conducted using text data for fine-grained food hazard classification Randl et al. (2024b) , as most existing literature focused on binary classification of food hazards. A recent study by Randl et al. (2024b) introduced the dataset that we used in SemEval-2025 Task 9 and they benchmarked multiple ML and DL algorithms. Randl et al. (2024b) proposed a large language model (LLM)-in-the-loop framework named Conformal In-Context Learning (CICLe), that leveraged Conformal Prediction to optimize context length for predictions of a base classifier. By using fewer, more targeted examples, performance increased and energy consumption reduced compared to regular prompting. 2.2 Text augmentation for minority classes Data augmentation creates synthetic data from an existing dataset by inserting small changes into copies of the data Shorten et al. (2021) . Data augmentation mitigates the class imbalance issues for DL Henning et al. (2023) . According to Shorten et al. (2021) , Data augmentation approaches in NLP can be divided in two types: symbolic and neural. Symbolic techniques, such as rule-based EDA Wei and Zou (2019) , employ simple word-level operations like synonym replacement and random insertion. Symbolic techniques are effective in small datasets. Neural techniques rely on auxiliary neural networks such as back-translation or generative augmentation. A recent study showed that LLMs for data augmentation, such as to generate new samples, increase accuracy and address class imbalance in skewed datasets ( Gopali et al., 2024 ) . In our study, we explore both symbolic and simple neural augmentation strategies, such as contextual words insertion using BERT, to improve classification performance. Additionally, in the SemEval shared task of 2023, Al-Azzawi et al. (2023) explored the effects of data augmentation, particularly back translation, on minority classes. They compared it with augmenting the entire dataset using transformer-based models. They observed that targeting the underrepresented classes for augmentation proved more effective than broad dataset augmentation. Following their approach, we also focus our augmentation strategies on the minority classes rather than the entire dataset. 3 Data The Food Recall Incidents dataset used in SemEval-2025 Task 9 contains 6,644 food-recall announcements in the English language Randl et al. (2024a) . This dataset is split into 5,082 announcements in the train set, 565 in the development set, and 997 in the test set. The data is collected from 24 different websites ( Table 4 ). The samples consist of a title and text describing announcements from a recalled food product and includes other metadata. Experts manually labeled each sample into four coarse classes of hazards ( hazard-category ) and products ( product-category ) and fine-grained classes ( hazard and product ). The classes and the number of classes per category are listed in Table 10 , and examples are presented in Table 11 . The distribution of the four categories’ classes is highly imbalanced, showing a long-tail effect ( Figure 4 , Figure 5 ). In coarse categories, 75% of classes have 513 samples in hazard-category and 263 in product-category , while the largest classes contain 1,854 and 1,434 samples, respectively. This imbalance is even more severe in the fine-grained hazard and