Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors
Dominik Macko, Robert Moro, Ivan Srba · 2025
Read Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors by Dominik Macko, Robert Moro, Ivan Srba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15128.txt.
Introduction
Macko, Moro, and Srba present a 2025 paper on making machine-text detectors harder to fool. Language models now write text that people cannot tell from human work. Detectors exist, but tricks like paraphrasing or odd spacing can slip past them. The team focuses on fine-tuned detectors that work across many languages.
Their robust training process hardens the detectors against such disguises. It also helps them handle text styles they never saw in training. The work builds on fears that fake text can spread harm at scale. Stronger detectors give readers a credibility signal in the online information space. The authors test the approach on multilingual data sets.
Catalog record
| CREATOR | Dominik Macko, Robert Moro, Ivan Srba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15128
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Increasing the Robustness of the Fine-tuned Multilingual Machine-Generated Text Detectors Dominik Macko, Robert Moro, Ivan Srba Abstract Since the proliferation of LLMs, there have been concerns about their misuse for harmful content creation and spreading. Recent studies justify such fears, providing evidence of LLM vulnerabilities and high potential of their misuse. Humans are no longer able to distinguish between high-quality machine-generated and authentic human-written texts. Therefore, it is crucial to develop automated means to accurately detect machine-generated content. It would enable to identify such content in online information space, thus providing an additional information about its credibility. This work addresses the problem by proposing a robust fine-tuning process of LLMs for the detection task, making the detectors more robust against obfuscation and more generalizable to out-of-distribution data. 1 Introduction The number, variety, enhanced capabilities, and simple usage of large language models (LLMs) cause growing concerns about a massive generation of harmful content with LLMs being potentially maliciously misused ( Borji 2023 ; Zhuo et al. 2023 ; Crothers et al. 2023 ) . Previous research have demonstrated the ability of LLMs to produce disinformation texts ( Vykopal et al. 2024 ; Lucas et al. 2023 ) , even personalized ones Zugecova et al. 2024 ; Gabriel et al. 2024 , frauds for academic exams ( Achiam et al. 2023 ) , or plagiarism ( Wahle et al. 2022 ) . The fears of misuse are more severe due to indistinguishability between human-written texts and high-quality machine-generated texts ( Sadasivan et al. 2025 ) . Thus, the machine-generated text detection (MGTD) belongs to the key challenges associated with state-of-the-art LLMs as identified by ( Kaddour et al. 2023 ) . Since the harmful online content, such as misinformation, disinformation, propaganda, or impersonation, can be also generated or adjusted by LLMs, its automated detection can crucially help to fight such phenomena. Indication that a potentially harmful content is generated by using artificial intelligence (AI) can alleviate its eventual effect on people. By its statistical nature, the AI generated content should be considered less credible (in comparison to authentic human-written texts), mainly due to the simplicity and speed of creating the content as well as a potential presence of hallucinations (unintended misleading information). However, the state-of-the-art MGTD methods suffer by performance degradation in out-of-distribution settings, such as new generators, domains, languages, and paraphrasing or adversarial attacks ( Antoun et al. 2023 ; Li et al. 2024 ) . Therefore, increasing their robustness in this manner is the primary concern. This work addresses this problem by mixing the training data in a way to include multiple languages, domains, generators and even obfuscated (attack) data. To the best of our knowledge, this is the first attempt to systematically fine-tune robust multilingual MGT detectors (the source code available 1 1 1 https://anonymous.4open.science/r/robust_mgt_detector ). Although such detectors manifest a small performance degradation on in-distribution data, the out-of-distribution performance is significantly increased, making them more suitable to be deployed in the real-world settings. 2 Datasets For the detectors training, we have used a combination of multiple datasets described below. The statistics regarding sample counts for training are summarized in Table 1 . Dataset Human Machine Generator Language MultiSocial 290,618 290,618 7 40 MULTITuDE_v2 0 120,090 7 3 MULTITuDE_v3 136,577 143,527 7 22 MIX2k 1,000 1,000 75 7 Table 1: Text samples counts for the train splits of the included datasets (MIX2k used for validation only). MultiSocial ( Macko et al. 2024a ) A dataset of real social-media texts in 22 languages from 5 platforms, containing also the generated counterparts (by using 3 iterations of paraphrasing) by 7 generators. We have balanced the train split of this dataset by additional human samples (using the published procedure 2 2 2 https://anonymous.4open.science/r/multisocial ), covering also languages out of the intended 22 (specifically, 40 languages with at least 1,000 samples). MULTITuDE_v2 ( Macko et al. 2024b ) A subset of MULTITuDE_v2 dataset of obfuscated MULTITuDE ( Macko et al. 2023 ) texts, covering the three selected authorship obfuscation methods (DFTFooler adversarial attack, ChatGPT paraphrasing, and m2m100-1.2B backtranslation). Due to modification (obfuscation) of the texts by machines, we include only machine-labeled samples. MULTITuDE_v3 ( Macko et al. 2024a ) An extended version of the MULTITuDE ( Macko et al. 2023 ) dataset of real human-written news articles in 22 languages accompanied by counterparts generated by 7 generators, created for out-of-distribution evaluation by ( Macko et al. 2024a ) . We have requested this dataset from the authors (since it is not published) and we have balanced their train split of the dataset by additional human samples (using the same published procedure 3 3 3 https://github.com/kinit-sk/mgt-detection-benchmark ). MIX2k Our 2k subset of a mixture of existing datasets on binary machine-generated text detection task, covering various domains, 7 languages and over 75 generators (for more information about the dataset creation process, see Appendix B ). It contains 1,000 human samples and 1,000 machine samples. We have combined the train splits of the first three datasets for detectors fine-tuning. After de-duplication, it includes 427,195 human samples and 554,216 machine samples out of 2 domains (news and social media), 44 languages (of more than 1,000 samples), and 16 generators. A small imbalance in human vs machine classes was dealt by up-sampling of the minority class (i.e., human) during the training. The MIX2k dataset (containing mostly out-of-distribution data) was used for validation during the fine-tuning, i.e., for checkpoint selection. 3 Fine-tuning Procedure We used two kinds of fine-tuning, based on the model size. For smaller models, we have used the full fine-tuning process utilizing half precision, with the default AdamW optimizer with learning rate of 2e-6 , batch size of 32, validation each 2,000 steps, running at maximum for 10 epochs or 4 days. For the models having more than 2B parameters, we have used QLoRA ( Dettmers et al. 2024 ) parameter efficient fine-tuning process with the weighted cross entropy with logits for loss calculations and 4-bit quantization using BitsAndBytes 4 4 4 https://pypi.org/project/bitsandbytes . The LoRA configuration 5 5 5 https://pypi.org/project/peft used an alpha of 16, a dropout of 0.1, r of 64, and the task type of sequence classification targeting all linear modules. In comparison to full fine-tuning, the QLoRA fine-tuning version used paged AdamW optimizer with learning rate of 2e-5 , gradient accumulation of 4 steps, and evaluation each 200 steps. We used these hyperparameter values for all models to make them comparable. However, if a smaller batch size was necessary due to memory constraints in case of bigger models, we tuned the values accordingly. 4 Experimental Settings As a baseline, we have used the mDeBERTa-v3-base model (as shown by existing works as the best of smaller pre-trained models) fine-tuned on a combination of MULTITuDE_v3 and MultiSocial datasets (train splits without our balanced extension by additional human samples). We have selected 5 models with diverse architectures and numbers of parameters for robust fine-tuning. For the in-distribution evaluation, we have used the test splits of the MULTITuDE_v3 and MultiSocial datasets. For the out-of-distribution evaluation, we have used the test split of the SemEval-2024 Task 8 multilingual dataset ( Wang et al. 2024 ) , as well as the full MIX dataset (200k samples). The statistics