Paraphrasing Attack Resilience of Various AI-Generated Text Detection Methods

Andrii Shportko, Inessa Verbitsky · 2026

Read Paraphrasing Attack Resilience of Various AI-Generated Text Detection Methods by Andrii Shportko, Inessa Verbitsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14240.txt.

Introduction

Shportko and Verbitsky test how well AI-generated text detectors survive paraphrasing attacks. The spread of large language models has raised plagiarism and false-information risks, while detector-bypassing tools make reliable machine-generated text spotting vital. The paper tests three methods, fine-tuned RoBERTa, Binoculars and text feature analysis, plus ensembles built with Random Forest sorters. Ensembles with Binoculars score strongest overall yet suffer the steepest losses under attack, showing a performance-versus-resilience split in AI text spotting.

Catalog record

CREATORAndrii Shportko, Inessa Verbitsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26588
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14240

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Paraphrasing Attack Resilience of Various AI-Generated Text Detection Methods Andrii Shportko, Inessa Verbitsky Abstract The recent large-scale emergence of LLMs has left an open space for dealing with their consequences, such as plagiarism or the spread of false information on the Internet. Coupling this with the rise of AI detector bypassing tools, reliable machine-generated text detection is in increasingly high demand. We investigate the paraphrasing attack resilience of various machine-generated text detection methods, evaluating three approaches: fine-tuned RoBERTa, Binoculars, and text feature analysis, along with their ensembles using Random Forest classifiers. We discovered that Binoculars-inclusive ensembles yield the strongest results, but they also suffer the most significant losses during attacks. In this paper, we present the dichotomy of performance versus resilience in the world of AI text detection, which complicates the current perception of reliability among state-of-the-art techniques. 1 Introduction The widespread use of LLMs can be precarious when left unchecked, with the consequences ranging from intellectual dishonesty to the spread of fake news on social media. 5 found that AI chatbots can easily produce both realistic-looking academic results and a polished manuscript that may well be accepted to a conference and published. Since scientific research, especially medical, is often falsified, the emergence of such a possibility opens up a dangerous playing field ( 21 ) . It was found that 14% of scientists were aware of colleagues who falsified results, whereas 72% of scientists knew of colleagues who engaged in questionable research practices ( 7 ) . More incidents of AI being used in the case of fake news spreading on the internet can be found in the Ethics Statement. What is particularly concerning about this is that humans have been found to perform rather poorly on manual detection of AI-written text. In particular, human performance has shown to be only marginally better than random classification ( 25 ) . In fact, in a study involving over 130 subjects, 14 found that participants were only able to correctly identify AI-generated text with an accuracy rate of 24%. Concerning the use of AI in academia, 8 conducted an experiment where participants were to identify whether abstracts for academic papers were written by ChatGPT or a human. They found that only 68% of the AI-detected abstracts were correctly classified. Such a precedent makes a strong case for the necessity of precise automated AI text detection mechanisms. With the emergence of freely accessible sites such as ZeroGPT, DetectGPT, and Quillbot, bypassing attacks have been developed against these technologies. Methods which are commonly used include automated paraphrasing tools, prompt engineering, and the calculated addition of errors into AI-generated text ( 20 ) . It has been generally shown that the use of these methods decreases the efficacy of the detection tool; however, we aim to put together a more comprehensive analysis of the leading AI detection methods against bypassing attacks. In this paper, we will focus on paraphrasing attacks. The leading state-of-the-art detectors can be categorized into two paradigms, those using training-based and training-free mechanisms ( 24 ) . Most training-free approaches rely on statistical feature analysis and commonly look at perplexity, log probability, and n-grams ( 2 ) . Although training-based models have been widely leading, a recently developed methodology – Binoculars – proves successful in a zero-shot context, which stands out over multiple metrics ( 11 ) . This approach is developed further in the Related Work section. Training-based methods largely rely on transformer models, namely RoBERTa ( 15 ) , a masked-based model, easily fine-tunable for downstream tasks such as text classification. The methods we stacked to develop our own model include Binoculars, RoBERTa, and text feature analysis, which we justify due to their leading benchmarks (detailed in Related Work). 2 Related Work 2.1 Binoculars The Binoculars method relies on calculations from two closely related LLMs. It has a significant advantage over other SOTA methods in that it uses no training from the LLM that it is being tested on. This is significant, considering Binoculars still manages to surpass every open-source model that detects ChatGPT. Because other detectors rely on pretraining of the models they then test, the results fail to generalize when tested across multiple AI models. The Binoculars method, however, achieves high performance on a variety of datasets, which gather texts from different LLM sources. Furthermore, Binoculars addresses what they call the “Capybara Problem”, which in essence refers to the phenomenon of an LLM generating high-perplexity text simply due to a high-perplexity prompt being used. Other models which focus on raw perplexity will fail in such cases. Binoculars has an accuracy rate above 90%, and a false positive rate of 0.01%, using datasets which include Writing prompts, News, and Student essays ( 23 ) . 2.2 Text Features 18 analyzed linguistic patterns in human and LLM text to determine which features would provide for the most robust detection mechanism. Using extensive data from six different LLMs, including Llama and Falcon 7-b, they found that human writing tends to have less uniform sentence length distribution than AI. This conclusion is supported by 4 , who found that the standard deviation of sentence length was an important identifier in text classification. As one of our five text features used, we thus implement standard deviation of sentence length. 2.3 Ensembling 1 analyze the success in using ensemble approaches for text classification. Their ensemble involves stacking DeBERTa, RoBERTa, and xLM-RoBERTa, fine-tuning each model for the appropriate tasks. They found that this approach reached 5th place in the English task and first place in the Multilingual for the Automated Text Identification shared task. In fact, ensembling was highly used in Task 1 of the COLING 2025 GenAI Text detection workshop , from which we use the dataset provided to train and evaluate our own model ( 24 ) . 17 , whose approach scored 4th among contestants, relied on ensembling RoBERTa-base with other pre-trained transformer models. Our methodology also relies on RoBERTa, however, we ensemble it with Binoculars and text feature analysis, as justified above. 2.4 Bypassing Figure 1: Pipeline of our model The most prominent AI text-detection models relying on transformer fine-tuning have been tested against bypassing and proven to largely withstand it. 13 provide a critical baseline by demonstrating that controlled paraphrasing can significantly undermine the performance of AI-generated text detectors while maintaining semantic integrity. Their work, through the DIPPER model, shows that even minimal paraphrasing – changing wording and sentence structure – can drop detection accuracy significantly. Some common AI detectors saw decreases of around 17% in accuracy ( 20 ) when bypassing methods were employed. However, some more recently developed models were created specifically to withstand such attacks, such as the RADAR model ( 12 ) , which trains the detector on paraphrasing schemes and achieves over 31.64% of additional accuracy compared to previous methods. The Binoculars method, however, has not been tested against bypassing, thus its general efficacy remains unclear. This concern is explored in our paper. 3 Data and Methodology To track the progress on machine-generated text detection, we use the materials of the competition on Detecting AI Generated Content @COLING 2025 Task 1: Binary Machine-Generated Text Detection ( 24 ) . It is an aggregation of

Read the complete text