AI Text Detectors and the Misclassification of Slightly Polished Arabic Text
Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi · 2025
Read AI Text Detectors and the Misclassification of Slightly Polished Arabic Text by Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16690.txt.
Introduction
Saleh Almohaimeed and four co-authors expose a fairness problem in AI text detectors. Human writing that an AI merely polished gets flagged as machine-written, at least in Arabic, where the issue had not been studied. Such false accusations can damage authors and discredit the detectors themselves.
The team built two datasets, one of 800 Arabic articles split between human and machine authorship, and Ar-APT, with 400 human articles polished by ten language models under four settings each. Fourteen candidate detectors were screened, and the best eight faced the polished texts. Every detector misattributed a large share. The best commercial tool fell from ninety-two percent accuracy to twelve percent on lightly polished text, and the best language model judge dropped by about a quarter. That is a stark warning for Arabic academic integrity tools.
Catalog record
| CREATOR | Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16690
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI Text Detectors and the Misclassification of Slightly Polished Arabic Text Saleh Almohaimeed, Saad Almohaimeed, Mousa Jari, Khaled A. Alobaid, Fahad Alotaibi Abstract Many AI detection models have been developed to counter the presence of articles created by artificial intelligence (AI). However, if a human-authored article is slightly polished by AI, a shift will occur in the borderline decision of these AI detection models, leading them to consider it as AI-generated article. This misclassification may result in falsely accusing authors of AI plagiarism and harm the credibility of AI detectors. In English, some efforts were made to meet this challenge, but not in Arabic. In this paper, we generated two datasets. The first dataset contains 800 Arabic articles, half AI-generated and half human-authored. We used it to evaluate 14 Large Language models (LLMs) and commercial AI detectors to assess their ability in distinguishing between human-authored and AI-generated articles. The best 8 models were chosen to act as detectors for our primary concern, which is whether they would consider slightly polished human-authored text as AI-generated. The second dataset, Ar-APT, contains 400 Arabic human-authored articles polished by 10 LLMs using 4 polishing settings, totaling 16400 samples. We use it to evaluate the 8 nominated models and determine whether slight polishing will affect their performance. The results reveal that all AI detectors incorrectly attribute a significant number of articles to AI. The best performing LLM, Claude-4 Sonnet, achieved 83.51%, its performance decreased to 57.63% for articles slightly polished by LLaMA-3. Whereas the best performing commercial model, originality.AI, achieves 92% accuracy, dropped to 12% for articles slightly polished by Mistral or Gemma-3. keywords Arabic AI Detection, , AI-polished text, Detection fairness 1 Introduction Recent years have seen a great deal of competition between Large Language Model (LLM) companies to provide better multilingual models with an advanced level of capabilities. One of their abilities is to generate text that cannot be distinguished from authentic human writing. Due to this, a large number of English text-detector models have been developed. These detectors are designed to distinguish human-authored text from AI-generated content. However, a critical issue arises when someone uses LLM to slightly polish their own human writing without changing its meaning or adding additional information, as shown in Figure 1 . In such cases, text detectors struggle to distinguish whether it is a human-written or entirely generated by artificial intelligence. Consequently, there is a risk of falsely accusing the article of being generated by artificial intelligence when it is not. According to 22 , 10.86% of articles on 246 Fortune 500 companies’ blogs are generated by artificial intelligence. However, this may not be true, as authors may only use LLM to enhance the readability of their articles slightly. Figure 1: This is an example of an Arabic human-written article that has been polished twice under 10% and 50% polishing settings. Words with red colors are polished words in comparison to the human-written text From 40 , the authors study this issue and provide clear explanations as to why better models are needed to solve this problem. However, the study focused only on English, and did not give much consideration to other languages, such as Arabic. Compared to many other languages, the Arabic language has a richer morphology and a variety of dialects. Moreover, the use of similar words with different diacritics may lead to a different meaning, even if they are within the same context. Having to deal with these issues makes it more difficult to build a text detector for Arabic articles. In this paper, we have conducted a comprehensive investigation using more than 300 experiments in order to address three main concerns. One, what is the effectiveness of popular LLMs and commercial AI models in distinguishing between Arabic human-written articles and those generated by AI. Second, we investigate the effectiveness of these LLMs in terms of polishing text without changing the meaning. Third, we retest the LLMs and commercial AI models to determine whether they consider slightly polished Arabic text as AI-generated. In order to accomplish this, we first constructed two datasets. The first dataset consists of 800 samples, half of which are human-written and the other half are generated by AI. The second dataset is Ar-APT 1 1 1 github.com/Saleh-Almohaimeed/Ar-APT (Arabic AI polish text), which contains 400 authentic Arabic human texts polished by 10 LLMs, each with four different polishing percentage levels, resulting in 16400 samples. The first dataset was used as a means to address our first concern, which is testing the ability of LLMs and very popular commercial models to differentiate between human-authored articles and AI-generated articles. The second dataset Ar-APT is used to address the aforementioned second and third concerns. We also used Ar-APT to demonstrate our main objective, which is the problem of falsely accusing human-written articles as AI-generated when only slight polishing was done. There were 14 experiments conducted on the first dataset using different LLMs, such as the GPT-4o 37 , LLaMA-3 70B 31 , and Claude-4 Sonnet 9 , as well as commercial models, such as ZeroGPT 11 and Oraginality-AI 21 . We will nominate models that achieve acceptable results to act as text detectors on the Ar-APT dataset. After that, we conducted 320 experiments on the Ar-APT dataset to determine whether nominated LLMs and Commercial models consider human-polished text to be AI-generated or not. As a result of our findings, it is necessary to design models specifically developed to serve as text detectors for Arabic texts. As the best performing publicly available model is Claude-4 Sonnet 9 , which achieved an accuracy of 83.51% and a false positive rate of 16.49%, while the best commercial model achieved 96% accuracy and 8% false positive rate. Moreover, on Arabic slightly polished text, all experimented LLMs and commercial models experienced a drop in performance in determining whether the text was generated using AI or not. Under 10% polishing percentage, the Claude-4 Sonnet 9 produces an accuracy rate of 65.03% and a false positive rate of 34.97% with articles slightly polished by Mistral 30 . Moreover, with articles that were polished by Gemma-3 15 , Claude-4 Sonnet got 65.71% accuracy, 34.29% false positive rate. Thus, with only slightly polished articles, Claude-4 Sonnet’s accuracy decreased by approximately 18%. Furthermore, the best performing commercial model, Originality.AI 21 , exhibited an impressive performance decline when detecting human articles that were slightly polished (10%) by Mistral 30 and Gemma-3 15 . Originality.AI accuracy has dropped to 12% and the false positive rate jumped to 88%. This clearly highlights the necessity of addressing this issue, both for researchers and for the owners of the popular Originality.AI 21 model. Moreover, an analysis was conducted to determine whether LLMs are capable of polishing arabic text while keeping the meaning unchanged. In the results, Claude-4 Sonnet performed best, followed by Deepseek-3.1 16 and GPT-4o 37 . Additionally, all of the 10 LLMs used for polishing managed to maintain the meaning and got a high cosine similarity score, except for the Qwen-3 3 model, which appears to have difficulty with Arabic texts. Contributions: 1. Constructed the first dataset that contains 800 samples of AI-generated text and human-authored text. 2. A second dataset (Ar-APT) was constructed using 400 human-authored