AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models
Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, Edward Bowen · 2025
Read AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models by Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, Edward Bowen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05157.txt.
Introduction
Chinnappa Guggilla and co-authors build detectors for AI-written text in this 2025 paper. Machine text now mimics human writing closely enough to fuel scams and fake news. Two tasks matter: telling machine text from human text, and naming which model wrote it. The team fine-tuned three well-known language models for both tasks.
Their best setups separated human from machine text with about ninety-five percent accuracy. Naming the exact source model proved far harder, at under half correct. The gap shows that spotting machine text is easier than tracing its origin. Detectors like these may help schools, journals, and platforms screen submissions.
Catalog record
| CREATOR | Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, Edward Bowen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models Chinnappa Guggilla, Budhaditya Roy, Trupti Ramdas Chavan, Abdul Rahman, Edward Bowen Abstract Large Language Models (LLMs) possess an extraordinary capability to produce text that is not only coherent and contextually relevant but also strikingly similar to human writing. They adapt to various styles and genres, producing content that is both grammatically correct and semantically meaningful. Recently, LLMs have been misused to create highly realistic phishing emails, spread fake news, generate code to automate cyber crime, and write fraudulent scientific articles. Additionally, in many real-world applications, the generated content including style and topic and the generator model are not known beforehand. The increasing prevalence and sophistication of artificial intelligence (AI)-generated texts have made their detection progressively more challenging. Various attempts have been made to distinguish machine-generated text from human-authored content using linguistic, statistical, machine learning, and ensemble-based approaches. This work focuses on two primary objectives Task-A, which involves distinguishing human-written text from machine-generated text, and Task-B, which attempts to identify the specific LLM model responsible for the generation. Both of these tasks are based on fine tuning of Generative Pre-trained Transformer (GPT_4o-mini), Large Language Model Meta AI (LLaMA) 3 8B, and Bidirectional Encoder Representations from Transformers (BERT). The fine-tuned version of GPT_4o-mini and the BERT model has achieved accuracies of 0.9547 for Task-A and 0.4698 for Task-B. [[email protected] ] [[email protected] ] [[email protected] ] [[email protected] ] [[email protected] ] keywords Text detection ,GPT-4o-mini ,BERT ,LLaMA-3 ,LLMs ,fine-tuning ,human-authored ,machine-generated † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: De-Factify - Fourth Workshop on Multimodal Fact-Checking and Hate Speech Detection † † address: Deloitte & Touche Assurance and Enterprise Risk Services India Private Limited, India † † address: Deloitte & Touche LLP, USA 1 Introduction The advancement in natural language processing, demonstrated by the emergence of large language models (LLMs) has presented both opportunities and challenges. These models provide numerous beneficial applications; however, they also pose potential risks such as the widespread creation of synthetic disinformation, spam, and phishing content ( Solaiman et al. 2019 ) and fake news generation ( Uchendu et al. 2021 ) . To address these LLM challenges, a significant amount of work is being done to distinguish human-authored text from machine-generated text. Due to the ubiquitous nature of these neural language models, distinguishing machine-generated texts from human-written ones is no longer sufficient, and it is also necessary to identify which specific neural text generator authored a piece of text ( Uchendu et al. 2021 ) . To develop accurate detectors of machine-generated texts, sufficient data is essential. The authors ( Uchendu et al. 2021 ) created the first benchmark for Authorship Attribution in the form of the Turing Test, incorporating both human and neural language models. There are several proprietary tools such as Giant Language Model Test Room (GLTR) Needle 2024 , GPT-2 output detector, and GPTZero. However, these tools act as a black box and there is very little understanding about their working principal. The human and AI text classification literature can be broadly categorized into supervised, zero shot, retrieval-based, watermarking, and feature-based detection methods Abdali et al. 2024 ; Wu et al. 2024 . In the supervised methods, the pre-trained models are fine tuned for classification while zero-shot detection methods apply the pre-trained model in zero-shot settings. The retrieval-based algorithms uses similarity between given text and LLM generations. The watermarking approaches utilize the model signature in the generated text, while feature-based methods consider various features for classification. In the geneRative AI Detection viA Rewriting method (Raidar) Mao et al. 2024 , it is found that while using LLMs for the rewriting task, there is a high possibility of modification of human-generated text. Also, to identify AI-generated text, LLMs were prompted to rewrite and the editing distance was calculated between the outputs. Harika A. et al. Abburi et al. 2023 , used ensemble of five pre-trained transformer based models to generate the robust features and various ML algorithms such as logistic regression, gaussian naive bayes, random forest, and support vector machine were applied for text classification. In this work, two sub-tasks, namely, Task-A and Task-B are focused. The goal of Task-A is to determine whether a given text document is generated by AI or created by a human. Task-B focuses on the identification of which specific LLM such as DeBERTa, Falcon, GPT 4.0 ( OpenAI 2023 ) , Mamba, Phi-3.5 has generated a given text. The Open AI’s GPT-4o-mini ( OpenAI 2024 ) , LLaMA-3 8B pre-trained LLMs, and transformer-based BERT ( Kenton and Toutanova 2019 ) models are fine-tuned on the training datasets for Task-A and Task-B with simple prompts. The performance of the proposed approach is evaluated using a given validation and test dataset. The organization of the paper is as follows. Section 2 describes the dataset used in this work. The simple prompt-based instruction tuning methodology, and experimental results are presented in Sections 3 and 4 , respectively. This work is concluded in Section 5 2 Dataset This work uses the dataset provided by the Defactify workshop ( Roy et al. 2025a ; Roy et al. 2025b ) . The validation and testing data sets are provided by the workshop organizers. The details of the dataset distribution are mentioned in Table 1 . 2.1 Data Analysis The dataset includes human-authored stories from a variety of domains, including medical and election topics, as well as corresponding AI-generated text produced by several LLMs. The diverse set of LLMs used in the dataset creation process includes gemma-2-9b, GPT 4.0, llama-8b, mistral-7b, qwen2-72b and Yi-large. These models belong to LLMs families such as encoder, open-source, closed-source, small language models, mixture of experts and state space models. The snapshot of the dataset sample consisting of example prompt, human story and gemma-2-9b model generated text is shown in Figure 1 . Figure 4 illustrates the lexical diversity, while Figure 5 shows the average length of sentences from the training dataset. The majority of the human and LLM-generated text samples in the training dataset have a sequence length of 0 to 100. Human-written samples exhibit a wider lexical diversity, ranging from 0.2 to 0.8, whereas LLM-generated model samples fall within the range of 0.4 to 0.6. Figure 1 : An illustration of the sample format includes an example prompt, a human-generated story, and text generated by the gemma-2-9b model from the text detection training dataset. ( Roy et al. 2025b ) The labels in train and validation set are equally distributed as shown in the Table 1 . The dataset consists of seven labels that includes human-written and six different LLMs. Table 1 : Distribution of text detection dataset ( Roy et al. 2025b ) for Task-A and Task-B Data set Human Machine Total Train 7,321 43,926 51,147 Validation 1,569 9,414 10,983 Test - - 10,963 3 Proposed Approach In this section, we describe the proposed methodology for modeling Task-A and Task-B. Task-A is approached as a binary classification problem, where the goal is to determine whether a given input text is human-written or machine-generated. Task-B, on the other hand,