LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA

Houji Jin, Negin Ashrafi ID, Armin Abdollahi, Wei Liu, Jian Wang, Ganyu Gui, Maryam Pishgar ID, Huanghao Feng · 2025

Read LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA by Houji Jin, Negin Ashrafi ID, Armin Abdollahi, Wei Liu, Jian Wang, Ganyu Gui, Maryam Pishgar ID, Huanghao Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00731.txt.

Introduction

Houji Jin leads a team of eight authors on detecting AI-written Chinese text. Subtle features of the Chinese language make this task hard. Detectors must also survive shifts in the data they face. The team compares encoder models with decoder models head to head. A simple FastText baseline is included as well.

Encoders such as BERT and RoBERTa are fine-tuned with prompt-based masked modeling. A decoder model, Qwen, is adapted with LoRA instead. LoRA is a low-rank method that tunes few weights. Tests use the NLPCC 2025 task dataset. Encoders nearly memorize their training data but degrade under shift. FastText stays robust on word-level cues. The LoRA-adapted decoder reaches the highest test accuracy. Its precision and recall are also well balanced.

Catalog record

CREATORHouji Jin, Negin Ashrafi ID, Armin Abdollahi, Wei Liu, Jian Wang, Ganyu Gui, Maryam Pishgar ID, Huanghao Feng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA Houji Jin, Negin Ashrafi ID, Armin Abdollahi, Wei Liu, Jian Wang, Ganyu Gui, Maryam Pishgar ID, Huanghao Feng Abstract The rapid growth of large language models (LLMs) has heightened the demand for accurate detection of AI-generated text, particularly in languages like Chinese, where subtle linguistic nuances pose significant challenges to current methods. In this study, we conduct a systematic comparison of encoder-based Transformers (Chinese BERT-large and RoBERTa-wwm-ext-large), a decoder-only LLM (Alibaba’s Qwen2.5-7B/DeepSeek-R1-Distill-Qwen-7B fine-tuned via Low-Rank Adaptation (LoRA)), and a FastText baseline using the publicly available dataset from the NLPCC 2025 Chinese AI-Generated Text Detection Task. Encoder models were fine-tuned using a novel prompt-based masked language modeling approach, while Qwen2.5-7B was adapted for classification with an instruction-format input and a lightweight classification head trained via LoRA. Experiments reveal that, although encoder models nearly memorize training data, they suffer significant performance degradation under distribution shifts (RoBERTa: 76.3% test accuracy; BERT: 79.3%). FastText demonstrates surprising lexical robustness (83.5% accuracy) yet lacks deeper semantic understanding. In contrast, the LoRA-adapted Qwen2.5-7B achieves 95.94% test accuracy with balanced precision–recall metrics, indicating superior generalization and resilience to dataset-specific artifacts. These findings underscore the efficacy of decoder-based LLMs with parameter-efficient fine-tuning for robust Chinese AI-generated text detection. Future work will explore next-generation Qwen3 models, distilled variants, and ensemble strategies to further enhance cross-domain robustness. K eywords Chinese AI-generated text detection \cdot LoRA \cdot Transformer encoder \cdot Transformer decoder \cdot Bert \cdot Qwen. 1 Introduction Large Language Models (LLMs) can now generate text that closely mimics human writing, raising concerns about misuse and the difficulty of distinguishing AI-generated content from human-written text Solaiman et al. 2019 , Zellers et al. 2019 , Mitchell et al. 2023 . In fact, studies have shown that humans perform only slightly better than random chance when trying to detect AI-generated text Clark et al. 2021 , Ippolito et al. 2019 . This has motivated extensive research into LLM-generated text detection, which aims to discern if a given text was produced by an AI or a human – essentially a binary classification problem Zellers et al. 2019 , Mitchell et al. 2023 , Jawahar et al. 2020 . This challenge has taken on growing importance across domains such as education, journalism, and scientific publishing, where trust and authorship are critical. In parallel, the detection of AI-generated Chinese text presents additional complexity due to its character-level structure, absence of explicit word boundaries, and rich semantic ambiguity Ng et al. 2014 , Zhao et al. 2013 . Despite this, relatively few studies have rigorously evaluated model architectures tailored for Chinese AI-text detection Wang et al. 2024 , Tao et al. 2024 . The NLPCC 2025 Shared Task 1, LLM-Generated Text Detection, introduced a publicly available dataset of Chinese texts annotated as either human-written or AI-generated. In this study, we use that dataset to investigate how different model architectures—encoder-based Transformers (such as BERT Devlin et al. 2019 and RoBERTa Liu et al. 2019 ) versus decoder-based LLMs (such as the GPT-style Qwen family Brown et al. 2020 , Bai et al. 2023 ), perform on the task of Chinese AI-generated text detection. In this paper, we compare representative encoder and decoder models on the binary classification task of detecting AI-generated Chinese text. We fine-tune two prevalent encoder models (Chinese BERT and RoBERTa Cui et al. 2020a ) and strong decoder LLMs (Alibaba’s Qwen-7B and DeepSeek-R1-Distill-Qwen-7B Bai et al. 2023 , Guo et al. 2025 ) on the same dataset, and also include a classical linear baseline (FastText Joulin et al. 2016 ) for reference. We then evaluate their precision, recall, and F1 performance on training, development, and test sets. This comparative study of encoder versus decoder Transformer architectures highlights the relative strengths of each approach and provides practical insights for building robust detectors of Chinese AI-generated text. The key contributions of this study are threefold. First, we provide the first direct comparison between encoder and decoder architectures for Chinese AI-generated text detection under identical conditions. Second, we introduce a prompt-based masked language modeling approach for encoders that improves out-of-domain generalization. Third, we demonstrate that a LoRA-adapted decoder (Qwen2.5-7B) achieves state-of-the-art robustness, substantially outperforming both strong encoder baselines and classical lexical models. To guide the reader through the remainder of this paper, Section 2 reviews prior work on AI-generated text detection with a focus on Chinese. Section 3 presents the selected models, including encoder-based Transformers, decoder-only LLMs, and a lexical baseline. Section 4 outlines the training and evaluation methodology. Section 5 reports the experimental findings and provides comparative analysis. Finally, Section 6 concludes with implications and directions for future research. 2 Related Work Early efforts on AI-generated text detection mostly employed encoder-based classifiers. For instance, Petropoulos et al. fine-tuned a RoBERTa model to classify whether a given text (in English or Chinese) was written by a human or by a model Petropoulos and Petropoulos 2024 . Such BERT/RoBERTa-based detectors achieved strong in-domain accuracy, though they risk overfitting to the writing style of seen AI models and may struggle on out-of-domain data. On the other hand, researchers have explored zero-shot detection methods that do not require training data. One prominent example is DetectGPT Mitchell et al. 2023 , which examines the probability curvature of a text under a language model; AI-generated text tends to have lower likelihood than paraphrases of the same text. While zero-shot methods like DetectGPT and others based on perplexity or burstiness can flag AI-generated text without supervised training, they often require long inputs to be reliable and their accuracy drops on shorter texts, which is a limitation for sentence-level detection Tian et al. 2023 . Another line of work uses watermarking: imperceptibly imprinting a signal in AI-generated text so it can later be identified. Kirchenbauer et al. introduced a cryptographic watermarking method for LLM outputs Kirchenbauer et al. 2023 . However, watermark approaches have downsides: they can slightly degrade text quality, and adversaries may remove or obfuscate the watermark, so these methods are not foolproof Liu et al. 2024 . Furthermore, watermarking requires control over the text generation process, which is not applicable if we need to detect arbitrary AI text ex-post Zhang et al. 2023 . More recently, the emergence of large instruct-tuned models has opened a new avenue: using the LLMs themselves as detectors Wang et al. 2024 . The intuition is that an LLM possesses a great deal of knowledge from pre-training, potentially including subtle cues of AI-generated text. Similar strategies leveraging pretraining and teacher–student architectures have also shown strong performance in other domains Jin et al. 2025 . Instruction tuning an LLM on the detection task can align it to produce the desired classification behavior. For example, the LLM-Detector system fine-tuned open-source LLMs (7B–13B parameters) with detection instructions and achieved promising results, outperforming earlier methods on both in-domain and cross-domain detection Wang et al. 2024 . These advances suggest that decoder-based models, despite being generative, can be adapted into powerful classifiers for AI-generated text. Overall, previous research has demonstrated the effectiveness of encoder-based models for detection, and more recent evidence shows that instruction-tuned decoder LLMs can match or even surpass their performance. However, there has been little direct comparison

Read the complete text