Watermark in the Classroom: A Conformal Framework for Adaptive AI Usage Detection

Yangxinyu Xie, Xuyang Chen, Zhimei Ren, Weijie J. Su · 2025

Read Watermark in the Classroom: A Conformal Framework for Adaptive AI Usage Detection by Yangxinyu Xie, Xuyang Chen, Zhimei Ren, Weijie J. Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23113.txt.

Introduction

Yangxinyu Xie and three co-authors tackle AI use in school writing in a 2025 arXiv paper. Teachers want to allow helpful AI edits yet catch misuse, but today's detectors flag innocent students too often and lean against some groups. Hidden marks in AI text, called watermarks, give firm odds, yet no one had tried them for class work with mixed levels of allowed help.

The team adds a checking method that keeps false flags in check across varied classes. They test essays from native and non-native English writers, with seven levels of AI help, from grammar fixes to added content. The setup spans several models and mark schemes, and it gives schools hard numbers for fair rules on AI help.

Catalog record

CREATORYangxinyu Xie, Xuyang Chen, Zhimei Ren, Weijie J. Su
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watermark in the Classroom: A Conformal Framework for Adaptive AI Usage Detection Yangxinyu Xie, Xuyang Chen, Zhimei Ren, Weijie J. Su Abstract As artificial intelligence tools become ubiquitous in education, maintaining academic integrity while accommodating pedagogically beneficial AI assistance presents unprecedented challenges. Current AI detection systems fail to control false positive rates (FPR) and suffer from bias against minority student groups, prompting institutional suspensions of these technologies. Watermarking techniques offer statistical rigor through precise p p -values but remain untested in educational contexts where students may use varying levels of permitted AI edits. We present the first adaptation of watermarking-based detection methods for classroom settings, introducing conformal methods that effectively control FPR across diverse classroom settings. Using essays from native and non-native English speakers, we simulate seven levels of AI editing interventions—from grammar correction to content expansion—across multiple language models and watermarking schemes, and evaluate our proposal under these different setups. Our findings provide educators with quantitative frameworks to enforce academic integrity standards while embracing AI integration in the classroom. Keywords: AI in Education, Academic Integrity, Large Language Model Watermarking, Conformal Prediction 1 Introduction 1.1 Background and Motivation In August 2023, Vanderbilt University disabled Turnitin’s AI detector ( Turnitin, LLC 2025 ) indefinitely, despite the growing pressure on institutions to maintain academic integrity in the age of generative artificial intelligence (AI) ( Coley 2023 ) . Soon after, universities across the country followed suit ( Moorhouse et al. 2023 ; McDonald et al. 2025 ; Ple ) . The culprit was a loss of confidence in the tools’ reliability. Commercial automated AI-text detectors—often built on training-based classifiers, such as Turnitin’s AI detector and GPTZero ( GPTZero 2025 ) —lack statistical guarantees regarding false-positive rates ( Khalil and Er 2023 ; Perkins et al. 2024b ; Fowler 2023 ; Bahar 2023 ) . Furthermore, these tools have been shown to bias against non-native English speakers (NNES) and first-generation students ( Liang et al. 2023 ; Campino 2024 ; Stone 2024 ) . Students subjected to false accusations experience profound psychological distress, heightened anxiety, and diminished trust in educational institutions ( Stone 2023 ; Stone 2024 ; Coldwell 2024 ) . Moreover, the disproportionate impact on NNES and first-generation students can exacerbate existing educational inequities and may impede their academic progression ( Wee and Reimer 2023 ; Stone 2024 ; Morris et al. 2019 ) . Figure 1 : Distribution of watermarking p -values (on \log_{100} scale) for human-written essays (gray) and essays with permitted AI grammar assistance (blue). Prompt 1 (blue) refers to prompting an AI model for minimal grammar correction; see definition in Section 3.2 . (a) Initial view without labels (b) View with true labels Figure 2 : Histogram of watermarking p -values (on \log_{100} scale) across 100 simulated student submissions. We encourage the reader to examine panel (a) and determine a threshold that would optimally differentiate between permitted and prohibited AI usage. The right panel (b) reveals ground truth labels, demonstrating the inherent difficulty of threshold selection without proper calibration data. Meanwhile, generative watermarking techniques have emerged as a potentially promising alternative ( Zhao et al. 2024 ; Lancaster 2023 ; Xie et al. 2023 ; Majovsky et al. 2024 ) . Unlike training-based classifiers, a watermarking protocol embeds verifiable statistical signals into model outputs during the generation process itself ( Kirchenbauer et al. 2023 ; Aaronson and Kirchner 2023 ; Xie et al. 2024 ; Abdalla and Vershynin 2025 ) . This approach yields precise p -values against the null hypothesis that a piece of text is entirely human-written, offering a more rigorous statistical foundation for detection than the heuristic “AI-likelihood” scores that classifier-based solutions provide. Despite this theoretical promise, watermark detectors remain untested in classrooms ( Perkins et al. 2024b ) . The challenge lies in the evolving nature of AI use in classrooms. Like classifier-based solutions, existing watermark techniques focus on distinguishing purely human-written texts versus machine-generated ones. However, as schools progressively integrate AI technologies into their pedagogy and encourage limited AI assistance to enhance student learning outcomes ( Milano et al. 2023 ; Perkins et al. 2024a ; Cai et al. 2025 ; Heaven 2023 ; Kangwa et al. 2025 ; Amato and Schoettle 2025 ; Mollick and Mollick 2024 ; Nie et al. 2024 ; Eaton 2025 ) , educators need to distinguish between allowed and prohibited AI use—a nuanced task that current watermarking techniques cannot directly address. For example, if a professor allows AI-based grammatical editing but forbids content generation, a naïve application—flagging any submission with watermark p -value below a fixed \alpha threshold—can fail to maintain the expected false positive rate at \alpha . Figure 1 illustrates this challenge through a simulation: while only 4% of the human-written essays attain watermark p -values below the \alpha=0.05 threshold, 25.5% of the essays with permitted AI-grammar editing attain watermark p -values below the same threshold. In other words, a student who follows the AI use guidelines can easily be falsely accused without mechanisms to adjust the threshold. Establishing valid thresholds that effectively control false positives is challenging. In large classrooms, instructors may collect many watermark p -values from student submissions, potentially enabling visual inspection of the histograms (Figure 2 ) to make decisions. However, in the absence of prior knowledge, determining the cutoffs can be subjective and inconsistent. Moreover, such a problem can be further exacerbated in small classrooms, where the limited sample size renders visual inspection methods even more unreliable. 1.2 This Article: A Conformal Framework for Adaptive AI Usage Detection Our analysis proceeds from the premise that hybrid writing content created through human-AI collaboration will become the new norm ( Eaton 2023 ; Eaton 2025 ; Luo 2024 ) . As such, while existing commercial AI detection tools and research focus on the binary classification of solely human versus solely machine-generated content ( Jiang et al. 2024a ; Hyatt et al. 2025 ; Turnitin, LLC 2025 ; GPTZero 2025 ) , we examine the nuanced spectrum of human-AI collaboration. Inspired by emerging pedagogical frameworks ( OgunleyeA et al. 2024 ; Perkins et al. 2024a ; Furze et al. 2024 ; Kılınç 2024 ) , we consider the following setting: First, instructors provide explicit guidelines regarding permitted AI use for writing assignments. For example, instructors may outline a set of approved prompts to use when interacting with large language models (LLMs) to edit their essays. After students submit their assignments, instructors receive the corresponding scores from AI watermark detection tools. To ensure academic integrity, instructors are interested in which submissions may have involved more AI-edits than the guidelines allow. Nonetheless, comprehensively enumerating all possible guideline violations is intractable. Given this complexity, we propose the philosophy that submissions with AI-edit signals statistically indistinguishable from those produced under permitted guidelines should not trigger academic integrity violations. This approach prioritizes controlling the false positive rate (FPR) of detection methods, defined as the probability of incorrectly rejecting the null hypothesis that a submission adheres to AI usage guidelines, in parallel with recent research trends in education literature ( Hyatt et al. 2025 ; Giray 2024 ) . A priori, the implementation of such a framework requires knowing the distribution of watermark scores under permitted AI usage

Read the complete text