A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering
Yi Liu · 2026
Read A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering by Yi Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00402.txt.
Introduction
Liu builds a firm frame to spot machine-made text through rewrites, with set false-call caps. Rewrite-based spotters ask how a text shifts when reworked. Such spotters had no firm cap on false calls in the past. His frame adds that cap with no new training at all.
The key point is that rewrites act like spare twin texts. That twin form lets the task be cast as a batch of claim tests. False-call share can then be held by a plain tune step. Old rewrite tools gain the cap through that step alone. Tests span three spot tools, 19 fields, and four text models. False calls stay in bounds while true finds stay strong.
Catalog record
| CREATOR | Yi Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering Yi Liu Abstract We propose a distribution-free statistical framework that converts arbitrary rewrite-based detectors into detectors with finite-sample FDR guarantees without retraining. Our key observation is that rewrite-based detection implicitly constructs knockoff samples, enabling LLM-generated text detection to be formulated as a multiple hypothesis testing problem with knockoff structure. This perspective separates the design of detection statistics from the control of false discoveries, allowing existing rewrite detectors to inherit finite-sample false discovery rate (FDR) guarantees through a simple calibration procedure. We demonstrate reliable FDR control with meaningful detection power across three detection models, 19 domains, and four LLMs. 1 Introduction Detecting LLM-generated text has attracted significant recent attention. Existing approaches generally fall into two categories: logit-based methods, which analyze token probability statistics, and rewrite-based methods, which compare a text with an LLM-generated rewrite 5 ; 8 ; 18 ; 27 ; 11 ; 24 ; 28 ; 26 ; 6 ; 1 ; 20 ; 4 . We observe that rewrite-based detection implicitly constructs text–rewrite pairs that are approximately exchangeable under AI-generated text, naturally inducing a knockoff structure. Under this perspective, rewrite-based detection can be reformulated as a multiple hypothesis testing problem with finite-sample false discovery rate (FDR) guarantees. Building on this observation, we introduce a distribution-free statistical framework that converts arbitrary rewrite-based detectors into detectors with finite-sample FDR guarantees without retraining. Our contributions are threefold: 1. We show that rewrite-based detection implicitly generates knockoff samples, enabling a multiple hypothesis testing formulation. 2. We introduce a distribution-free detection procedure that converts rewrite-based statistics into detectors with provable FDR control under a knockoff symmetry condition. 3. We provide theoretical and empirical results showing that the proposed framework controls the FDR whenever the knockoff symmetry condition holds. 1.1 Rewrite-based approaches Rewrite-based methods compare a text with an LLM-generated rewrite. For AI-generated text, the original text and its rewrite are approximately exchangeable, since both arise from the same underlying LLM distribution. In contrast, rewriting human text induces a directional shift toward the LLM distribution 27 ; 22 . This asymmetry naturally connects rewrite-based detection to knockoff filtering. In many NLP applications, LLM detection is not used to classify a single isolated document, but to screen large corpora such as reviews, assignments, web pages, or benchmark data. In corpus-scale detection, threshold selection matters as much as score quality. A detector with high AUROC may still produce many false accusations when applied to thousands of documents. FDR control gives a deployment-facing guarantee: among texts classified as human-written, the expected fraction that are actually AI-generated is bounded by a user-chosen level. Moreover, many approaches require retraining whenever the rewrite model changes, making them costly and brittle in practice. 1.2 Knockoff Filter We treat AI-generated text as the null hypothesis and human-written text as the alternative; discoveries therefore correspond to texts selected as human-written. Treating this as a hypothesis testing problem and solving it using knockoff filters offer three principal advantages 3 ; 2 . First, knockoff methods are intrinsically distribution-free, eliminating the need to explicitly train a classifier. Second, knockoff methods provide strong theoretical guarantees—–an attribute that remains relatively uncommon in NLP literature. Third, the framework can extend many logit-based detection methods to the rewrite setting. This construction enables existing logit-based detectors to obtain FDR control. 2 Methodology Our framework consists of three steps: 1. Generate a knockoff by rewriting. 2. Compute a comparison statistic between the original and knockoff. 3. Apply the knockoff filtering rule to select texts classified as human-written while controlling FDR. 2.1 Generate Knockoffs We consider a collection of texts T_{1},T_{2},\ldots,T_{n} , among which N_{0} is generated by AI. We assume that T_{1},T_{2},\cdots T_{n} are independently generated. The objective is to identify human-written texts. Let H_{i} denote the indicator variable that equals 1 if T_{i} is generated by a human and 0 otherwise. For each text T_{i} , we prompt an LLM to generate a corresponding rewrite R_{i} . Prior work 27 ; 11 ; 24 ; 28 introduces a statistic s_{i}=f(T_{i},R_{i}) , which serves as a measure of distance between T_{i} and R_{i} . Both theoretical and empirical results in 27 demonstrate that larger values of s_{i} are associated with a higher likelihood that T_{i} was written by a human rather than generated by an LLM. We denote the resulting collection of statistics by \mathcal{S}={s_{1},\ldots,s_{n}} . To prove our theoretical results, we require the standard knockoff sign-flip (symmetry) condition. Lemma 1 gives a sufficient condition under which it holds for rewrite pairs. Specifically, Assumption 1 . \mathbb{P}(f(T_{i},R_{i})<-a\mid H_{i}=0)=\mathbb{P}(f(T_{i},R_{i})>a|H_{i}=0) for all a\in\mathbb{R}^{+} . This symmetry arises from an exchangeability property of rewrite pairs under the null hypothesis that the text is AI-generated. 2.2 Finding Threshold In the final step, we want to generate the threshold using \mathcal{S} . We let q be the target FDR. The threshold \tau is defined as \tau=\min_{c\in\mathbb{R}^{+}}\left\{\frac{\#\{s_{i}\leq-c\}+1}{\max\{\#\{s_{j}\geq c\},1\}}\leq q\right\} (1) larger values of s_{i} indicate human-written text, we classify T_{i} as human-written whenever s_{i}>\tau . 3 Theoretical Analysis The following lemma provides a sufficient condition under which Assumption 1 holds. This result is similar to definition 2 in 3 which inspires exchangeability conditions. The proof is provided in Appendix A . Lemma 1 . Let R_{i}\sim K(\cdot\mid T_{i}) be a stochastic kernel that generates rewrites for T_{i} and \mathbb{P}_{\text{LLM}}(\cdot) be the LLM for which T_{i} is sampled from. If \mathbb{P}_{\text{LLM}}(T_{i})K(R_{i}\mid T_{i})=\mathbb{P}_{\text{LLM}}(R_{i})K(T_{i}\mid R_{i}), Then for any anti-symmetric statistics, f(T_{i},R_{i})=-f(R_{i},T_{i}), \mathbb{P}(f(T_{i},R_{i})<-a)=\mathbb{P}(f(T_{i},R_{i})>a),\forall a\in\mathbb{R}^{+} When Assumption 1 holds only approximately—as is typical in practice since no rewriter exactly satisfies the exchangeability condition of Lemma 1—exact finite-sample FDR control is no longer guaranteed, so we monitor symmetry empirically using the frac+ diagnostic reported in Table 1 . Theorem 1 . Under Assumption 1, \mathcal{S}_{\tau}=\{i:s_{i}>\tau\} controls the usual FDR, \text{FDR}=\mathbb{E}\left(\frac{\#\{i\in\mathcal{S}_{\tau},H_{i}=0\}}{\max\{|\mathcal{S}_{\tau}|,1\}}\right)\leq q. The guarantee holds in finite samples. The proof follows the knockoff filtering argument of 3 ; 2 , adapted to the rewrite detection setting. The complete proof is presented in Appendix A . 4 Experiments Figure 1: Distribution of knockoff statistics s_{i} for human (alternative, blue) and AI-generated (null, red) texts in representative high-symmetry cases, one per rewriting method: Religious/GPT-3.5T (Likelihood, frac + = 0.510, KS- p = 0.988), PersonalCommunication/GPT-4o (IMBD, frac + = 0.510, KS- p = 0.713), and Sports/GPT-3.5T (L2D, frac + = 0.510, KS- p = 0.068). The dashed vertical line marks s_{i}=0 . The near-symmetric null distribution around zero confirms that Assumption 1 holds in practice for these cases. Table 1: Empirical validation of the symmetry condition (Assumption 1). We report the fraction of knockoff statistics s_{i}>0 (frac + ) and the KS test p-value for symmetry of s_{i} , averaged across domains. Values close to 0.5 for frac + indicate the symmetry assumption holds for AI texts. The cross-domain rows show results when the null mean is borrowed from other domains rather than estimated within-domain. Method GPT-3.5T GPT-4o Gemini Llama-70B frac + KS- p frac + KS- p frac + KS- p frac + KS- p L2D 0.308 0.000 0.302 0.006 0.312 0.024 0.287 0.002 Likelihood 0.499 0.070 0.517 0.078 0.509 0.114 0.513 0.059 IMBD 0.466 0.042 0.475 0.040 0.483 0.070 0.469 0.043 Table 2: Empirical FDR control and detection power averaged across domains when the null