Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model

Runheng Liu, Heyan Huang, Xingchen Xiao, Zhijing Wu · 2026

Read Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model by Runheng Liu, Heyan Huang, Xingchen Xiao, Zhijing Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21223.txt.

Introduction

Runheng Liu and three colleagues detect machine-written text without training, using implicit reward models derived from public instruction-tuned and base models. Their IRM method needs neither preference collection nor task-specific fine-tuning, unlike earlier reward-based detectors. On the DetectRL benchmark it tops both zero-shot and supervised rivals, offering a lightweight check against misuse of fluent generated text.

Catalog record

CREATORRunheng Liu, Heyan Huang, Xingchen Xiao, Zhijing Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70100
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model Runheng Liu, Heyan Huang, Xingchen Xiao, Zhijing Wu Abstract Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their ability to generate human-like text has raised concerns about potential misuse. This underscores the need for reliable and effective methods to detect LLM-generated text. In this paper, we propose IRM, a novel zero-shot approach that leverages Implicit Reward Models for LLM-generated text detection. Such implicit reward models can be derived from publicly available instruction-tuned and base models. Previous reward-based method relies on preference construction and task-specific fine-tuning. In comparison, IRM requires neither preference collection nor additional training. We evaluate IRM on the DetectRL benchmark and demonstrate that IRM can achieve superior detection performance, outperforms existing zero-shot and supervised methods in LLM-generated text detection. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks [ 6 , 19 ] . In particular, their ability to follow instructions and generate human-like content has made them essential tools in various real-world applications. However, this trend raises concerns about the misuse of LLMs, such as generating fake news [ 37 ] and plagiarism [ 4 ] . Since LLM-generated text can be indistinguishable from human-written text [ 9 ] , it is necessary to develop effective methods for detecting such content. Detection methods can generally be categorized into two paradigms: supervised and zero-shot approaches. Supervised methods typically train a classifier using both human-written and LLM-generated texts. However, such classifiers often fail to generalize well to texts generated by unseen LLMs [ 1 , 28 ] . In contrast, zero-shot methods provide an alternative with more generalization ability and without the need for training [ 18 ] . These methods leverage various LLMs and exploit statistical classification metrics to assess how likely a text is to be generated by an LLM, such as the average log likelihood or the log-rank of each tokens within the detected text [ 11 , 23 , 9 ] . However, in real-world the source LLM of the detected text is often unknown, thus these zero-shot methods have to rely on a proxy LLM and suffer from a performance drop. This highlights the need to identify model-agnostic metrics for detection. ReMoDetect [ 14 ] has recently proposed using the reward score—the output of a reward model that measures the extent to which a text aligns with human values—as a detection metric. The rationale is that powerful LLMs are typically trained to obtain high reward score, making this signal potentially model-agnostic. Specifically, ReMoDetect constructs a preference dataset from human and ChatGPT responses to the same prompt, and fine-tunes a pre-trained reward model for adaptation to the detection task. However, this fine-tuning inherits the drawbacks of supervised methods. As shown in Figure 1 (b), when ReMoDetect is evaluated on texts generated by unseen LLMs, there remains substantial overlap between the two score distributions of human-written and LLM-generated texts, suggesting the limited generalization ability. Figure 1 : Distributions of reward scores for human-written and LLM-generated texts from the multi-LLM sub-task of the DetectRL benchmark. The source LLMs include GPT-3.5-turbo, PaLM-2-Bison, Claude-Instant, and LLaMA-2-70B. (a) shows distributions under RM-deberta-v3-large-v2, which is a pre-trained reward model. (b) shows distributions under ReMoDetect, which is initialized from RM-deberta-v3-large-v2 and fine-tuned on task-specific preference dataset. (c) illustrates distributions under an implicit reward model derived from Llama-3.2-1B family without additional training. In this paper, we propose IRM, a novel zero-shot method that leverages implicit reward models for LLM-generated text detection. Such implicit reward models can be derived from publicly available instruction-tuned and base models. Given a detected text y , an instruction-tuned model \pi_{\theta} and a base model \pi_{\text{ref}} , the implicit reward score of y is formulated as r(y)=\log\frac{\pi_{\theta}(y)}{\pi_{\text{ref}}(y)} . This score serves as the detection metric in our method, where a higher score indicates that the input text is more likely to be generated by LLMs. This formulation is built on Direct Preference Optimization (DPO) [ 22 ] , which provides theoretical support for parameterizing implicit reward models. Compared to ReMoDetect, IRM requires neither preference collection nor additional training. As shown in Figure 1 (c), the two distributions under IRM are more separable than those in (b), suggesting that our zero-shot metric transfers well to unseen LLMs. We evaluate IRM on the DetectRL benchmark [ 30 ] and demonstrate that IRM is an effective zero-shot method for LLM-generated text detection. Notably, with Llama-3.2-1B family, IRM achieves an average score of 91.77%, surpassing previous zero-shot baselines such as Log-Likelihood (77.46%) and Log-Rank (77.42%), as well as Binoculars (87.67%), a method which also utilizes two LLMs during detection. Furthermore, IRM even outperforms the supervised reward-based method, ReMoDetect (85.86%). These results underscore IRM’s potential as a effective zero-shot detector for real-world applications. 2 Preliminaries The mainstream alignment approach for LLMs is Reinforcement Learning from Human Feedback (RLHF) [ 24 , 20 ] , which relies on a reward model as a proxy for human preferences. In this section, we first introduce the concept of reward modeling, followed by an overview of the RLHF process. Reward modeling. Typically, the reward model is initialized from a pre-trained language model with a additional linear head that outputs a scalar value. It is trained on a preference dataset, where human annotators compare multiple responses to the same prompt and rank them based on quality such as helpfulness or harmlessness. Formally, the preference is modeled using the Bradley-Terry (BT) model [ 5 ] : p(y_{1}\succ y_{2}|x)=\frac{\exp(r(x,y_{1}))}{\exp(r(x,y_{1}))+\exp(r(x,y_{2}))}, (1) where r denotes the reward model, x is the prompt, and y_{1} , y_{2} are two candidate responses. Given a prompt–response pair (x,y) , the reward model outputs a scalar score r(x,y) , where a higher score indicates stronger alignment with human values. RLHF. During RLHF, the learned reward model is used to provide feedback for optimizing the language model. The LLM is framed as a policy model \pi_{\theta} , which generates responses conditioned on input prompts. The policy is optimized to maximize its expected reward while remaining close to a reference model \pi_{\text{ref}} , typically the original base LLM prior to alignment. This leads to the following optimization objective: \max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y|x)}[r(x,y)]-\beta\mathbb{D}_{\text{KL}}[\pi_{\theta}(y|x)\|\pi_{\text{ref}}(y|x)], (2) where x denotes the prompt, y the response, r the reward model, and \beta a hyper-parameter controlling the strength of KL regularization. 3 Method We introduce IRM, a zero-shot detection method for LLM-generated text detection. We formalize the approach and show how it can be instantiated using publicly available instruction-tuned and base models. Figure 2 : IRM leverages open-source instruction-tuned and base models to construct an implicit reward model. The resulting reward score is used as a detection metric, where a higher score indicates a higher probability that y is generated by an LLM. Implicit reward model for detection. Direct Preference Optimization (DPO) [ 22 ] offers a closed-form solution to the RLHF objective in Eq. 2 : \pi^{*}_{\theta}(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right), (3) where Z(x)=\sum_{y}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right) is the partition function that normalizes the distribution. By rearranging this equation, we can express the reward as a function of the policy and reference models, yielding an implicit reward model: r(x,y)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x). (4) For simplicity, we let x=\varnothing and y be the detected text. In this case, the term

Read the complete text