RoFL: Robust Fingerprinting of Language Models

Yun-Yun Tsai, Chuan Guo, Junfeng Yang, Laurens van der Maaten · 2025

Read RoFL: Robust Fingerprinting of Language Models by Yun-Yun Tsai, Chuan Guo, Junfeng Yang, Laurens van der Maaten (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12682.txt.

Introduction

Owners of large language models need a way to prove a model is theirs, even after others fine-tune it. Yun-Yun Tsai and co-authors present a 2025 method, called RAFP in the text itself, that marks models through rare prompts.

The key insight is simple: fine-tuning mostly reshapes common, everyday responses. Rare prompts get little training signal, so model replies to them stay stable across later changes. The method builds its marks by searching for such rare prompts, without changing model weights at all. Tests across four model families show the marks survive fine-tuning, compression, and changed settings. The approach beat prior marking methods when the model could only be queried, not opened.

Catalog record

CREATORYun-Yun Tsai, Chuan Guo, Junfeng Yang, Laurens van der Maaten
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12682

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoFL: Robust Fingerprinting of Language Models Yun-Yun Tsai, Chuan Guo, Junfeng Yang, Laurens van der Maaten Abstract AI developers are releasing large language models (LLMs) under a variety of different licenses. Many of these licenses restrict the ways in which the models or their outputs may be used. This raises the question how license violations may be recognized. In particular, how can we identify that an API or product uses (an adapted version of) a particular LLM? We present a new method that enable model developers to perform such identification via fingerprints : statistical patterns that are unique to the developer’s model and robust to common alterations of that model. Our method permits model identification in a black-box setting using a limited number of queries, enabling identification of models that can only be accessed via an API or product. The fingerprints are non-invasive: our method does not require any changes to the model during training, hence by design, it does not impact model quality. Empirically, we find our method provides a high degree of robustness to common changes in the model or inference settings. In our experiments, it substantially outperforms prior art, including invasive methods that explicitly train watermarks into the model. Code is available at https://github.com/yunyuntsai/robust_llm_fingerprint 1 Introduction The widespread commercial application of large language models (LLMs) has led to concerns about safeguarding the intellectual property (IP) of model developers. A particular concern is model theft in which an adversary obtains access to a model and uses it in ways that violate the model’s license ( OWASP 2025 ) . It can be relatively easy for an adversary to steal a model, for example, when proprietary models are deployed on the premises of a customer or when model weights are released publicly under a bespoke license. The substantial development costs of modern LLMs imply that adversaries may be incentivized to commit model theft. Model theft can be difficult to detect. Similarity detection approaches ( Pei et al. 2021 ; Gabel et al. 2010 ) used to detect theft in traditional software do not work well for LLMs ( Yao et al. 2024 ) , as these models consist of floating-point weights that can easily be adapted via finetuning, pruning, or quantization. Moreover, an adversary can deploy a stolen model via an API or product, preventing the original model developer from accessing those weights. Combatting model theft thus requires the development of model identification methods that are robust against common model changes and require solely black-box access to the model under investigation. This paper introduces Robust Fingerprinting of Language models ( RoFL ), a new method that enables robust black-box identification of LLMs via fingerprints. RoFL fingerprints are statistical patterns in the output of an LLM that are unique to that model and robust against other common model changes (such as finetuning, pruning, or quantization). They consist of a particular user prompt x and the corresponding model response y . Fingerprints are unique (with high probability): given prompt x , all models of the same lineage consistently generate response y , whereas other models yield different responses. As a result, the presence of these fingerprints in a model can be verified via black-box access to the model under a small number of queries. RoFL enables model developers to verify model ownership via standard cryptographic commitments: before deploying a model, the developer uses RoFL to generate a set of fingerprints, stores them securely, and publishes cryptographic hashes of the fingerprints. In cases of suspected model theft, the model developer can first investigate the suspicious model by generating new fingerprints on-the-fly to query the API without exposing the secret fingerprints. If this produces evidence of model theft, the developer can prove model ownership by revealing a previously committed fingerprint and demonstrating that the model under investigation contains this fingerprint. While RoFL does not permit formal security proofs because it lacks a formal definition of robustness, our empirical security analyses on modern LLMs demonstrate that RoFL is effective under common changes to the model, including model finetuning, changes in the model’s system prompt, and quantization. RoFL also protects against fingerprint race attacks in which adversaries publish valid alternative fingerprints after model release in an attempt to claim the model ownership. 2 Model Identification with RoFL We first introduce the threat model we consider in this study, present an overview of the general RoFL scheme for model identification, and perform a security analysis of that scheme. 2.1 Threat Model We assume a scenario in which an adversary aims to steal an LLM for economic gain, for example, by selling products based on the stolen model that violate the original model developer’s license terms. To secure its economic gain, the adversary seeks to hide the model theft from the model developer. In order for an attack to bring economic value to the adversary, the attack must: (1) require limited resources to be executed and (2) not alter the original model so much that its value is destroyed. Based on this scenario, we identify four main limitations of the adversary: 1. Limited resources. The adversary cannot use compute resources that match or exceed those used to train the original model. Such attacks do not have economic value for the adversary, as they could have simply trained their own model. 2. Model training integrity. The adversary cannot tamper with the (pre)-training process of the original model. They cannot alter the training data, the training algorithm, or the training infrastructure. 3. Common model changes. The adversary is restricted to common model changes that are designed to maintain or increase the economic value of the model. These changes include supervised fine-tuning, low-rank adaptation (LoRA), system prompt engineering, quantization, and tuning of inference hyperparameters (e.g., for specific productions) by an adversary. It is trivial to remove fingerprints from a model – for example, by randomly re-initializing all the model weights – but such model changes would destroy the economic value of the stolen model. 4. Unfiltered black-box access. The adversary provides black-box access to the actual LLM, without applying additional pre-processing filters to the input or post-processing filters to the output. Admittedly, this threat model lacks a formal definition of what model changes are permitted. As a result, RoFL does not permit formal theoretical proofs of security. 2.2 Overview of the Scheme We denote the original LLM by \mathcal{M}_{\theta} , where \theta represents the model weights. We denote adapted versions of this LLM by \mathcal{M}_{\theta_{u}} . The goal of model identification is to verify that \mathcal{M}_{\theta_{u}} is, indeed, an adapted version of \mathcal{M}_{\theta} . Our model identification scheme involves two functions: (1) a function, \gen(\mathcal{M}_{\theta})=F , that generates a fingerprint for model \mathcal{M}_{\theta} ; and (2) a function, \verify(F,\mathcal{M}) , that decides whether or not F is a fingerprint of model \mathcal{M} . The first function is used to generate and (cryptographically) commit to fingerprints prior to the release of the original model, whereas the second function is used to investigate suspected model theft after model release. In order to create an effective model identification scheme, these two functions must have the following properties: 1. Effective. Model identification using \verify(F,\mathcal{M}_{\theta}) should succeed with high probability. (See Table 2 ) 2.

Read the complete text