Asking Back: Interaction-Layer Antidistillation Watermarks

Guang Yang, Amir Ghasemian, Fengchen Liu, Zhong Wang, Ninareh Mehrabi, Homa Hosseinmardi · 2026

Read Asking Back: Interaction-Layer Antidistillation Watermarks by Guang Yang, Amir Ghasemian, Fengchen Liu, Zhong Wang, Ninareh Mehrabi, Homa Hosseinmardi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16462.txt.

Introduction

Yang and five others propose interaction-layer watermarks, moving the trace into interaction conduct. Finding sneaky copying is hard: the guard controls neither setup nor logits. The guard wraps the teacher with a system prompt inducing a marker: a follow-up question, a low-rate variant, or a restatement. An oblivious distiller takes it; watermarks transfer at up to 88.9 percent relative fidelity across 63 LoRA-distilled students. Under DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling and student-relative retention of 21-112 percent.

Catalog record

CREATORGuang Yang, Amir Ghasemian, Fengchen Liu, Zhong Wang, Ninareh Mehrabi, Homa Hosseinmardi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104537
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Asking Back: Interaction-Layer Antidistillation Watermarks Guang Yang, Amir Ghasemian, Fengchen Liu, Zhong Wang, Ninareh Mehrabi, Homa Hosseinmardi Abstract This work is motivated in part by a parallel genome-model study ( GenoTrace on GenomeOcean [ 44 ] ; App. N ), where a token-level distillation watermark suggested a broader question: can a trace survive distillation when its carrier is not token statistics, but interaction behavior? Detecting unauthorized knowledge distillation from a deployed language-model API is hard precisely because the defender controls neither the attacker’s training pipeline nor, in the API setting, the next-token logits. Existing defenses operate on the teacher’s output tokens, either by biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or by rewriting outputs after generation (defensive output rewriting). Recent work shows that an attacker who paraphrases harvested responses before training can strip these signals without losing the underlying knowledge—defeating the watermark while preserving the distillation. We propose interaction-layer antidistillation watermarks , which move the trace one layer higher, into the teacher’s interaction behavior : the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker—an explicit follow-up question, a low-frequency variant of it, or a declarative restatement. An oblivious distiller inherits the behavior in their student, and the defender later audits through black-box queries with a human-validated LLM-as-judge (Cohen’s κ = 0.84 \kappa{=}0.84 / 0.78 0.78 on strong/style rubrics). Across 63 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher ( 35,343 35{,}343 judged samples), behavioral watermarks transfer at 88.9 % 88.9\% (Gemma) / 80.9 % 80.9\% (OLMo) / 45.2 % 45.2\% (Qwen) relative fidelity (H1, H2); under non-adaptive DIPPER prompt paraphrasing, robustness decomposes into a teacher-self ceiling ( ∼ 66.4 % \sim\!66.4\% ) and a student-relative retention of 21 21 – 112 % 112\% , with one family (OLMo) preserving the watermark above the teacher itself (H3, F-Amp); low-density ( ∼ 20 % \sim\!20\% ) explicit and implicit declarative variants both transfer reliably above per-family baseline (H4, F-Style). An N = 20 N{=}20 in-lab study under a pre-registered Latin-square protocol shows all marker variants within 0.22 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon all support H5. The interaction layer is, on present evidence, a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses. 1 Introduction Frontier language models are increasingly served through paid black-box APIs, and unauthorized knowledge distillation against those APIs is now a mature commercial threat that has motivated a range of antidistillation 1 1 1 We use antidistillation in the broader sense of 40 —making unauthorized distillation auditable through a downstream-detectable mark—rather than the strict sense of 29 , where the teacher’s outputs are crafted to deliberately degrade the resulting student. defenses [ 29 , 24 , 19 ] . An adversary harvests prompt–response pairs from the teacher, fine-tunes a smaller student, and ships the student without permission. The defender’s central question is therefore not whether harvesting can be blocked —it cannot, in any deployable sense—but whether the defender can leave an auditable trace that survives distillation and that can later be verified through black-box queries to the suspected student. Existing defenses are organized by the layer at which the trace is embedded. The token-distribution layer family [ 15 , 1 , 4 , 29 ] perturbs the teacher’s next-token distribution. The model layer family [ 39 , 19 ] edits the teacher itself so that the harvested text carries a model-side mark. The reasoning-trace layer family [ 20 , 7 ] reorders or rewrites chain-of-thought traces. All three share an attack surface operating on the tokens the attacker harvests, and recent removal-attack work [ 24 , 17 , 27 , 2 ] shows that pre-distillation rewriting can detach a student from the inherited signal while preserving the knowledge transfer. The deeper structural gap is that even when a surface-text signal survives, it does not specify a distinctive, queryable behavior in the distilled student. Our starting point was a parallel genome-model study, where a KGW-style logit-bias watermark [ 15 ] was ported into the GenomeOcean [ 44 ] LoRA distillation pipeline and retained the radioactive-data property of [ 28 ] in a non-text regime ( GenoTrace ; App. N ). For conversational LLMs, the analogous carrier need not remain at the token layer: the defender also controls the interaction frame between user and model. That observation led to the asking-back markers studied here. We return to this connection in § 6.1 . In this work, we propose a fourth, complementary locus—the interaction layer (Figure 1 ). The defender wraps the teacher with a system prompt that intermittently induces a behavioral marker B : asking-back (a follow-up question), or its implicit cousins (a low-density variant, or a declarative restatement of the user’s request). An oblivious distiller who collects teacher outputs without access to the hidden trigger policy trains a student on a corpus in which B appears at some defender-chosen density \rho . Three properties make this useful: B requires no logit access (the trace lives in the response text); it is distillation-transferable because B is part of the next tokens the student is asked to imitate; and it operates at a semantic granularity partially insensitive to prompt-side paraphrasing (response-side rewriting is left to future work, Appendix, section O.1 ). We organize the experiments around five hypotheses any deployable interaction-layer watermark 2 2 2 We use watermark throughout in an operational sense—a detectable behavioral bias deliberately induced by the defender—rather than in the cryptographic or information-theoretic sense of an undetectable or provably secure scheme. must satisfy: H1–Learnability : a student trained on watermarked teacher outputs exhibits B at a rate substantially above its baseline-trained counterpart; H2–Cross-family generalization : the effect holds across pretrained model families with different architectures, vocabularies, and post-training intensities; H3–Paraphrase robustness : the watermark partially survives a non-adaptive prompt-side paraphrase (DIPPER on the user prompt), decomposed into a teacher-self ceiling and a student-relative component; H4–Density–detectability trade-off : even when B appears in only \sim\!20\% of training rows, the student-side detection rate is reliably above the per-family noise floor; H5–Stealth : a \sim\!20\% -density watermark is not perceived by users as a clear degradation of interaction quality. Our evaluation has two phases: a within-family Qwen3.5-9B-Instruct \to Qwen3.5-0.8B-Base case study at N{=}452 paired prompts under a deterministic regex (judge-free) and a 3\!\times\!7\!\times\!3=63 -student matrix under a Llama-3.3-70B-Instruct [ 9 ] teacher and a structured-output gpt-oss-120b [ 23 ] judge ( 35{,}343 judged samples). The three markers and the three student families are also described in sections § 3.2 and § 4 . Figure 1: The three layers of the Asking-Back threat model. Top: the attacker harvests prompt–response pairs from the teacher API; the defender has no control over which prompts. Middle: the defender wraps the teacher with a system prompt that injects a behavioral marker (asking-back, soft, or style-control) at a chosen density \rho ; harvested pairs carry the marker into the student. Bottom: the defender audits the suspect student through black-box queries with an LLM-as-judge whose rubric is blind to family and condition, and computes the relative transfer rate \tau_{\mathrm{rel}}(S,B)=\tau(S,B)/\tau(T,B) . Our contributions are threefold. (i) Problem framing. We formalize the interaction layer watermark (Definition 1 ) and three metrics: \tau,\tau_{\mathrm{rel}},R (Definitions 3

Read the complete text