Lossless Anti-Distillation Sampling

Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He, [2pt] Peking University · 2026

Read Lossless Anti-Distillation Sampling by Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He, [2pt] Peking University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18829.txt.

Introduction

Zibo Diao and five colleagues defend frontier models against distillation without changing what users see. Harvested outputs let rivals train competing models cheaply, while existing defences either degrade quality or rely on detection that multi-account querying defeats. Their Lossless Anti-Distillation Sampling couples randomness across accounts while keeping each single account experience unchanged. Similar queries spread across accounts then yield dependent data that weakens the distilled model. Tests on image, maths, and code tasks confirm the defence while preserving exact fidelity for single users.

Catalog record

CREATORZibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He, [2pt] Peking University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85579
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18829

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lossless Anti-Distillation Sampling Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He, [2pt] Peking University Abstract Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model of its own at drastically lower cost. Existing defenses either rely on modifying the model’s outputs, thereby sacrificing response quality for benign users, or on behavioral detection methods, which can be readily circumvented by distributing queries across multiple accounts. In this work, we propose Lossless Anti-Distillation Sampling ( LADS ), a novel sampling scheme specifically designed to counter multi-account distillation while maintaining a lossless experience for benign users. Concretely, LADS derives the randomness underlying each generation from a private seed determined by the semantic content of the query and the number of times the user has queried the model. By construction, every benign user receives a response independently sampled from the original model at each visit, and thus experiences no distortion. In contrast, for a distiller, different accounts share latent randomness whenever their queries fall in the same semantic bucket. As a result, the harvested data becomes correlated, potentially reducing sample diversity and degrading generalization. Using uniform convergence theory, we show that LADS provably degrades the convergence rate of the distiller’s generalization gap relative to standard i.i.d. sampling in both unconditional and conditional generation settings. Experiments on image generation, mathematical reasoning, and code generation confirm that LADS substantially degrades the performance of distilled students while preserving exact statistical fidelity for individual users. Contents 1 Introduction Commercial AI evolves rapidly. Once a leading company releases a new frontier model, its competitors come under strong pressure to catch up. A well-known fast and low-risk approach is distillation, in which adversaries harvest large volumes of generated responses by querying the API and then use them to train their own models via supervised fine-tuning or reinforcement fine-tuning. Such a strategy is undesirable for service providers, as it undermines the technical advantage of original innovators and weakens incentives for continued investment in fundamental research. If the adversary collects data through a single account, ways to defend against it are relatively straightforward, since the provider can monitor account-level behavior (e.g., usage volume, query semantics) and identify anomalous activity. In practice, however, the more challenging threat arises from multi-account strategies, where the adversary controls hundreds of accounts, each designed to behave like a benign user. Existing defenses either degrade benign-user response quality through output-side perturbations ( Orekondy et al. 2019 ; Ma et al. 2021 ; Savani et al. 2025 ) , attribute distillation only post hoc through watermarking ( Zhao et al. 2023 ; Sander et al. 2024 ; Xu et al. 2026 ) , or rely on accurate detection models that are readily circumvented under the multi-account distillation setting ( Juuti et al. 2019 ; Chen et al. 2025 ; Mei et al. 2025 ) . This highlights the need for a defense that is lossless for benign users while remaining effective against multi-account adversaries. In this paper, we introduce Lossless Anti-Distillation Sampling ( LADS ), a server-side mechanism that, rather than altering the output distribution, controls the realization of the noise variable \epsilon underlying generation. Concretely, for each generation request q , LADS derives the random seed through a private injective seed generator \mathrm{SG}(h(q),a[h(q)]) , where h is a private clustering function that maps each query to a semantic bucket, and a is the user’s per-bucket visit counter. Generation is then performed using the resulting seed. It is straightforward to verify that this construction is lossless for benign users: for any individual account, the distribution of responses at each time step is identical to that of the original model, while responses across time remain independent. Consequently, benign users observe no distortion. At the same time, semantically similar queries submitted across different accounts with the same per-bucket access count are deterministically routed to the same seed. As a result, the samples collected by a multi-account distiller become correlated rather than independent. Such a correlation can potentially reduce sample diversity and consequently hurt the generalization of the student model. We support this claim through both a uniform convergence analysis and empirical experiments, as summarized below. Figure 1: Overview of our approach. Under standard sampling, each query is associated with an independently drawn noise variable, yielding i.i.d. samples in multi-account distillation. In contrast, LADS couples the underlying randomness across accounts based on the semantic content of queries and users’ query counts. Consequently, the collected samples of a multi-account distiller become correlated, reducing their effective diversity, leading to worse generalization ability. Theoretical Guarantee of LADS . We formalize the multi-account threat as a distiller controlling K accounts, each issuing T queries, and minimizing the standard empirical risk defined over the collected data. We consider two main settings: a warm-up unconditional generation setting ( LADS-simple ), where the query q is omitted, and a conditional generation setting ( LADS ), which is more representative of real-world distillation scenarios. Through a uniform convergence analysis, we show that the generalization gap decreases as \mathcal{O}(1/\sqrt{KT}) under standard i.i.d. sampling, but degrades to \mathcal{O}(1/\sqrt{T}) under LADS-simple (Proposition 3.2 ); under LADS , the generalization gap behaves similarly, depending on the size of the task-associated query space (Theorem 3.3 ). In other words, by introducing cross-account coupled noise, additional dummy accounts no longer provide sufficiently informative training signals. As a result, the student model’s test error cannot be reduced at the same rate by simply scaling up the number of accounts. Empirical Validation Across Language and Image Scenarios. For image generation, we deploy LADS on diffusion-based EDM2 models ( Karras et al. 2024 ) . Specifically, we use EDM2-Large (777M) and EDM2-XS (125M) as teachers and EDM2-XXS (31M) as the student. To model the multi-account setting, we distribute ImageNet ( Deng et al. 2009 ) class queries across 50 accounts. Under this setup, we introduce a noise-mixing mechanism that interpolates between standard i.i.d. sampling and the full LADS scheme, while preserving the exact per-request noise distribution \mathcal{N}(0,I) . For language generation, we use Qwen3.5-397B-A17B-FP8 ( Qwen Team 2026 ) as the teacher, and Llama-3.1-8B ( Grattafiori et al. 2024 ) , DeepSeek-Math-7B-Base ( Shao et al. 2024 ) , and Qwen2.5-7B ( Qwen et al. 2025 ) as students. We evaluate distillation quality on MATH ( Hendrycks et al. 2021 ) , GSM8K ( Cobbe et al. 2021 ) and HumanEval ( Chen et al. 2021 ) . Across both scenarios, we observe that for the student models, FID, validation loss, and task accuracy degrade under LADS , while samples observed by benign users remain of high quality — confirming that LADS achieves the multi-account distillation defense predicted by our theory under a strict losslessness guarantee. 2 Related Work Perturbing Distribution. A natural defense strategy is to alter the model’s output so that it remains useful to benign users but becomes harder for a student model to distill. Orekondy et al. 2019 proposed Prediction Poisoning, which perturbs the returned posterior probabilities to maximally distort attacker’s gradient signal while keeping perturbations small enough to preserve utility for benign users. Kariyappa and Qureshi 2020 introduced Adaptive Misinformation, which returns incorrect predictions for out-of-distribution queries while preserving

Read the complete text