Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective

Hao Fang ∗, Tianyi Zhang ∗, Tianqu Zhuang, Jiawei Kong, Kuofeng Gao, Bin Chen, Leqi Liang, Shu-Tao Xia, Ke Xu · 2026

Read Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective by Hao Fang ∗, Tianyi Zhang ∗, Tianqu Zhuang, Jiawei Kong, Kuofeng Gao, Bin Chen, Leqi Liang, Shu-Tao Xia, Ke Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03396.txt.

Description

A company that serves a language model through an API can lose its investment when rivals distill the model's answers into a copy. Hao Fang and 8 co-authors study how to resist that theft in a 2026 paper. They focus on logit-based distillation, where attackers learn from the model's output scores rather than its text alone.

The authors measure the stealable information with conditional mutual information between the teacher's scores and the input, given the true labels. Their defense learns a transformation that scrubs that information from the outputs while keeping answers useful. Experiments across several models and strong distillation attacks show the protected outputs teach far less, while normal task accuracy stays intact.

Catalog record

CREATORHao Fang ∗, Tianyi Zhang ∗, Tianqu Zhuang, Jiawei Kong, Kuofeng Gao, Bin Chen, Leqi Liang, Shu-Tao Xia, Ke Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03396

Browse the full library · About the Tower of Babel