CTPD: Cross Tokenizer Preference Distillation

Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen · 2026

Read CTPD: Cross Tokenizer Preference Distillation by Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11865.txt.

Introduction

Truong Nguyen, Phi Van Dat, Ngan Nguyen, and their co-authors move preference lessons between models that split text in unlike ways. Distillation lets a small student learn from a big teacher. It is much used in training, but less for aligning models with human taste.

The cross-tokenizer case is the real world case, since models rarely share a text split. That split clash has blocked fine, direct transfer of taste data. Their method, named CTPD in the work, works past that clash. It could let small models gain taste lessons from any strong teacher.

Catalog record

CREATORTruong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CTPD: Cross Tokenizer Preference Distillation Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen Abstract While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplored, particularly in the more realistic cross-tokenizer setting. The incompatibility of tokenization schemes between teacher and student models has largely prevented fine-grained, white-box distillation of preference information. To address this gap, we propose Cross-Tokenizer Preference Distillation (CTPD), the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers. CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student tokens to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher’s preferences in a DPO-style objective. Our theoretical analysis grounds CTPD in importance sampling, and experiments across multiple benchmarks confirm its effectiveness, with significant performance gains over existing methods. These results establish CTPD as a practical and general solution for preference distillation across diverse tokenization schemes, opening the door to more accessible and efficient alignment of language models. Code — https://github.com/dinhtruongng/CTPD 1 Introduction Aligning Large Language Models (LLMs) with human values and preferences has become a cornerstone of modern AI research. This alignment aims to guide LLMs to generated outputs that are not only fluent but also beneficial, non-harmful, and consistent with intricate human norms. While early efforts relied on Reinforcement Learning from Human Feedback (RLHF) ( 4 ) , recent methods like Direct Preference Optimization (DPO) ( 18 ) and its variants offer more stable and computationally efficient alternatives, proving highly effective in creating state-of-the-art, user-aligned models. The effectiveness of preference alignment has been primarily demonstrated on large-scale, proprietary language models. However, the substantial computational requirements and the closed-source nature of these models pose significant barriers to accessibility and broad adoption, particularly in resource-constrained settings. In contrast, small language models (SLMs) offer a more practical alternative in such contexts but face notable challenges in achieving alignment comparable to that of larger models, largely due to their limited representational capacity. This often leads to an alignment tax after RLHF training, where their broad task performance is negatively impacted ( 1 ) . Knowledge distillation (KD) ( 10 ) offers a promising solution, where a smaller student model learns from a larger, pre-aligned teacher. This approach is efficient, as the costly alignment process is performed only once by the teacher. While black-box KD methods use only teacher output text, white-box methods leverage richer internal signals like logits for more fine-grained supervision. However, white-box distillation faces a critical obstacle: the cross-tokenizer problem. Teacher and student models often use different tokenizers, leading to incompatible logit distributions and preventing direct token-level knowledge transfer. Although knowledge distillation has been extensively studied in the contexts of pre-training and instruction tuning ( 26 ; 3 ; 7 ) , its application to the critical task of aligning language models with human preferences remains relatively underexplored. To date, only a single work ( 8 ) has investigated white-box distillation in this setting, and it was restricted to a simplified scenario where the teacher and student share an identical tokenizer. Importantly, the more realistic and challenging case of cross-tokenizer distillation for preference alignment has received little to no attention in the literature. Given the abundance of high-performing large language models (LLMs) with varying architectures and tokenization schemes that could serve as teacher models, advancing cross-tokenizer distillation techniques for human preference alignment is crucial to fully exploit their capabilities. However, existing approaches designed for cross-tokenizer distillation in pretraining or finetuning ( 26 ; 3 ; 7 ) are not directly applicable to this setting. These methods are primarily tailored to align the final-layer logits of teacher and student models for general-purpose learning tasks and do not address the specific challenges posed by preference-based supervision. To bridge this gap, we propose Cross-Tokenizer Preference Distillation (CTPD) which is the first unified framework that enables the transfer of human-aligned behavior from a high-capacity teacher model to a smaller student model. CTPD is motivated by the observation that, while the tokenizations used by teacher and student models may differ syntactically, both ultimately encode the same underlying natural language substrings. By projecting the teacher’s supervision signals onto the student’s tokens through precisely aligned character-level spans and redefining the DPO objective accordingly, CTPD enables fine-grained white-box supervision even in the presence of heterogeneous tokenizers. Concretely, CTPD comprises three key components: 1. Aligned Span Projection: CTPD constructs a dynamic lattice to partition input sequences into aligned spans—pairs of teacher and student token subsequences that correspond to identical character-level intervals. This alignment allows us to compute projected log-probabilities over the student vocabulary without introducing any additional learnable parameters. 2. Cross-tokenizer Importance Weighting: Building on this alignment, we extend the TIS-DPO framework ( 14 ) to the cross-tokenizer setting. Token-level importance weights from the teacher are aggregated within each aligned span and transferred to the corresponding student spans, resulting in span-specific weights that enhance credit assignment across mismatched token spaces. 3. Teacher-Anchored Reference: CTPD adopts the teacher model itself as the reference distribution \pi_{\text{ref}} in the DPO-style objective. Through the span projection mechanism, the student can approximate the teacher’s log-probabilities over its own tokens, enabling the definition of a teacher-anchored DPO-style objective. This loss function retains the structure of standard DPO but naturally accommodates heterogeneous tokenizers, allowing the student to benefit directly from the teacher’s preferences. CTPD addresses the core problem cross-tokenizer in preference distillation, providing the first practical solution for full-resolution white-box preference transfer. By decoupling alignment from tokenizer compatibility, CTPD makes it feasible to distill sophisticated alignment behaviors from any powerful teacher into any smaller student, thereby facilitating the development of efficient and robustly aligned language models. Especially, we provide a theoretical foundation for the CTPD framework based on importance sampling, which enhances its reliability and provides deeper insights into the dynamics of cross-tokenizer preference distillation. We conduct extensive experiments to demonstrate significant improvements of CTPD across multiple benchmarks. Furthermore, comprehensive ablation study and analysis confirm the effectiveness of our weighting strategy, the aligned span and teacher-anchored approaches, providing valuable insights into the underexplored space of preference distillation. 2 Related work 2.1 Preference Alignment The prevailing approach for human alignment is Reinforcement Learning from Human Feedback (RLHF) ( 4 ; 22 ; 17 ) . This multi-stage process, which involves training a reward model and then optimizing a policy with reinforcement learning (e.g., PPO ( 21 ) ), has shown empirical success but is often criticized for its training complexity and instability ( 18 ; 1 ; 16 ) . To mitigate these issues, Direct Preference Optimization (DPO) ( 18 ) was introduced as a more direct method that bypasses the explicit reward modeling and RL loop. DPO reframes the problem as a simple binary classification task on preference pairs, enabling stable training via a simple objective and demonstrating performance competitive with PPO-based RLHF ( 18 ) . Building on DPO’s success, several extensions have emerged. Of particular relevance to our work is Token-level Importance-Sampling DPO (TIS-DPO) ( 14 ) , which addresses DPO’s uniform treatment

Read the complete text