Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
Charles Goddard, Fernando Fernandes Neto · 2025
Read Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit by Charles Goddard, Fernando Fernandes Neto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06607.txt.
Introduction
Charles Goddard and Fernando Fernandes Neto swap tokenizers between language models in a 2025 paper, without any retraining. A tokenizer splits text into tokens, and models normally cannot change theirs after training. Their method rebuilds each new token's vector as a sparse blend of tokens the two models share.
The blend is found in the donor model's space, then the same weights rebuild the vector in the base model's space. Across two hard swaps, between big model families, their approach preserved zero-shot skill best, beating several earlier methods. Number tokenization mismatches emerged as the main threat to math skill. The tool ships inside a public model-merging kit.
Catalog record
| CREATOR | Charles Goddard, Fernando Fernandes Neto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit Charles Goddard, Fernando Fernandes Neto Abstract We present a training-free method to transplant tokenizers in pretrained large language models (LLMs) by reconstructing unseen token embeddings via Orthogonal Matching Pursuit (OMP). Specifically, we approximate each out-of-vocabulary token as a sparse linear combination of shared tokens, in two phases: first, compute each new token’s representation in the donor embedding space with a small dictionary of shared anchor tokens, then transfer these same sparse coefficients back into the base model’s embedding space. On two challenging cross-tokenizer tasks—Llama → \to Mistral NeMo (12B) and Qwen → \to Llama (1B)—we show that OMP achieves best zero-shot preservation of the base model’s performance across multiple benchmarks, while other zero-shot approaches degrade significantly. Compared to baselines (zero-init, mean-init, and existing approaches like WECHSEL, FOCUS, ZETT), OMP consistently achieves the best overall performance, effectively bridging large tokenizer discrepancies without gradient updates. Our analysis further identifies mismatched numerical tokenization schemes as a critical challenge for preserving mathematical reasoning capabilities. This technique enables direct reuse of pretrained model weights with new tokenizers, facilitating cross-tokenizer knowledge distillation, speculative decoding, ensembling, merging, and domain-specific vocabulary adaptations. We integrate our method into the open-source mergekit-tokensurgeon tool for post hoc vocabulary realignment. 1 Introduction Large language models (LLMs) are typically constrained by the tokenizer chosen during pretraining, which defines a fixed vocabulary for processing text. However, these vocabularies are not universal; they may sub-optimally tokenize other languages, dialects, or domains [ 31 , 9 ] . While replacing a model’s tokenizer post-training is highly desirable in many scenarios, doing so without performance degradation remains a formidable technical challenge. The core difficulty lies in embedding initialization for new tokens. Naive initialization can significantly degrade performance [ 26 , 12 ] , while continuing pretraining to learn new embeddings can be prohibitively expensive. [ 4 , 2 ] Meanwhile, existing zero-shot heuristics yield uneven results and can cause large performance drops on tasks like question answering or reasoning [ 9 , 16 ] , especially when fundamental representational structures, like numerical tokenization schemes, differ between models. This vocabulary mismatch creates practical barriers in several important scenarios. In speculative decoding [ 22 , 1 ] , a small model proposes partial outputs for a larger model to verify, requiring aligned tokenizers for direct logit comparison. Knowledge distillation [ 17 ] similarly requires matched vocabularies when transferring token-level supervision from teacher to student [ 3 ] . Model ensembling or merging [ 13 ] presupposes that tokens map to the same embedding dimensions, with different tokenizers blocking direct output combination. An effective method for post hoc tokenizer transfer—without retraining—would unlock these capabilities, greatly expanding the uses for existing models. In this paper, we propose a fully training-free approach to “tokenizer transplantation,” where we reconstruct embeddings for the new vocabulary in the base model’s embedding space using Orthogonal Matching Pursuit (OMP). Our key insight is to represent each new token’s donor embedding as a sparse combination of shared token embeddings. Transplantation then becomes straightforward: we replicate those same sparse coefficients in the base model’s embedding space. This yields a new vocabulary embedding matrix aligned with the new tokenizer but consistent with the base model ’s embedding geometry. We evaluate OMP-based transplantation in two cross-tokenizer experiments: Llama \to Mistral NeMo (12B) and Qwen \to Llama (1B). In both scenarios, OMP preserves model performance across classification, reasoning, and perplexity benchmarks far better than simple heuristics (zero, mean) or other zero-shot token-initialization approaches. We integrate this strategy into the mergekit-tokensurgeon open-source toolkit, enabling immediate usage in cross-tokenizer knowledge distillation, speculative decoding pipelines, or domain-vocabulary expansions, with no retraining required. Contributions. 1. We propose a training-free tokenizer transplant method using Orthogonal Matching Pursuit in embedding space. 2. We demonstrate superior zero-shot performance over existing approaches, evaluated on Llama \to Mistral NeMo and Qwen \to Llama tasks. 3. We identify and analyze the significant negative impact of mismatched numerical tokenization schemes on mathematical reasoning performance, highlighting a key limitation of zero-shot transplantation. 4. We highlight how OMP-based transplantation resolves key bottlenecks in cross-tokenizer use cases (e.g., teacher-student distillation, speculative decoding, or domain expansions). 5. We provide an efficient incremental-QR OMP implementation and release it in mergekit-tokensurgeon for reproducibility. 2 Related Work Embedding Initialization for New Vocabularies. Assigning embeddings to new or unseen tokens is a well-explored problem in model adaptation. Naive (random/zero) initialization typically degrades performance substantially [ 16 ] . More sophisticated approaches often compute embeddings by averaging or combining subword pieces from the old vocabulary [ 26 , 12 ] . For example, Minixhofer et al. 2022 propose WECHSEL, which uses bilingual word embeddings to map subword pieces from one language to another, then continues pretraining to refine the new embeddings. Similarly, Dobler and de Melo 2023 (FOCUS) and Ostendorff and Rehm 2023 (CLPTransfer) base their initialization on overlapping tokens, weighting them to approximate the new token’s distribution. These methods yield better initial embeddings but often still rely on some continued training to recover the original model performance. In contrast, our method requires no further training for a high quality result. Additionally, OMP produces signed coefficients and therefore explores a larger linear subspace instead of restricting to the convex hull of anchors. Zero-Shot Tokenizer Transfer. ZeTT [ 27 ] trains a hypernetwork for arbitrary tokenizers, applicable zero-shot after meta-training. Other tools [ 18 , 13 ] copy or average existing embeddings. Our work aligns with these methods’ goals of minimal or no additional training, but uses Orthogonal Matching Pursuit (OMP) to anchor new token embeddings in a sparse set of existing embeddings. This approach yields more accurate approximations, especially with large vocabulary differences, and does not require a meta-trained hypernetwork . Sparse Approximation Techniques. Orthogonal Matching Pursuit (OMP) [ 30 ] is a standard technique for sparse reconstruction, used widely in signal processing and dictionary learning. While it has seen limited use in NLP [ 24 , 32 ] , it has not to our knowledge been adopted for cross-tokenizer embedding transfer. Our approach is the first to apply OMP as a solution to cross-tokenizer embedding alignment. 3 Methodology 3.1 Problem Setup and Notation Let \mathcal{M}_{\mathrm{base}} be a pretrained base model with vocabulary V_{\mathrm{base}} and embedding matrix E^{\mathrm{(base)}}\in\mathbb{R}^{|V_{\mathrm{base}}|\times d_{\mathrm{base}}} . We want to replace its vocabulary with V_{\mathrm{donor}} from another model \mathcal{M}_{\mathrm{donor}} , which has an embedding matrix E^{\mathrm{(donor)}}\in\mathbb{R}^{|V_{\mathrm{donor}}|\times d_{\mathrm{donor}}} . Note that d_{\mathrm{donor}} may differ from d_{\mathrm{base}} . We denote V_{\cap}=V_{\mathrm{base}}\cap V_{\mathrm{donor}} as the set of shared (overlapping) tokens. Case 1: Shared Tokens. If t\in V_{\cap} , we simply copy e_{t}^{\mathrm{(base)}} to the new embedding matrix E^{\mathrm{(new)}} . Case 2: Unseen Tokens. When t\notin V_{\mathrm{base}} , we have a donor embedding e_{t}^{\mathrm{(donor)}} but no corresponding e_{t}^{\mathrm{(base)}} . We must approximate e_{t}^{\mathrm{(base)}} by referencing anchor tokens that exist in both vocabularies. In other words: e_{t}^{\mathrm{(donor)}}\approx\sum_{j\in A}\alpha_{j}\,e_{j}^{\mathrm{(donor)}},\quad A\subseteq V_{\cap},\quad|A|\leq k, and we then place e_{t}^{\mathrm{(new)}}\;=\;\sum_{j\in A}\alpha_{j}\,e_{j}^{\mathrm{(base)}}. Here, k is a chosen sparsity level (e.g., 8, 32, 64), and A is the small set of anchor tokens chosen by an OMP solver in the donor’s embedding space. By applying those coefficients in the base model’s space, we create the new embedding e_{t}^{\mathrm{(new)}} . As only the unit-agnostic coefficient vector \alpha is transferred, there is no dependence on