AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation

Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, Roi Reichart · 2025

Read AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation by Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, Roi Reichart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19693.txt.

Introduction

Itay Nakash and co-authors speed up large language models for narrow jobs by retraining their vocabularies, in a 2025 paper. A general model spends computing power on words it will never need in, say, a medical or legal niche. Each output step costs a full pass through the model, so shorter token streams mean faster answers. Their method, AdaptiVocab, swaps general tokens for chunks common in the target domain.

The new chunks start life as blends of the embeddings of the pieces they replace, weighted so common pieces count more. A short, light fine-tuning phase then settles them in. The approach works with any tokenizer and model shape, the authors say. Cutting token counts on both the way in and the way out promises cheaper, quicker models for specialist tools where broad knowledge is wasted.

Catalog record

CREATORItay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, Roi Reichart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19693

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, Roi Reichart Abstract Large Language Models (LLMs) have shown impressive versatility as general-purpose models. However, their broad applicability comes at a high-cost computational overhead, particularly in auto-regressive decoding where each step requires a forward pass. In domain-specific settings, general-purpose capabilities are unnecessary and can be exchanged for efficiency. In this work, we take a novel perspective on domain adaptation–reducing latency and computational costs by adapting the vocabulary to focused domains of interest. We introduce AdaptiVocab, a complete approach for vocabulary adaptation, designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be applied to any tokenizer and architecture, modifying the vocabulary by replacing tokens with domain-specific n n -gram-based tokens, thereby reducing the number of tokens required for both input processing and output generation. AdaptiVocab initializes new n n -token embeddings using an exponentially weighted combination of existing embeddings and employs a lightweight fine-tuning phase that can be efficiently performed on a single GPU. We evaluate two 7B LLMs across three niche domains, assessing efficiency, generation quality, and end-task performance. Our results show that AdaptiVocab reduces token usage by over 25% without compromising performance. 1 1 1 Our code and data are available at: https://github.com/itay-nakash/AdaptiVocab 1 Introduction Large Language Models (LLMs) have revolutionized the field of NLP and are now integrated into everyday technologies ( Inaba et al., 2023 ; Chen et al., 2024 ; Huang et al., 2024 ) . However, their large size and high computational demands lead to significant latency and cost, often hindering their deployment in many real-world scenarios. Practitioners frequently aim to apply LLMs in domain-specific settings ( Saad-Falcon et al., 2023 ; Eschbach-Dymanus et al., 2024 ; Afzal et al., 2024 ) , where general-purpose capabilities are unnecessary, and efficiency and hardware constraints become critical concerns ( Calderon et al., 2023 ) . The field of domain adaptation addresses the challenge of adapting NLP models to new domains ( Blitzer et al., 2007 ; Ben-David et al., 2022 ; Lu et al., 2024 ) , typically in low-resource settings where only a limited amount of unlabeled domain data is available (e.g., a few million tokens) ( Ramponi & Plank, 2020 ; Calderon et al., 2022 ; Marashian et al., 2025 ) . In this work, we take a novel perspective on domain adaptation. Rather than focusing solely on improving end-task performance ( Chu et al., 2017 ; Calderon et al., 2024 ) , we focus on efficiency by reducing latency and computational costs when applying LLMs to new domains. Unlike traditional domain adaptation methods, which are either model-centric (modifying the architecture or training objective) or data-centric (modifying, augmenting, or refining the data) ( Yavuz et al., 2020 ; Shakeri et al., 2020 ) , we introduce a vocabulary-centric approach. Our method enhances the efficiency of LLMs without compromising performance by adapting their vocabulary to better fit a target low-resource domain. Mistral-v0.3-7b BPE tokenizer: 60 tokens Source: Pérez & Valls (2015) Boh r takes advantage of this peculiar fact to dev ise transform ations that connect in a continuous way different station ary states of the same system . In other words , the quant ization of ad i ab atic in vari ants meets the pre conditions for the applic ability of B olt z mann ’ s principle . AdaptiVocab (on top of Mistrals tokenizer): 39 tokens Bohr takes advantage of this peculiar fact to dev ise transformations that connect in a continuous way different stationary states of the same system. In other words , the quantization of adiab atic invariants meets the pre conditions for the applicability of B oltzmann ’ s principle. Table 1: Tokenization before and after vocabulary adaptation to the History of Physics domain. LLMs process text by dividing it into predefined tokens composed of subwords and words, a process known as tokenization. The vocabulary, which consists of this set of tokens, is typically selected using widely adopted methods such as Byte Pair Encoding (BPE) ( Sennrich et al., 2016 ; Schuster & Nakajima, 2012 ; Kudo, 2018 ) . Each token is represented by a high-dimensional continuous vector, which is then fed into the transformer layers of the LLM. Tokenization and vocabulary selection directly impact computational efficiency since representing input text with fewer tokens reduces memory consumption and speeds up processing. The vocabulary has an additional impact on generative models, particularly on auto-regressive architectures. Each generated token requires a forward pass, and optimizing the vocabulary can directly reduce decoding steps and improve latency. Since LLM vocabularies are optimized for general-purpose use, domain-specific applications can benefit from replacing redundant tokens with domain-relevant ones, thus enhancing efficiency. In this paper, we propose AdaptiVocab, the first complete approach for adapting an LLM’s vocabulary to a new focused low-resource domain. Figure 1 illustrates the main components of AdaptiVocab. In § 2 , we thoroughly discuss how previous vocabulary adaptation research differs from ours. To summarise: (1) We focus on generative decoder-only models and optimize for efficiency; (2) We address monolingual adaptation to low-resource focused domains; (3) We incorporate multi-word tokens into the vocabulary; (4) Our approach is tokenizer-agnostic and works on top of any tokenizer. AdaptiVocab works as follows: given target domain-specific data, we first compute a frequency-based score for each candidate n -token: a combination of n existing tokens that can represent a subword, a single word, or an n-gram. We then modify the vocabulary by replacing low-scoring tokens with high-scoring n -tokens. Additionally, we propose an n -token tokenization algorithm for encoding and decoding text with the modified vocabulary. Second, we initialize the embeddings of the new n -tokens using an exponentially weighted combination of the embeddings of their original constituent tokens, ensuring a smoother integration compared to plain averaging or random initialization. Third, we explore strategies for a lightweight adaptation phase, suggesting fine-tuning only the model’s embedding layers and the layers that directly interact with them. AdaptiVocab can be applied to any tokenizer and LLM, requiring minimal computational overhead. The entire adaptation process runs within a few hours on a single RTX A6000 GPU with 48GB of memory. To evaluate AdaptiVocab, we conducted experiments with two open-source LLMs, Mistral-7B-0.3 ( Jiang et al., 2023 ) and Llama-2 7B ( Touvron et al., 2023 ) , across three niche, low-resource domains: Earth Sciences, History of Physics, and Games and Toys. We performed multiple types of evaluation to ensure that AdaptiVocab does not degrade performance, including automatic generation quality assessment using both an LLM-as-a-Judge and human evaluation, as well as end-task performance analysis. For the latter, we created three open-book multiple-choice datasets for our focused domains, totaling 900 questions. Our results show that AdaptiVocab reduces input and output token usage by more than 25% on average without compromising generation quality or end-task performance. Additionally, we found that lightweight fine-tuning improves domain-specific end-task performance even for off-the-shelf pre-trained LLMs. Our contributions are as follows: (1) We propose AdaptiVocab, the first complete approach for adapting an LLM’s vocabulary to a new low-resource domain. Our method encompasses a vocabulary modification algorithm, an embedding initialization technique, a lightweight fine-tuning strategy, and an n -token tokenization

Read the complete text