Optimizing Korean-Centric LLMs via Token Pruning

Hoyeol Kim, Hyeonwoo Kim · 2026

Read Optimizing Korean-Centric LLMs via Token Pruning by Hoyeol Kim, Hyeonwoo Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16235.txt.

Introduction

Kim and Kim test many-language large language models adapted for Korean-centred tasks by token pruning. Pruning drops tokens and embeddings for languages irrelevant to the target use. They test Aya, Qwen3, Gemma-3, and Llama-3 across original, English-Korean, and English-Korean-Chinese vocabularies. Pruning improves steady output by cutting language confusion. It often lifts machine translation on Korean-specific tasks, though following instructions varies by design.

Catalog record

CREATORHoyeol Kim, Hyeonwoo Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15282
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16235

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Korean-Centric LLMs via Token Pruning Hoyeol Kim, Hyeonwoo Kim Abstract This paper presents a systematic benchmark of state-of-the-art multilingual large language models (LLMs) adapted via token pruning—a compression technique that eliminates tokens and embedding parameters corresponding to languages irrelevant to the target application. Focusing on Korean-centric natural language processing (NLP) tasks, we evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations: Original, English-Korean (EnKo), and English-Korean-Chinese (EnKoZh). Performance is assessed using established benchmarks for general aptitude, cultural literacy, instruction following, and machine translation. Our findings indicate that token pruning significantly improves generation stability by eliminating language confusion, and in the case of machine translation, frequently enhances performance on Korean-specific tasks. While instruction-following capabilities display architecture-dependent variance linked to latent cross-lingual representations, the significant reduction in vocabulary size validates token pruning as a highly effective optimization strategy for memory-constrained, domain-specific deployments, despite modest gains in inference latency. 1 Introduction The proliferation of massively multilingual LLMs has significantly expanded the accessibility of advanced natural language processing. Models such as LLaMA-3 ( Dubey et al. 2024 ) , Qwen ( Yang et al. 2025 ) , and Gemma ( Team et al. 2025 ) rely on vast multilingual training corpora to achieve universal applicability. However, this breadth introduces a "curse of multilinguality," wherein a substantial fraction of the model’s parameters—particularly within the vocabulary and embedding layers—is allocated to languages that are superfluous for specific downstream applications. For deployments in strictly monolingual or bilingual contexts, such as Korean-centric services, this redundancy imposes unnecessary memory overhead and computational inefficiency. In the context of Korean NLP, this challenge is two-pronged. First, despite being a high-resource language, Korean often comprises a small minority of the training data in dominant English- or Chinese-centric models, which risks diluting cultural nuance. Second, while local deployment requires lightweight models, optimization strategies must not come at the cost of linguistic proficiency. Token pruning offers a targeted solution. By systematically excising tokens linked to non-target languages, this method compresses the vocabulary and embedding matrices, reducing the memory footprint without altering the core transformer architecture. While prior works have explored pruning generally, systematic research quantifying its effects on Korean language tasks remains limited. This paper bridges that gap by benchmarking token pruning on State-of-the-Art (SOTA) multilingual LLMs. We specifically evaluate how aggressive vocabulary compression impacts performance across three distinct pillars: general aptitude, cultural competence, and translation quality. 2 Related Work 2.1 Token Pruning Token pruning is increasingly recognized as a vital component of transformer efficiency. Li et al. 2025 provide a comprehensive review of token pruning methods, categorizing them into heuristic-based, learnable, and reinforcement learning-based approaches, and emphasize its role in reducing inference complexity while retaining accuracy. In multimodal contexts, Wen and Gao 2024 question the utility of complex attention-based importance scores, demonstrating that simple random baselines can approach the performance of specialized methods that lack strong theoretical grounding. Building on the theoretical view of token redundancy in transformers, Tai et al. 2025 formalized benchmarks to capture FLOP reductions and latency trade-offs under different pruning strategies. Complementing these input-level approaches, Lee and Kim 2024 introduced dynamic vocabulary pruning to restrict the output search space during testing, suggesting a synergistic effect between vocabulary- and token-level pruning for practical deployment. 2.2 Korean-Centric LLMs and Language Specialization Efforts to enhance Korean LLM performance bifurcate into adapting English-centric models and developing sovereign AI. Vo et al. 2024 showed that continual pretraining with tokenizer augmentation effectively adapts English foundation models, while Kim et al. 2025 introduced Thunder-LLM, achieving near-SOTA results via balanced bilingual corpora. Domain-specific applications have also expanded, exemplified by the bilingual GECKO model for code generation ( Oh and Kim 2024 ) and the FINKRX model for financial NLP ( Son et al. 2025a ) . 3 Methods 3.1 Target Models and Configurations We evaluate a diverse set of open-weight multilingual LLMs, specifically the Qwen3 series (ranging from 0.6B to 14B parameters), Gemma-3 (270M to 12B), and the Llama-3 family (3.1-8B and 3.2-3B). Additional models include Tri ( Han et al. 2025 ) , Ministral-8B ( Liu et al. 2026 ) , and Aya-23 ( Aryabumi et al. 2024 ) . To assess the impact of vocabulary compression, each model is processed into three distinct configurations: the Original setup, which retains the full multilingual vocabulary; the EnKo configuration, pruned to retain only English and Korean tokens; and the EnKoZh configuration, which preserves English, Korean, and Chinese tokens. The pruning process necessitates tokenizer reconstruction, positional encoding realignment, and the remapping of output layer weights to correspond to the reduced vocabulary indices. 3.2 Vocabulary Pruning Procedure We implement a “language-aware filtering” strategy that proceeds in three distinct stages. First, tokens are categorized by language using Unicode block ranges and script properties such as Hangul, Latin, and Hanzi (Chinese characters). Subsequently, tokens irrelevant to the target configuration—whether EnKo or EnKoZh—are discarded, and the retained tokens are mapped onto a new, continuous index space. Finally, the embedding matrix and output projection layer are physically rearranged to align with the new indices, effectively reducing the parameter count. This procedure leaves the internal transformer blocks and positional encodings intact, preserving the model’s sequence modeling capabilities. 3.3 Benchmarks and Metrics To quantify performance, we utilize four categories of Korean-centric benchmarks. General aptitude is assessed via KMMLU ( Son et al. 2025b ) , which tests general knowledge and logical reasoning. Cultural and linguistic understanding are evaluated using HAERAE ( Son et al. 2024 ) for cultural literacy and CLIcK ( Kim et al. 2024 ) for linguistic nuance. Instruction-following capabilities are measured through LogicKor ( Park 2024 ) and KoMTBench ( Research 2024 ) , which assess reasoning and constraint satisfaction. Finally, machine translation performance is evaluated using the WMT 24++ (Korean–English) benchmark ( Deutsch et al. 2025 ) , scored via XCOMET-XXL ( Guerreiro et al. 2024 ) . 3.4 Latency Measurement Token-level inference latency is quantified using the Seed-X-PPO-7B model ( Cheng et al. 2025 ) on the Google/WMT-pp dataset ( Deutsch et al. 2025 ) . We compare configurations under identical hardware conditions to isolate the speedups resulting from the reduced softmax computational load. 4 Experiments and Analysis Table 1: General Aptitude (KMMLU), Cultural (HAERAE), and Linguistic (CLIcK) Benchmark Results. Scores: accuracy. Higher values are more accurate. Model KMMLU HAERAE CLIcK Original EnKo EnKoZh Original EnKo EnKoZh Original EnKo EnKoZh Qwen3-0.6B 0.1482 0.1482 0.1483 0.2328 0.2328 0.2328 0.3188 0.3188 0.3183 Qwen3-1.7B 0.3589 0.3589 0.3591 0.4244 0.4235 0.4235 0.4010 0.3980 0.4005 Qwen3-4B 0.4581 0.4583 0.4585 0.5243 0.5234 0.5234 0.5574 0.5554 0.5569 Qwen3-8B 0.5254 0.5244 0.5249 0.5995 0.5995 0.5995 0.6005 0.5985 0.6005 Qwen3-14B 0.5340 0.5320 0.5325 0.6700 0.6700 0.6700 0.6221 0.6206 0.6221 Gemma-3-270m-it 0.3057 0.3058 0.3056 0.1989 0.1989 0.1989 0.3033 0.3033 0.3033 Gemma-3-1b-it 0.3094 0.3086 0.3090 0.3575 0.3584 0.3584 0.3388 0.3373 0.3378 Gemma-3-4b-it 0.3801 0.3788 0.3792 0.5170 0.5170 0.5160 0.4982 0.4982 0.4997 Gemma-3-12b-it 0.4812 0.4804 0.4806 0.6838 0.6838 0.6838 0.6311 0.6321 0.6321 Llama-3.1-8B-Inst 0.4165 0.4153 0.4156 0.5775 0.5775 0.5775 0.5213 0.5178 0.5208 Llama-3.2-3B-Inst 0.3307 0.3311 0.3311 0.3639 0.3621 0.3621 0.4190 0.4190 0.4180 Tri-7B 0.4944 0.4932 0.4934 0.7929 0.7919 0.7929 0.6486 0.6471 0.6486 Aya-23-8B 0.1788 0.1784 0.1787 0.5619 0.5619 0.5619 0.4802 0.4802 0.4792 Aya-expanse-8b 0.3436 0.3427 0.3428 0.6627 0.6627 0.6636 0.5153 0.5138 0.5153 4.1 General Aptitude and Cultural Alignment Table

Read the complete text