Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study
Lifeng Chen, Ryan Lai, Tianming Liu · 2025
Read Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study by Lifeng Chen, Ryan Lai, Tianming Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03976.txt.
Introduction
Large language models speak major languages fluently and low-resource ones poorly, because training data for the latter barely exists. Lifeng Chen, Ryan Lai, and Tianming Liu tackle that gap for Tibetan in a 2025 language technology preprint. Tibetan is morphologically rich and underrepresented online, so a general model starts with weak grounding in how the language works.
Their two-stage recipe first continues pretraining a three-billion-parameter Qwen model on Tibetan text, then fine-tunes it under supervision for tasks and translation. Perplexity on Tibetan fell sharply, and Chinese-to-Tibetan translation quality rose several-fold on standard scores. A layer-by-layer look inside a related model showed adaptation concentrated in embeddings, output heads, and later projection layers. The authors read this as the first stage building a Tibetan semantic space while the second sharpens task behaviour gently. They release the framework openly for other low-resource languages.
Catalog record
| CREATOR | Lifeng Chen, Ryan Lai, Tianming Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-03976
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adapting Large Language Models to Low-Resource Tibetan: A Two-Stage Continual and Supervised Fine-Tuning Study Lifeng Chen, Ryan Lai, Tianming Liu Abstract Adapting large language models (LLMs) to low-resource languages remains a major challenge due to data scarcity and cross-lingual drift. This work presents a two-stage adaptation of Qwen2.5-3B to Tibetan, a morphologically rich and underrepresented language. We employ Continual Pretraining (CPT) to establish Tibetan linguistic grounding, followed by Supervised Fine-Tuning (SFT) for task and translation specialization. Empirical evaluations demonstrate a consistent decrease in perplexity (from 2.98 → 1.54) and substantial improvements in Chinese→Tibetan translation quality (BLEU: 0.046 → 0.261; chrF: 2.2 → 6.6). Layer-wise analysis across 435 layers in Qwen3-4B reveals that adaptation primarily concentrates on embedding and output heads, with mid–late MLP projections encoding domain-specific transformations. Our findings suggest that CPT constructs a Tibetan semantic manifold while SFT sharpens task alignment with minimal representational disruption. This study provides the first quantitative exploration of Tibetan adaptation dynamics for LLMs, and offers an open, reproducible framework for extending multilingual foundation models to low-resource settings. The code of this project can be found on: https://github.com/clf28/Tibetan-Finetuning/tree/main 1 Introduction Figure 1: Performance metrics overview showing consistent improvements from Base through CPT to SFT across language modeling and translation tasks. Large language models (LLMs) such as GPT-4 1 , LLaMA 16 , and Qwen 2 have achieved remarkable success across a wide range of natural-language understanding and generation tasks. Nevertheless, their linguistic competence remains heavily biased toward high-resource languages such as English and Chinese, while low-resource languages continue to be marginalized 11 . Among them, Tibetan is particularly under-served: both large-scale corpora and standardized NLP benchmarks are scarce, limiting the development of inclusive and equitable language technologies for Tibetan-speaking communities. Recent advances in multilingual pretraining have attempted to narrow this gap 4 ; 5 . Models such as mBERT 6 , XLM-R 4 , mBART 13 , and NLLB-200 5 demonstrate the promise of cross-lingual transfer, yet they still struggle with extremely low-resource languages characterized by limited token coverage, orthographic variation, and complex morphology. In the case of Tibetan, prior work has focused primarily on small-scale translation datasets 8 ; 12 or monolingual BERT-style models 21 ; 17 , without systematically examining how modern LLMs can be adapted to this linguistic domain. Moreover, the internal mechanisms of such adaptation—how model parameters evolve across layers when exposed to scarce and structurally unique data—remain largely unexplored. To address these challenges, we investigate the adaptation of Qwen2.5-3B 10 to Tibetan through a two-stage training pipeline. First, a Continual Pretraining (CPT) phase establishes lexical and structural grounding by exposing the model to large-scale Tibetan monolingual text. Second, a Supervised Fine-Tuning (SFT) phase introduces task-specific and cross-lingual instruction data in Tibetan, Chinese, and English, thereby reinforcing translation and instruction-following capabilities. This separation of linguistic adaptation and task alignment allows efficient specialization under low-resource conditions while mitigating catastrophic forgetting. Our empirical evaluation reveals substantial performance gains under low-resource settings. Perplexity consistently decreases from the base model through CPT and SFT, while translation quality improves by over 5× in BLEU and 3× in chrF on Chinese→Tibetan translation. Cross-lingual generalization is also observed in English→Tibetan transfer. Furthermore, a layer-wise weight analysis across 435 layers uncovers that adaptation concentrates primarily in the embedding matrices and mid-to-late MLP gate projections, suggesting a semantic re-anchoring of token and logit spaces rather than a full network re-parameterization. The correlation of CPT and SFT weight changes ( r\approx 1.0 ) indicates that fine-tuning consolidates rather than overwrites the linguistic manifold established during continual pretraining. In summary, this study contributes: • A reproducible two-stage framework for adapting large language models to Tibetan, effectively bridging low-resource and multilingual scenarios; • Comprehensive empirical evidence of improvements in both language modeling and translation; • A layer-wise analysis that sheds new light on how large decoder-only architectures internalize low-resource linguistic structure. 2 Related Works 2.1 Multilingual Language Models Multilingual language models have made significant strides in supporting cross-lingual transfer for diverse languages. Early efforts such as mBERT 6 demonstrated that multilingual pretraining enables zero-shot transfer across languages. XLM-R 4 scaled this approach to 100 languages using 2.5TB of CommonCrawl data, achieving substantial improvements on cross-lingual benchmarks, particularly for low-resource languages like Swahili and Urdu. For sequence-to-sequence tasks, mBART 13 introduced multilingual denoising pretraining, while NLLB-200 5 extended translation support to 200 languages through large-scale mining and filtering of parallel corpora. Despite these advances, extremely low-resource languages such as Tibetan continue to pose significant challenges due to limited token coverage and capacity dilution 4 . Recent foundation models in the Qwen series 2 ; 10 have incorporated broader multilingual support, including Tibetan, through pretraining on diverse web-scale corpora. Qwen2.5-3B, the base model in this study, benefits from native Tibetan tokenization, which provides a more favorable starting point for task-specific adaptation compared to models with limited or no Tibetan coverage. 2.2 Tibetan NLP Tibetan natural language processing remains in its early stages due to data scarcity, orthographic variation, and morphological complexity. Recent surveys 9 highlight the fragmented landscape of Tibetan resources, with most datasets limited to small-scale parallel corpora for machine translation 8 ; 12 . Prior translation efforts have focused primarily on Tibetan-Chinese language pairs using recurrent architectures such as GRUs 8 , though these approaches lack the generalization capacity of modern LLMs. Recent efforts have introduced Tibetan-specific language models. Zhou et al. 21 explored parameter-efficient fine-tuning (PEFT) techniques such as prompt-tuning and adapters for Tibetan pretrained models, demonstrating improvements in downstream tasks with minimal parameter updates. Wang et al. 17 developed T-LLaMA, a Tibetan adaptation of LLaMA2 through continual pretraining, marking the first large-scale decoder-only model for Tibetan. Evaluation benchmarks such as TLUE 18 have also emerged to standardize assessment of Tibetan language understanding. However, prior work has not systematically examined the two-stage pipeline of continual pretraining followed by supervised fine-tuning for Tibetan LLMs, nor has it investigated the internal parameter changes during adaptation. Our study addresses this gap by leveraging Qwen2.5-3B’s native Tibetan support, which substantially reduces the vocabulary adaptation overhead and facilitates more effective fine-tuning. 2.3 LLM Fine-Tuning Techniques Continual pretraining and supervised fine-tuning have become essential paradigms for adapting large language models to specific domains and tasks. Gururangan et al. 7 demonstrated that domain-adaptive pretraining (DAPT) and task-adaptive pretraining (TAPT) yield significant performance gains by exposing models to in-domain text before task-specific fine-tuning, particularly under low-resource conditions. This two-stage approach mitigates catastrophic forgetting by separating linguistic adaptation from task alignment. For instruction-following capabilities, supervised fine-tuning on instruction-formatted datasets has proven highly effective. Alpaca 15 and Vicuna 3 popularized instruction tuning by fine-tuning LLaMA models on self-instruct and conversational data, respectively, demonstrating substantial gains in zero-shot task performance. Recent frameworks such as LLaMA-Factory 20 have further streamlined the fine-tuning process by providing unified interfaces for efficient training with parameter-efficient methods and distributed optimization. In multilingual settings, prior work has shown that fine-tuning with a carefully balanced multilingual mixture preserves cross-lingual transfer while specializing to target languages 19 . Our approach follows this principle by including a modest Chinese component in the supervised fine-tuning stage, which serves as an anchor to maintain the model’s multilingual capabilities while primarily focusing on Tibetan task specialization. 3 Method Figure 2: Two-stage training pipeline and data