TFD: A Comprehensive Structured Tibetan Foundation Dataset for Low-Resource Language Processing and Large-Scale Modeling

Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Yadi Liu, Wenbin Wei, Xiangxiang Wang, Yongbin Yu, * Equal Contribution, Corresponding Authors · 2025

Read TFD: A Comprehensive Structured Tibetan Foundation Dataset for Low-Resource Language Processing and Large-Scale Modeling by Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Yadi Liu, Wenbin Wei, Xiangxiang Wang, Yongbin Yu, * Equal Contribution, Corresponding Authors (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18288.txt.

Introduction

Cheng Huang and co-authors build a large dataset for the Tibetan language in a 2025 paper. Language models do well in major languages but lag in Tibetan, where training data are scarce. Prior efforts eased the shortage of raw text, yet no resource covered the whole path of building a model, from first training to safety tuning.

Their dataset holds more than eleven billion tokens, with curated parts for teaching, safety, preference ranking, and step-by-step reasoning. The reasoning part is the first large one of its kind for Tibetan. Using the data, the team trained a family of Tibetan models that beat strong baselines on tests of understanding, safety, and reasoning. They release the data to support open work on low-resource languages.

Catalog record

CREATORCheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Yadi Liu, Wenbin Wei, Xiangxiang Wang, Yongbin Yu, * Equal Contribution, Corresponding Authors
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TFD: A Comprehensive Structured Tibetan Foundation Dataset for Low-Resource Language Processing and Large-Scale Modeling Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Yadi Liu, Wenbin Wei, Xiangxiang Wang, Yongbin Yu, * Equal Contribution, Corresponding Authors Abstract Large Language Models (LLMs) have achieved remarkable success in high-resource languages, yet progress in Tibetan remains severely constrained. While recent efforts have begun to address pre-training data scarcity for Tibetan, a more fundamental gap persists: no existing resource supports the complete LLM development pipeline, spanning pre-training, instruction tuning, safety alignment, preference optimization, and reasoning supervision. We introduce the T ibetan F oundation D ataset ( TFD ), the first structured, large-scale, and expert-curated dataset covering all key stages of Tibetan large language modeling. TFD comprises TIBSTC , a unified corpus of over 11 billion tokens with curated sub-datasets for instruction tuning, safety alignment, and preference optimization, and TIBSTC-CoT , the first large-scale Tibetan chain-of-thought dataset. We demonstrate its utility by training the Sun-Shine family of Tibetan LLMs, achieving substantial improvements over strong baselines on understanding, safety, reasoning, and generation benchmarks. These results underscore that advancing low-resource language modeling requires not only scale, but a structurally complete data ecosystem. We release TFD to facilitate reproducible research and the development of robust, culturally aligned Tibetan LLMs. Code and data are available at https://github.com/Vicentvankor/sun-shine . 1 Introduction LLMs Hurst et al. (2024) ; Achiam et al. (2023) ; Dubey et al. (2024) ; Guo et al. (2025) ; Liu et al. (2024) have demonstrated impressive performance across a broad range of natural language processing (NLP) tasks, exhibiting strong generalization and emergent capabilities. However, LLMs in low-resource and linguistically complex languages Zhang et al. (2025) ; Maji et al. (2025) ; Kim et al. (2025) , such as Tibetan Huang et al. (2025) , remain significantly underserved. A major factor underlying this limitation is not only the scarcity of raw Tibetan text, but the absence of a structurally complete data ecosystem covering all stages of modern LLM development. For Tibetan itself, existing data resources are extremely limited in scale, uneven in quality, and lack standardized processing pipelines Huang et al. (2025) ; Wagner and Zeisler (2004) . Most prior Tibetan datasets Cuo et al. (2017) ; Hao et al. (2024) ; Ma et al. (2021) ; Zhao et al. (2020) ; Gongqu et al. (2024) cover only narrow domains, contain substantial noise or inconsistencies, and fail to provide the structured and large-scale supervision required by modern language modeling Liang et al. (2024) ; Lv et al. (2025) . While recent efforts have begun to address pre-training data scarcity for Tibetan Pan et al. (2025) , a critical gap remains: no existing resource jointly supports instruction tuning, safety alignment, preference optimization, and reasoning supervision, components that are now standard and indispensable in modern LLM pipelines. As a result, current Tibetan LLMs exhibit limited generalization, weak discourse modeling, and unstable generation across domains Liu and Best (2025) . To address this gap, we introduce the T ibetan F oundation D ataset ( TFD ), the first structured, large-scale, and expert-curated Tibetan dataset designed to cover the complete LLM development pipeline. TFD consists of two sub-datasets, TIBSTC and TIBSTC-CoT . TIBSTC covers over 11 billion tokens from domains such as literature, law, religion, medicine, and daily communication, and uniquely includes curated sub-datasets for instruction tuning ( Alpaca-Ti ), safety alignment ( Safety-Prompts-Ti ), and preference optimization ( CValues-Ti , hh-rlhf-Ti ), components absent from prior Tibetan corpora. TIBSTC-CoT is the first large-scale Tibetan chain-of-thought dataset, constructed to provide explicit intermediate reasoning supervision across multiple domains and task types. Together, these two components form a unified foundation that addresses both general language modeling and reasoning-oriented learning, filling a critical structural gap in the Tibetan NLP ecosystem. To validate TFD ’s utility, we train and release the Sun-Shine family of reference Tibetan LLMs based on Llama Dubey et al. (2024) and Qwen Yang and others (2025) backbones. While the primary contribution of this work lies in the construction of TFD, the Sun-Shine family serves as a proof of concept for building culturally aligned and linguistically robust Tibetan LLMs. Our main contributions are summarized as follows: • We introduce TFD , the first Tibetan dataset covering the complete LLM development pipeline, comprising TIBSTC , a multi-domain corpus with over 11 billion tokens together with curated datasets for instruction tuning, safety alignment, and preference optimization, and TIBSTC-CoT , the first large-scale Tibetan chain-of-thought dataset for explicit multi-step reasoning. • We design a comprehensive data processing and model training pipeline that supports large-scale collection, cleaning, deduplication, formatting, and structured organization tailored to Tibetan linguistic characteristics. • We release the Sun-Shine family, a set of reference Tibetan LLMs trained on TFD , to demonstrate the dataset’s effectiveness in instruction following, safety-aware generation, and multi-step reasoning. • We conduct error analysis, case studies, and human evaluations to provide qualitative insights into the strengths and limitations of the proposed models. 2 Related Work 2.1 Tibetan Language Modeling Tibetan remains a severely low-resource language in NLP, characterized by complex stacked syllables, rich morphology, and limited digitized corpora Wagner and Zeisler (2004) . Existing resources mainly target narrow tasks, and are typically small-scale, domain-specific, and lack standardization Liu and Best (2025) ; Huang et al. (2025) ; Liu et al. (2025b) ; Liu et al. (2025a) . Recent work has begun to address pre-training data scarcity Pan et al. (2025) , demonstrating that scale alone can yield meaningful improvements. However, pre-training corpora address only one stage of the LLM pipeline. No existing Tibetan dataset jointly supports instruction tuning, safety alignment, preference optimization, and reasoning supervision—components now standard in modern LLM development Zhuang and Sun (2025) ; Xu et al. (2025) ; Feng et al. (2025) ; Zhang et al. (2025) . TFD is designed to fill this structural gap as the first unified resource covering the complete LLM pipeline for Tibetan. 2.2 Chain of Thought Chain-of-thought (CoT) prompting has proven effective for improving multi-step reasoning in LLMs Wei et al. (2022) ; Wang et al. (2023) , but remains almost entirely concentrated on high-resource languages Li et al. (2024) ; Wei et al. (2022) ; Cobbe et al. (2021) . For Tibetan, there is a complete absence of native, large-scale reasoning datasets that preserve linguistic structure and cultural grounding. TIBSTC-CoT directly addresses this gap as the first large-scale native Tibetan chain-of-thought dataset. 3 TFD In order to address the limitations discussed above, we introduce the TFD , the foundational dataset developed for Tibetan LLMs, which consists of two sub-datasets: TIBSTC for general supervised fine-tuning and TIBSTC-CoT for chain-of-thought reasoning tasks. Details of the data collection process are provided in Appendix A . Dataset Instance Tokens Size Corpus 122,756,184 11B 18 GB Alpaca-Ti 42,676 76M 109 MB Safety-Prompts-Ti 38,156 62M 116 MB CValues-Ti 2,000 6M 9 MB hh-rlhf-Ti 47,411 98M 154 MB Table 1: Statistics of the TIBSTC sub-datasets 3.1 TIBSTC TIBSTC is a multi-component Tibetan data collection for LLM development. It consists of five categories: Corpus , Alpaca-Ti , Safety-Prompts-Ti , CValues-Ti , and hh-rlhf-Ti . As shown in Table 1 , all sub-datasets in TIBSTC vary substantially in instance count, token volume, and storage size. 3.1.1 Corpus As illustrated in

Read the complete text