CMHG: A Dataset and Benchmark for Headline Generation of Minority Languages in China
Guixian Xu Zeli Su Ziyin Zhang Jianing Liu, XU Han Ting Zhang Yushuang Dong, [email protected] [email protected] Corresponding author · 2025
Read CMHG: A Dataset and Benchmark for Headline Generation of Minority Languages in China by Guixian Xu Zeli Su Ziyin Zhang Jianing Liu, XU Han Ting Zhang Yushuang Dong, [email protected] [email protected] Corresponding author (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09990.txt.
Introduction
Guixian Xu and 6 co-authors introduce a new data set in this 2025 paper, built for headline generation in three minority languages of China: Tibetan, Uyghur, and Traditional Mongolian. These languages use writing systems that differ from common standards. The lack of labeled text has held back supervised tasks. One such task is writing a headline for a news story.
Their collection, called CMHG, holds a hundred thousand entries for Tibetan and fifty thousand each for Uyghur and Mongolian, all prepared for headline generation. The team also built a test set checked by native speakers to serve as a shared benchmark. They offer the resource as groundwork for language technology that serves communities most tools pass over.
Catalog record
| CREATOR | Guixian Xu Zeli Su Ziyin Zhang Jianing Liu, XU Han Ting Zhang Yushuang Dong, [email protected] [email protected] Corresponding author |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-09990
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CMHG: A Dataset and Benchmark for Headline Generation of Minority Languages in China Guixian Xu Zeli Su Ziyin Zhang Jianing Liu, XU Han Ting Zhang Yushuang Dong, [email protected] [email protected] Corresponding author Abstract Minority languages in China, such as Tibetan, Uyghur, and Traditional Mongolian, face significant challenges due to their unique writing systems, which differ from international standards. This discrepancy has led to a severe lack of relevant corpora, particularly for supervised tasks like headline generation. To address this gap, we introduce a novel dataset, Chinese Minority Headline Generation (CMHG), which includes 100,000 entries for Tibetan, and 50,000 entries each for Uyghur and Mongolian, specifically curated for headline generation tasks. Additionally, we propose a high-quality test set annotated by native speakers, designed to serve as a benchmark for future research in this domain. We hope this dataset will become a valuable resource for advancing headline generation in Chinese minority languages and contribute to the development of related benchmarks. https://huggingface.co/KEVVVV/CMHG 1 Introduction Recently, the rapid development of large language models (LLMs) has been fueled by the availability of high-quality pre-training data. However, these advancements have primarily benefitted high-resource languages such as English and Chinese. In contrast, many languages with substantial user bases remain excluded due to the scarcity of suitable corpora, especially for specific tasks like headline generation . This exclusion poses challenges for both academic research and the practical application of AI technologies. Figure 1: The relationship between population size and dataset size in OSCAR (y-axis, in MB) for various high-, middle-, and low-resource languages. This paper focuses on underrepresented minority languages in China, including Tibetan, Uyghur, and Mongolian, which have rich linguistic and cultural significance but suffer from a lack of resources. Although these languages appear in multilingual datasets like OSCAR Jansen et al. (2022) and CulturaX Nguyen et al. (2024) , quality issues limit their usefulness. As shown in Figure 1 , there is a clear gap between the large speaker populations of these languages and the small amount of data available in major corpora. Studies also reveal problems with data quality: Zhang et al. (2024) found that 34% of the Uyghur data in CulturaX contains Kazakh or Arabic texts, pointing to issues like language misidentification and noise. These challenges of data scarcity and quality undermine efforts to build effective natural language processing (NLP) systems for these communities. Moreover, there is a complete lack of open-source datasets tailored for headline generation in these minority languages. This gap hinders the development of supervised methods and benchmarks for headline generation tasks. To address this limitation, we introduce Chinese Minority Headline Generation (CMHG) , a novel dataset specifically designed for headline generation in Tibetan, Uyghur, and Mongolian. CMHG consists of 100,000 Tibetan samples and 50,000 samples each for Uyghur and Mongolian. In addition to the main dataset, we collaborated with native speakers of these languages to further ensure data quality. From the existing data, we selected 3,000 samples for each language and conducted a detailed annotation process to evaluate the alignment and quality of the headlines. These samples were reviewed by multiple annotators for each language, and only the data deemed high-quality by consensus among native annotators was retained. This subset of data provides a reliable benchmark for future research, ensuring consistency and reproducibility in evaluating headline generation models. By combining a large-scale dataset with carefully curated benchmark samples, CMHG bridges the gap in resources for headline generation in Chinese minority languages. In summary, this paper makes the following key contributions: • We present CMHG , a novel and large-scale open-source dataset specifically designed for headline generation in three Chinese minority languages: Tibetan, Uyghur, and Mongolian. We release this dataset. • We provide a carefully curated benchmark test set, annotated by native speakers, to ensure high-quality evaluation and support transparent, reproducible research in headline generation for Chinese minority languages. By introducing CMHG, we aim to fill the resource gap and pave the way for advancing natural language processing research on underrepresented languages. 2 Data Sources The Chinese Minority Headline Generation (CMHG) dataset, proposed in this paper, is sourced from various online platforms in China, including government documents and news articles (detailed list in Appendix A ). We used web crawlers to collect the data, where the webpage title serves as the headline and the main text as the source content. To ensure data quality and reliability, we applied a thorough cleaning process, with the main methods outlined as follows: • Removal of Non-Textual Content : We filtered out non-textual elements such as advertisements, pop-ups, navigation bars, and multimedia (e.g., images, videos, and audio files). This ensured that only relevant text was retained. • Duplicate Detection and Removal : We identified and removed duplicate entries to avoid redundancy in the dataset, which could potentially bias the headline generation models. • Text Normalization : We converted all characters to uniform encoding and removing any extraneous white spaces, special characters, or formatting inconsistencies. • Language Purity Check :We sourced content from monolingual websites and used regex-based filtering to ensure linguistic purity in manually annotated data. Controlled noise was allowed in training data to improve model robustness. Additionally, we evaluated several existing Language Identification tools that claimed to support our target languages but found high false-positive rates. These steps were essential to ensure that the final dataset consisted of high-quality, relevant, and clean text that could be reliably used for our tasks. 3 Annotation After crawling the data, we further enhanced the quality of the three languages’ evaluation set by native speaker annotation. For each language, two annotators assessed the quality of the title-content matching. A total of 3000 randomly selected samples from the crawled data were annotated, with the task focused on evaluating how well the article titles matched the content of the articles. 3.1 Annotation Guidelines The annotation process was designed to ensure the reliability and consistency of the evaluations. The annotators were provided with the following specific guidelines: Language Size Length In Token (Title/Content) Length In Characters (Title/Content) Cohen’s \kappa ICC Same Tendency Tibetan 2901 12.3 / 376.7 74.0 / 1884.1 0.71 0.80 1.00 Mongolian 2931 27.2 / 429.8 136.1 / 2149.0 0.28 0.42 0.85 Uyghur 2950 30.2 / 815.7 151.0 / 4078.5 0.44 0.67 1.00 Table 1: Annotation Results and Inter-Annotator Agreement Analysis for Valid Samples in the CMHG Dataset. Token length is counted by the CINO ( Yang et al., 2022 ) tokenizer; character length by raw character count. • Task Objective: Annotators were asked to assess the degree of relevance between the title and the content of the article and assign a score accordingly. • Title Evaluation: Annotators were first asked to identify any issues between the title and the article content, such as: – Incomplete Article: whether the article content is incomplete, making it impossible for the title and content to match. – Text Quality: whether the title contains spelling, grammatical, or contextual errors that would significantly hinder its match with the article content. – Other Issues: whether there is any other noticeable discrepancies between the title and content, such as irrelevance. If no major issues were identified, the title-content match was considered "Normal." • Matching Score: Annotators were instructed to rate the match based on how well the