No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem
Dasol Choi, Woomyoung Park, Youngsook Song, Yonsei University, MODULABS, SAIONIC AI, Lablup [email protected]@[email protected] · 2025
Read No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem by Dasol Choi, Woomyoung Park, Youngsook Song, Yonsei University, MODULABS, SAIONIC AI, Lablup [email protected]@[email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04329.txt.
Introduction
Dasol Choi, Woomyoung Park, and Youngsook Song map the language data sets available for Chinese, Japanese, and Korean. Their 2025 study starts from a gap: English enjoys rich data resources. The three East Asian languages serve more than a billion and a half people, yet their resources remain patchy and little compared.
Mining over thirty-three hundred sets on the Hugging Face platform, the team finds each community builds data its own way. Chinese sets lean large and institution-led. Korean sets grow from grassroots community effort. Japanese sets skew toward entertainment and subculture themes. The authors draw lessons on records, licenses, and sharing, aiming at stronger resources for language models in all three tongues.
Catalog record
| CREATOR | Dasol Choi, Woomyoung Park, Youngsook Song, Yonsei University, MODULABS, SAIONIC AI, Lablup [email protected]@[email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem Dasol Choi, Woomyoung Park, Youngsook Song, Yonsei University, MODULABS, SAIONIC AI, Lablup [email protected]@[email protected] Abstract Recent advances in Natural Language Processing (NLP) have underscored the crucial role of high-quality datasets in building large language models (LLMs). However, while extensive resources and analyses exist for English, the landscape for East Asian languages—particularly Chinese, Japanese, and Korean (CJK)—remains fragmented and underexplored, despite these languages together serving over 1.6 billion speakers. To address this gap, we investigate the HuggingFace ecosystem from a cross-linguistic perspective, focusing on how cultural norms, research environments, and institutional practices shape dataset availability and quality. Drawing on more than 3,300 datasets, we employ quantitative and qualitative methods to examine how these factors drive distinct creation and curation patterns across Chinese, Japanese, and Korean NLP communities. Our findings highlight the large-scale and often institution-driven nature of Chinese datasets, grassroots community-led development in Korean NLP, and an entertainment- and subculture-focused emphasis on Japanese collections. By uncovering these patterns, we reveal practical strategies for enhancing dataset documentation, licensing clarity, and cross-lingual resource sharing—ultimately guiding more effective and culturally attuned LLM development in East Asia. We conclude by discussing best practices for future dataset curation and collaboration, aiming to strengthen resource development across all three languages. 1 Introduction With the emergence of Large Language Models (LLMs) transforming the field of Natural Language Processing (NLP) Kenton and Toutanova 2019 ; Brown et al. 2020 ; Achiam et al. 2023 , the importance of high-quality datasets in model development has become increasingly critical. For datasets to be valuable in this context, they must meet both quantitative requirements (sufficient size and coverage) and qualitative standards (reliability and representativeness). While English-language resources have been extensively studied, the landscape of datasets for East Asian languages—particularly Chinese, Japanese, and Korean (CJK)—remains comparatively underexplored Joshi et al. 2020 ; Bender 2019 . This gap is especially noteworthy given that these languages collectively serve over 1.6 billion speakers and originate from major hubs of technological innovation. In recent years, platforms such as HuggingFace have emerged as central repositories for distributing and accessing NLP datasets, making these resources widely accessible while introducing new challenges in dataset discovery, quality assessment, and cross-lingual collaboration Hugging Face 2023 ; Lhoest et al. 2021 . These challenges are particularly pronounced for CJK languages due to their unique linguistic features, distinct cultural contexts, and varying approaches to data sharing and documentation. While CJK languages play an increasingly important role in global NLP research, several critical issues need to be addressed. First, there is limited understanding of how dataset creation patterns differ across these language communities—and how those differences reflect their respective NLP ecosystems. Second, although cultural and institutional factors evidently influence dataset generation, their specific impacts on dataset characteristics and quality have yet to be systematically investigated. Third, the potential for cross-lingual synergies among CJK languages remains largely untapped, despite their many shared cultural and linguistic foundations. For instance, while benchmarks like MMLU Hendrycks et al. 2020 are being generated in multiple languages, insufficient comparative analysis exists regarding how these variants differ from traditional parallel corpora or how download patterns across regions reflect distinct cultural preferences. To address these challenges, this study investigates the HuggingFace ecosystem for Chinese, Japanese, and Korean NLP resources, focusing on dataset development and usage patterns. The specific objectives of this work are to: • Examine the current landscape of CJK datasets on HuggingFace, including key metadata such as domain, dataset size, documentation practices, and usage statistics. • Analyze the cultural characteristics and development patterns underlying dataset creation in each language community, highlighting commonalities and differences in curation and documentation. • Identify potential synergies and cross-lingual opportunities among the three languages, and propose strategies for more effective, collaborative dataset development. 2 Related Work Documentation efforts for CJK language resources have expanded progressively over time. For Chinese, Tao et al. 2009 introduced systematic approaches for constructing and evaluating linguistic data resources. More recently, Li et al. 2023a compiled a comprehensive repository of Chinese datasets, emphasizing detailed accessibility and classification frameworks. In the Korean context, Cho et al. 2020 provided a thorough survey of existing NLP resources, stressing the need for enhanced documentation and broader data accessibility. Building on this, Cho et al. 2023 examined the historical interplay between dataset creation and NLP model development, underscoring how local research cultures influence resource development. However, these earlier studies largely predate the rise of large generative models (e.g., GPT) and relied on GitHub as the primary hub for open-source NLP efforts. Since then, new platforms such as HuggingFace have substantially reshaped data-sharing practices, yet systematic, cross-lingual analyses of CJK datasets within these newer ecosystems remain rare. In parallel, major multilingual projects— including BigScience ( Le Scao et al. 2023 ) , CC100 ( Wenzek et al. 2019 ) , and LAION ( Schuhmann et al. 2022 ) —have expanded the global availability of non-English data. While these initiatives contribute valuable resources, they do not investigate the deeper cultural or linguistic factors affecting dataset usage within specific communities such as CJK. Moreover, the broader movement toward improved dataset documentation (e.g., Datasheets for Datasets ; Gebru et al. 2021 ) suggests increasing awareness of ethical and practical considerations, yet how such guidelines are adopted varies widely across languages and platforms. Over the past few years, HuggingFace has emerged as a central repository for sharing models, datasets, and evaluation metrics in NLP. Studies focusing on HuggingFace have flagged challenges such as inconsistent documentation ( Yang et al. 2024 ) , limited transparency about training data ( Pepe et al. 2024 ) , and ambiguous licensing information. These issues underscore the need to examine dataset curation and documentation in specific cultural contexts, as local norms and priorities can shape how resources are developed, annotated, and shared ( Lhoest et al. 2021 ) . Additionally, recent work by Darg̀is et al. 2024 offers valuable insights into building systematic evaluation frameworks for languages with particular linguistic and cultural traits—an approach that could inform the future assessment of CJK datasets. Nonetheless, no existing study has comprehensively analyzed how the unique ecosystem of Chinese, Japanese, and Korean language datasets on HuggingFace is shaped by local research cultures, licensing preferences, and community-driven development. Building on these perspectives, our research provides the first large-scale, comparative analysis of CJK datasets within the HuggingFace ecosystem. We investigate how cultural contexts, platform-specific practices, and documentation standards influence dataset usage patterns and resource quality. Additionally, we highlight potential synergies for cross-lingual collaboration and offer insights that may guide more effective and ethically grounded dataset curation for East Asian NLP. Table 1: Analysis framework for CJK datasets, organized by category. Category Evaluation Metrics Scale & Composition Dataset Size: Distribution across size categories (small, medium, large, extra-large) Language Makeup: Distribution of monolingual, English-paired, and multilingual datasets Task Types: Distribution of major NLP tasks (text generation, QA, classification, etc.) Development Patterns Ownership Structure: Proportions of corporate, institutional, and individual contributions License Types: Distribution of permissive, copyleft, unknown, and other licenses Community Activity: Dataset creation trends and instruction tuning development Documentation Quality Academic Validation: Presence of associated arXiv papers and research citations Documentation Standards: Adherence to HuggingFace dataset card