Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children’s Stories for Training Small Language Models

Deepon Halder, Angira Mukherjee · 2026

Read Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children’s Stories for Training Small Language Models by Deepon Halder, Angira Mukherjee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14563.txt.

Introduction

Halder and Mukherjee build Multilingual TinyStories, a story corpus in 17 Indian languages. Low-resource languages lack clean, coherent text for training small models. Their set holds simple children's stories written in native scripts. A hybrid pipeline pairs the Sarvam-M model with shaped prompts for native drafting. Google Translate then widens the set across languages. Strict filters leave 132,942 stories and over 93.9 million tokens. The corpus supports training and testing of small models for Indic languages.

Catalog record

CREATORDeepon Halder, Angira Mukherjee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19570
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children’s Stories for Training Small Language Models Deepon Halder, Angira Mukherjee Abstract The development of robust language models for low-resource languages is frequently bottlenecked by the scarcity of high-quality, coherent, and domain-appropriate training corpora. In this paper, we introduce the Multilingual TinyStories dataset, a large-scale, synthetically generated collection of children’s stories encompassing 17 Indian languages. Designed specifically for the training and evaluation of Small Language Models (SLMs), the corpus provides simple, narrative-driven text strictly localized to native scripts. We detail our hybrid curation pipeline, which leverages the Sarvam-M language model and a novel combinatorial prompt engineering framework for native generation, coupled with the Google Translate API for large-scale cross-lingual expansion. Through strict programmatic filtering, we compiled 132,942 stories and over 93.9 million tokens in our release, serving as a foundational resource for multilingual language modeling and transfer learning in the Indic linguistic sphere. 1 Introduction Recent advancements in Natural Language Processing (NLP) have been largely driven by the exponential scaling of Large Language Models (LLMs). However, the zero-shot capabilities of these models remain disproportionately skewed toward high-resource languages, predominantly English. For the vast linguistic diversity of the Indian subcontinent, the lack of extensive, high-quality, and structurally simple corpora poses a significant barrier to democratizing artificial intelligence capabilities. Training models from scratch or aligning existing models to low-resource languages requires data that is not merely voluminous, but also clean and linguistically coherent. Inspired by the original TinyStories dataset ( 1 ) , which demonstrated that SLMs can learn syntax, semantics, and reasoning from constrained, simple vocabularies, we propose a multilingual extension tailored to Indic languages. The Multilingual TinyStories dataset provides a comprehensive, synthetic corpus of children’s stories. By focusing on simple narratives, calibrated to the reading comprehension of a five-year-old child, the dataset intentionally restricts vocabulary complexity. This allows small-parameter models to effectively capture core linguistic structures without being overwhelmed by domain-specific jargon or web-scraped noise. To avoid redundancy with existing efforts, we explicitly exclude languages such as Bengali, Marathi, and Hindi, focusing instead on underrepresented demographics. In summary, the primary contributions of our work are as follows: • A Novel Multilingual Corpus: We introduce the Multilingual TinyStories dataset, a large-scale, open-source synthetic corpus comprising over 132,000 children’s stories and approximately 93.9 million tokens across 17 underrepresented Indic languages. • Tailored Resource for SLMs: We provide a clean, structurally simple, and highly constrained dataset strictly localized to native scripts, addressing the critical lack of foundational pre-training and alignment resources for Small Language Models in the Indic linguistic sphere. Figure 1: Pipeline for generating the Multilingual TinyStories dataset. Slot based prompts are instantiated and used to generate native stories with the Sarvam-M model, followed by automated filtering and corpus construction across seven Indic languages. The dataset is then expanded via translation to additional languages, producing 132,942 stories in 17 Indic languages ( 93.9M tokens). 2 Related Work Our work intersects with several active areas of research in natural language processing: synthetic data generation, small language models, and low-resource multilingual corpora. Synthetic Data for SLMs: The efficacy of using highly constrained synthetic text to train capable language models was popularized by the original TinyStories dataset ( 1 ) . They demonstrated that a model with merely 10 million parameters could produce fluent, consistent narratives when trained on synthetically generated children’s stories. This paradigm was further validated by the "Textbooks Are All You Need" approach ( 2 ) , proving that data quality and structural clarity can effectively substitute for raw parameter scale. Indic Language Corpora: Historically, NLP research in Indian languages has relied on massive, web-crawled datasets such as IndicCorp ( 3 ) and parallel translation corpora like Samanantar ( 5 ) . While these resources are invaluable for large-scale pre-training, they are inherently noisy, formally complex, and unstructured, making them suboptimal for the initial syntactical alignment of small-parameter models. Multilingual Synthetic Corpora: Recent efforts have begun adapting the TinyStories methodology to non-English languages. Notably, the Regional TinyStories project by Vizuara ( 7 ) initiated this transition by focusing on high-resource Indian languages. Our Multilingual TinyStories corpus directly complements this literature by expanding the narrative methodology to 17 heavily underrepresented Indic languages via a hybrid generation and translation pipeline. 3 Methodology To systematically generate millions of high-quality tokens across varied linguistic distributions, we designed a scalable Python pipeline. The methodology employs a hybrid approach: native synthetic generation via the Sarvam-M language model ( 6 ) for a core set of languages, followed by machine translation to project the dataset into further low-resource domains. 3.1 Combinatorial Prompt Engineering For the foundational generated corpus, we avoided static prompts to prevent mode collapse. Instead, we developed a highly dynamic, slot-based prompt template system capable of generating millions of unique prompt permutations. Our framework utilizes over 15 distinct narrative archetypes (e.g., Adventure , Moral , Mystery , Friendship ). The full set of prompt templates used in the generation pipeline is provided in Appendix B . Each template contains variable slots dynamically populated during runtime by uniformly sampling from extensively curated conceptual pools: • Characters ( N\geq 50 ): Archetypal entities (e.g., boy, girl, rabbit, robot, princess ). • Settings ( N\geq 40 ): Spatial contexts (e.g., forest, village, space station, desert ). • Objects/Problems ( N\geq 50 ): Narrative catalysts (e.g., lost lantern, hidden treasure ). • Themes ( N\geq 30 ): Underlying didactic lessons (e.g., bravery, curiosity, honesty ). The complete vocabulary pools used to populate these slots are listed in Appendix C . An example instantiated prompt takes the following form: Write a short children’s story in { Gujarati } . The story should include a { curious rabbit } , a { forest } , and a problem involving a { lost lantern } . The story should teach something about { bravery } . Keep the language simple for a 5-year-old. Use exactly 5 to 8 sentences. This stochastic slot-filling strategy naturally forces the Sarvam-M model into disparate regions of its latent space, ensuring high semantic variance. A quantitative estimate of the theoretical prompt combinatorial space is presented in Appendix D . 3.2 Decoding Strategy and Generation Constraints Stochastic decoding was enforced with a temperature of 0.9 and a nucleus sampling threshold ( top_p ) of 0.95. Furthermore, we imposed a strict length constraint via max_new_tokens = 400 , driving the model to construct concise stories (averaging 80 to 200 tokens) with a clear climax within a 5 to 8 sentence limit. Table 1: Lexical diversity statistics across all language splits. Lang Dist-1 Dist-2 Dist-3 AvgSim as 0.0763 0.5057 0.8125 0.1486 doi 0.0462 0.3681 0.7099 0.2812 gom 0.0913 0.5518 0.8382 0.1702 gu 0.0878 0.5383 0.8500 0.1917 kn 0.1427 0.6660 0.9227 0.1112 mai 0.0464 0.3861 0.7311 0.2166 ml 0.1711 0.6954 0.9134 0.1449 mni 0.0737 0.4434 0.7729 0.2022 ne 0.0859 0.5405 0.8325 0.1746 or 0.0919 0.5155 0.7900 0.1475 pa 0.0469 0.3334 0.6788 0.2710 sa 0.1063 0.5420 0.8157 0.1909 sat 0.0285 0.2556 0.5728 0.3819 sd 0.0434 0.3569 0.6937 0.3053 ta 0.1360 0.6563 0.9132 0.1470 te 0.1153 0.5894 0.8787 0.1633 ur 0.0418 0.3340 0.6737 0.3487 3.3 Cross-Lingual Expansion via Machine Translation While the primary corpus of seven languages was natively generated using the Sarvam-M model, scaling to the

Read the complete text