Turning Language Model Training from Black Box into a Sandbox
Nicolas Pope, Matti Tedre · 2026
Read Turning Language Model Training from Black Box into a Sandbox by Nicolas Pope, Matti Tedre (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21631.txt.
Introduction
Nicolas Pope and Matti Tedre turn language model training from a black box into a classroom sandbox. Most lessons on generative AI only let students prompt a finished model. The data and mechanics that shape the model stay hidden, which feeds both hype and fear.
The authors built a browser tool where students train a small transformer on their own device. Every step of training is visible and touchable. Students can swap the training text and watch behavior change. Seeing a model learn from data they chose gives a grounded sense of what these systems are and are not.
Catalog record
| CREATOR | Nicolas Pope, Matti Tedre |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Turning Language Model Training from Black Box into a Sandbox PubID: pubid: 000-0-0000-0000-0/00/$00.00 ©2026 IEEE Thanks: This study was funded by the Strategic Research Council (SRC) established within the Research Council of Finland, grant #352859. Nicolas Pope, Matti Tedre Abstract Most classroom engagements with generative AI focus on prompting pre-trained models, leaving the role of training data and model mechanics opaque. We developed a browser-based tool that allows students to train a small transformer language model entirely on their own device, making the training process visible. In a CS1 course, 162 students completed pre- and post-test explanations of why language models sometimes produce incorrect or strange output. After a brief hands-on training activity, students’ explanations shifted significantly from anthropomorphic and misconceived accounts toward data- and model-based reasoning. The results suggest that enabling learners to directly observe training can support conceptual understanding of the data-driven nature of language models and model training, even within a short intervention. For K-12 AI literacy and AI education research, the study findings suggest that enabling students to train—and not only prompt—language models can shift how they think about AI. I Introduction Large language models (LLMs) have become everyday tools in professional, creative, and educational contexts. Yet for most learners, these systems remain opaque black boxes: tools to use rather than mechanisms to understand [ 4 , 3 ] . Classroom activities often focus on prompting, evaluating, or auditing the behavior of industry scale models, leaving the processes that make them work hidden. Students may see polished generative outputs, but not the messy, iterative process of data curation and model training, and why models sometimes produce responses that are implausible, incoherent, or “hallucinated.” Critical AI literacy research argues that learners need opportunities to experience how AI systems are constructed and shaped by human choices, rather than encountering them only as finished products [ 6 , 11 , 9 , 12 , 5 ] . Making the mechanisms of LLM training visible supports conceptual understanding of their statistical, data-driven nature. Making the training process visible, however, has proven difficult in the case of language models. Training LLMs requires specialized hardware, large-scale datasets, and cloud-based inference pipelines that are not available in schools and introductory university courses. Although browser-based educational tools such as Teachable Machine have successfully supported learning about image classification workflows [ 1 , 8 ] , similar pedagogically oriented tools for generative language models are scarce, especially tools that reveal the complete workflow from data selection to training and inference, are aimed at education, and use real transformer technology. The Little Language Machine addresses this gap. It is a browser-native learning environment in which students can train a miniature transformer-based model directly on their own devices. Learners upload or compose small text corpora, select model size, and observe training unfold in real time. Within minutes, the model progresses from random character noise to increasingly coherent text sequences, making visible how training data and duration shape the model’s behavior. Because the tool runs entirely client-side using WebGPU acceleration, it ensures GDPR-compliance, no setup overhead, and works even on low-end classroom laptops. Fig. 1: Screen shot of Little Language Machine. When no XAI options are selected, the basic interface consists of five main windows: \raisebox{-.4pt} {\scriptsize{1}}⃝ Training data selection, \raisebox{-.4pt} {\scriptsize{4}}⃝ Model selection, \raisebox{-.4pt} {\scriptsize{5}}⃝ Training process tracker, \raisebox{-.4pt} {\scriptsize{6}}⃝ Model quality evaluation, and \raisebox{-.4pt} {\scriptsize{7}}⃝ Text generator window. This study deployed the Little Language Machine at scale in an Introduction to Computing (CS1) course. A total of 198 learners completed a pre-test and training activity, 222 did a post-test that measured changes in how they explained why language models sometimes produce incorrect or strange output. Altogether 162 students completed both activities. We analyzed students’ evolving explanatory models to understand how this short, hands-on training experience shifted learners’ reasoning about the mechanisms of language models. II System Description The Little Language Machine ( lm.gen-ai.fi ) runs a modified nanoGPT-style transformer [ 7 ] entirely within the browser. The system supports different model sizes; for instance, one configuration consists of three self-attention heads, eight transformer blocks, 192 hidden units, and a 128-token context window, totaling 4 million parameters. A tiny two-million-parameter, character-level model that occupies just 0.24GB memory is also available, designed to run within the memory constraints of typical student laptops and Chromebooks. The implementation uses root mean square layer normalization and rotary position embeddings (RoPE) for efficient positional encoding. Training and inference are hardware-accelerated through WebGPU or WebGL, enabling fast training and inference on low-end devices. Key–value caching is used to maintain responsiveness during inference and mixed-precision training is used to reduce memory demands. As no data are transmitted to external servers, the system is privacy-preserving, offline-capable, and GDPR-compliant by design. Figure 1 illustrates the interaction workflow. Learners begin by selecting or uploading a data set \raisebox{-.4pt} {\scriptsize{1}}⃝ . Ready-made corpora include, for example, children’s short stories and Shakespeare’s works \raisebox{-.4pt} {\scriptsize{2}}⃝ . A visual indicator \raisebox{-.4pt} {\scriptsize{3}}⃝ displays whether the data set is sufficiently large for the chosen model. The learner then selects a transformer configuration \raisebox{-.4pt} {\scriptsize{4}}⃝ —either untrained, to watch learning unfold from noise to structure, or pre-trained for quick fine-tuning. Training \raisebox{-.4pt} {\scriptsize{5}}⃝ occurs directly in the browser, with loss and sample counts displayed in real time. Coherent text typically emerges within minutes. The system provides an automatic evaluation \raisebox{-.4pt} {\scriptsize{6}}⃝ that encourages reflection on how data size, domain, and training time affect model behavior. Learners can prompt the model at any stage in the generator window \raisebox{-.4pt} {\scriptsize{7}}⃝ and generate new text \raisebox{-.4pt} {\scriptsize{8}}⃝ . Models can be exported or reloaded for further exploration or classroom sharing. This end-to-end, in-browser workflow from data curation through training, evaluation, and generation, makes the inner workflow of language model training visible and manipulable, turning generative AI from a black box into a hands-on medium for inquiry and critique. III Methods The study took place in a mandatory first-year CS1 (Introduction to Computing) course at a Finnish university. The intervention consisted of a one-week sequence within the course. Students completed a pre-test consisting of two short open-ended questions asking how and why language models sometimes produce text that is incorrect, implausible, or nonsensical. Students were then assigned a structured homework task using the Little Language Machine . The task instructed learners to choose datasets, select model configurations, train and fine-tune models in the browser, observe how the output changed over time, and experiment with generating new text. Students submitted short written reflections describing what they observed about how and why the model’s behavior changed during training. Finally, students completed a post-test containing the same open-ended questions as the pre-test. Importantly, the course curriculum had not yet introduced machine learning or language models at this point, meaning students’ explanations were not shaped by prior formal instruction—only their assigned activities with the tool. Participation in the learning activity was part of normal course work, but informed consent for inclusion in the research was requested separately, following national research ethics guidelines [ 10 ] . A total of 162 students completed the pre-test, homework activity, and post-test, as well as provided consent. The primary data consisted of students’ open-ended responses from the pre/post-tests. Responses were analyzed using qualitative