Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch
Evanfiya Logacheva, Arto Hellas, Tsvetomila Mihaylova, Juha Sorva, Ava Heinonen, Juho Leinonen · 2026
Read Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch by Evanfiya Logacheva, Arto Hellas, Tsvetomila Mihaylova, Juha Sorva, Ava Heinonen, Juho Leinonen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20476.txt.
Introduction
Evanfiya Logacheva and five colleagues test whether giving examples in context improves AI-generated diagrams for teaching. Generative AI is now widespread in computing education, yet the quality of its materials worries teachers and students. The authors build a method for diagram code generation that uses in-context examples shaped by rhetorical structure theory.
The method aims to align model output with user expectations. Their title flags the catch: one hallucinated example can spoil the whole set. The study speaks to any teacher using AI to draft figures, code, or exercises. One bad example, as the title warns, can spoil the whole bunch.
Catalog record
| CREATOR | Evanfiya Logacheva, Arto Hellas, Tsvetomila Mihaylova, Juha Sorva, Ava Heinonen, Juho Leinonen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch Evanfiya Logacheva, Arto Hellas, Tsvetomila Mihaylova, Juha Sorva, Ava Heinonen, Juho Leinonen Abstract Generative artificial intelligence (AI) has found a widespread use in computing education; at the same time, quality of generated materials raises concerns among educators and students. This study addresses this issue by introducing a novel method for diagram code generation with in-context examples based on the Rhetorical Structure Theory (RST), which aims to improve diagram generation by aligning models’ output with user expectations. Our approach is evaluated by computer science educators, who assessed 150 diagrams generated with large language models (LLMs) for logical organization, connectivity, layout aesthetic, and AI hallucination. The assessment dataset is additionally investigated for its utility in automated diagram evaluation. The preliminary results suggest that our method decreases the rate of factual hallucination and improves diagram faithfulness to provided context; however, due to LLMs’ stochasticity, the quality of the generated diagrams varies. Additionally, we present an in-depth analysis and discussion on the connection between AI hallucination and the quality of generated diagrams, which reveals that text contexts of higher complexity lead to higher rates of hallucination and LLMs often fail to detect mistakes in their output. 1 Introduction Recent works on diagram code generation 1 1 1 Generation of code that is rendered or displayed as a diagram, e.g., using the Graphviz library. use such approaches as entity and relation extraction Jiang et al. 2023 and information extraction using a user’s intent alongside in-context learning (ICL) Mondal et al. 2024 to structure the content of diagrams. While offering an exciting avenue for automation, the methods suffer from such issues as verbosity Jiang et al. 2023 , in addition to poor layout quality and hallucination Mondal et al. 2024 . Research shows that hallucination occurrence ranges between 3% and 86% for various tasks even among top performing responses for GPT-4 Ravichander et al. 2025 . This work addresses a growing real-world challenge in education: the use of generative AI systems in producing instructional materials, including diagrams, despite uncertainty about their quality. Diagrams play a central role in learning, particularly in computing education Naps et al. 2002 , yet their creation demands time and effort. As a result, AI-generated resources are increasingly used in practice, even though potential problems such as hallucinations Bernstein et al. 2025 present considerable risks to learners. Our project brings together research expertise from computing education and computational linguistics to develop and empirically evaluate a new ICL method to improve the reliability of automatically generated educational diagrams. Hallucinations in AI output can be classified into two general categories: factuality and faithfulness Huang et al. 2025 . As ICL relies on LLMs learning from prompt examples, it can be vulnerable to faithfulness hallucination, e.g., a model not adhering to demonstrations presented within a prompt. Empirical research shows that ICL struggles with overcoming preferences learned by a model during pre-training Kossen et al. 2024 , which may potentially cause issues with input context consistency in its output. Subpar AI-generated materials are not suitable for educational purposes, which is why our study proposes a novel method for diagram generation to improve quality of LLM-generated diagrams and minimize hallucination occurrence. Long and overly complex instructions have been shown to increase the rate of hallucinations Li et al. 2024 , and our method aims to simplify and tailor diagram generation prompt examples to minimize the risk of unexpected model output. Our approach uses the RST Mann and Thompson 1988 as a guide: we instruct LLMs to choose relevant in-context examples using RST analysis. Chosen examples contain RST clues that help LLMs visualize coherence relations in accordance with user expectations and therefore avoid issues with verbosity and unpredictable layout. To our knowledge, there has been no research investigating the use of RST as a scheme for this purpose. The study makes the following key contributions: (1) we propose a new RST-based method for selecting relevant demonstrations for ICL diagram code generation and compare it to the baseline zero-shot generation; (2) we present a five-scale rubric for assessing logical organization, connectivity, and aesthetic of diagrams that can be used for human and automated diagram evaluation; (3) we investigate the effect of LLM hallucination on various aspects of diagram quality. The preliminary results demonstrate that our method may decrease the occurrence of factual hallucination and improve diagram faithfulness to input context; however, multi-step generation pipelines increase the risk of LLM hallucination being carried from one step to another. We also identified that up to 20% of the AI-generated diagrams contained factual hallucination even when provided with a verified source text context. 2 Background 2.1 Rhetorical Structure Theory RST was initially developed for automatic generation of texts Taboada and Mann 2006 , making it a promising approach for automatic generation of diagrams. The theory explains text coherence through hierarchical relations between parts of a text Mann and Thompson 1988 ; Taboada and Mann 2006 . In an RST analysis, a coherent piece of text can be represented as a tree whose leaves are “elementary discourse units” (EDUs) connected with coherence relations Stede et al. 2017 . Relations vary depending on a specific RST framework Taboada and Mann 2006 , and in this work, we chose the RST annotation guide by Stede et al. Stede et al. 2017 . Their approach separates relations into multinuclear and mononuclear, where the latter can be pragmatic, semantic, or textual. Multinuclear relations consist of two or more nuclei, whereas mononuclear ones have only one nucleus. A nucleus is the main central unit of a span, and a satellite is secondary Stede et al. 2017 . André and Rist 1995 and Bateman et al. 2001 apply the RST for generation of coherent multimedia presentations. André and Rist 1995 argue that for sufficient understanding of a multimedia document, one needs to analyze relations between visual elements and between visual and textual elements. In their approach, they utilize rhetorical structure for planning multimedia presentations. Bateman et al. 2001 also note that visual and textual parts need to work in synergy rather than being simply placed in proximity. They demonstrate that the coherent expression of communicative intent and layout can be achieved by designing the latter on the basis of an RST-organized scheme. RST has also been adopted for multimodal diagram annotation Hiippala et al. 2020 . Similarly to those in texts Stede et al. 2017 , discourse relations in diagrams can be asymmetric (mononuclear) and symmetric (multinuclear) Hiippala et al. 2020 . 2.2 In-context Learning ICL has emerged as a popular method to teach a model to perform complex tasks without fine-tuning and updating model parameters Minaee et al. 2024 . Empirical research shows that it outperforms zero-shot inference Min et al. 2022 and makes it possible for a model to learn new tasks Kossen et al. 2024 . However, there has been a number of challenges in ICL applications revealed by empirical studies. First, it has been noted that ICL does not overrule preferences learned during pre-training Min et al. 2022 ; Kossen et al. 2024 , which questions its efficacy for tasks that are not present in a model’s training data