Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations
Xiaozao WangNew York University Shanghai, [email protected] Zhewei WangNew York University Shanghai [email protected] Hongyi WenNew York University Shanghai [email protected] · 2026
Read Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations by Xiaozao WangNew York University Shanghai, [email protected] Zhewei WangNew York University Shanghai [email protected] Hongyi WenNew York University Shanghai [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31012.txt.
Introduction
Xiaozao Wang, Zhewei Wang and Hongyi Wen build a way to grade the interactivity of AI-generated teaching material. Explorable explanations let learners manipulate a simulation and watch it respond, and language models can now generate them quickly. But existing checks mostly test whether the code runs or the pictures look right, not whether the interactions themselves teach well. Their framework, EE-Eval, extracts a finite state machine from each generated explanation: the states a learner can reach, the actions offered, and the feedback given.
Each extracted machine is compared with an ideal one encoding the teaching intent, using graph metrics and embedding similarity over states, actions and feedback. Across thousands of generated explanations spanning 127 concepts from 6 AI models, EE-Eval separated interaction quality beyond surface correctness. It also aligned with human judgments of interactivity and teaching value far more strongly than existing baselines.
Catalog record
| CREATOR | Xiaozao WangNew York University Shanghai, [email protected] Zhewei WangNew York University Shanghai [email protected] Hongyi WenNew York University Shanghai [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31012
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations Xiaozao WangNew York University Shanghai, [email protected] Zhewei WangNew York University Shanghai [email protected] Hongyi WenNew York University Shanghai [email protected] Abstract While large language models now enable rapid generation of interactive learning materials, evaluating the interaction quality of these explorable explanations remains an open challenge. Existing benchmarks largely focus on code executability or visual fidelity, providing limited insight into dynamic interaction behaviors such as learner-controlled state transitions and context-sensitive system responses, which are factors that critically shape learners’ conceptual understanding. We present EE-Eval , an automated evaluation framework that formalizes interactivity as a finite space of learner-controllable states and transitions, represented as a Finite State Machine (FSM). By extracting FSMs from AI-generated explorable explanations, EE-Eval externalizes implicit interaction logic into an explicit, machine-interpretable graph. Evaluation is performed by comparing each generated FSM to an ideal FSM that encodes pedagogical intent, using a combination of graph-based metrics and embedding-based comparison of states, actions, and feedback to measure their structural and semantic similarity. Across thousands of generated explorable explanations spanning 127 concepts and produced by 6 AI models, EE-Eval consistently differentiates interaction quality beyond surface-level criteria such as functional correctness or visual quality, and exhibits substantially stronger alignment with human judgments of interactivity and pedagogical effectiveness than existing baselines. By framing interactivity as testable behavioral models rather than an emergent byproduct of LLM generation, EE-Eval transforms evaluation into a reflective diagnostic tool, enabling pedagogically grounded and actionable human-AI collaboration in creating interactive educational content. Keywords Explorable Explanations ⋅ \cdot Interactive Learning Systems ⋅ \cdot Finite State Machines ⋅ \cdot Automated Evaluation ⋅ \cdot Large Language Models ⋅ \cdot Knowledge Representation ⋅ \cdot Human-AI Collaboration 1 Introduction Explorable explanations have reshaped how complex computational concepts are taught and understood, shifting learning from passive consumption to active engagement through interaction. By enabling learners to manipulate parameters, observe system behaviors, and test hypotheses, explorable explanations foster intuition-building that static explanations often cannot, leading to their adoption in various domains. Recent advances in generative AI further enable the automatic generation of explorable explanations from high-level specifications or natural language prompts, including interactive visualizations and learning interfaces [ 5 , 6 , 14 ] . Systems such as Chat2VIS [ 20 ] , ChartLlama [ 13 ] , and MatPlotAgent [ 32 ] illustrate this growing capability. However, evaluating the quality of AI-generated explorable explanations remains an open challenge. Existing approaches primarily assess static outputs such as screenshots, textual summaries, or rendered charts using benchmarks like nvBench [ 19 ] and VisEval [ 4 ] . While effective for measuring visual correctness and data fidelity, these methods fail to capture interaction dynamics and pedagogical effectiveness, which are central to learning outcomes. Evaluating explorable explanations differs from assessing static educational content. Effectiveness depends not only on visual or textual fidelity, but on the interaction structure itself: what states learners can reach, what actions are available at each state, and how actions affect system behavior. Yet this interaction logic is implicit, embedded in low-level implementation details such as JavaScript event handlers and DOM manipulation [ 7 , 8 ] , making it difficult for humans and automated systems to reason about or compare interactive behaviors. To address this gap, we propose EE-Eval , a framework that makes interaction logic explicit by representing explorable explanations as finite state machines (FSMs). This abstraction preserves interaction semantics while hiding implementation details. Each FSM encodes (1) meaningful interaction states, (2) user-triggered and conditional transitions, and (3) interface components involved in these transitions, yielding a shared and interpretable structure for analysis and benchmarking purposes [ 11 , 12 , 18 ] . Building on this representation, EE-Eval introduces structural and semantic similarity metrics to compare AI-generated explanations with ground truth references at the level of interaction behavior. Rather than evaluating whether explanations look similar, EE-Eval assesses whether they support comparable exploration and reasoning patterns that are essential for learning. We integrate a testing pipeline to measure coverage of states, transitions, and interface elements at scale. Experiments across algorithmic explorable explanations generated by multiple AI systems show that FSM-based evaluation yields more stable and informative signals than visual-based or execution-driven baselines, particularly in capturing interactivity and pedagogical structure, while producing interpretable diagnostics that complement human judgment. Our contributions are summarized as follows: 1. We introduce an FSM-based representation that explicitly models the interaction logic of explorable explanations with a machine-interpretable graph. 2. We propose interaction-level structural and semantic metrics for evaluating AI-generated explorable content that aligns with human judgements. 3. We demonstrate how automated evaluation can function as a collaborative analytic tool for educators and researchers, supporting reflective assessment and iterative improvement of AI-generated learning materials. 2 Related Work 2.1 Interactive Learning Systems Interactive learning systems emphasize learning through active exploration, hypothesis testing, and feedback. Prior work on interactive simulations and explorable explanations shows that direct manipulation and immediate system response can significantly improve conceptual understanding and engagement [ 1 , 9 , 16 , 24 ] . Empirical studies across STEM domains demonstrate that learners benefit more from interacting with dynamic models than from passively consuming static representations [ 1 , 16 , 17 ] . Systems such as TensorFlow Playground 1 1 1 https://playground.tensorflow.org/ , CNN Explainer [ 27 ] , Augmented Math [ 7 ] , and Augmented Physics [ 11 ] exemplify this paradigm by allowing users to manipulate parameters and observe state changes in real time. These systems tightly couple user actions with system feedback, reinforcing learning-by-doing and exploratory reasoning. However, their interaction structures are manually designed and embedded implicitly in code, limiting scalability, adaptability, and systematic analysis of interaction behavior. 2.2 AI-Generated Interactive Learning Materials Advances in large language models have shifted educational interfaces from manual authoring toward AI-generated learning materials. Meta-analyses report growth in systems that generate explanations, exercises, visualizations, and adaptive learning pathways [ 2 , 29 ] . Empirical studies show engagement and efficiency in AI-generated tutoring and content delivery [ 8 , 17 ] , while XR-based platforms integrate generative models into learning environments [ 10 , 33 ] . Within visualization and interface generation, prior work explores natural language–driven pipelines such as Chat2VIS [ 20 ] , ChartLlama [ 13 ] , NVAgent [ 22 ] , and NL2INTERFACE [ 6 ] . Agent-based and multimodal systems extend these ideas to pedagogical settings, including EduVisAgent [ 15 ] , TheoremExplainAgent [ 18 ] , Interactive Sketchpad [ 5 ] , Auto-Slides [ 31 ] , and ReactGenie [ 30 ] , demonstrating that LLMs can generate executable interactive interfaces. However, interaction is treated as an assumed outcome rather than a modeled object. Generated interfaces are presumed to function correctly, with limited mechanisms for verification or comparison. As interactivity shifts from a design-time artifact to a generation-time result, the lack of a machine-readable and verifiable representation becomes a limitation, particularly in educational contexts where correctness, controllability, and learner agency are critical [ 18 , 14 , 29 ] . 2.3 Evaluation for AI-Generated Learning Interfaces As AI systems generate interactive learning interfaces, evaluation has not kept pace with this shift. Most existing methods assess artifacts as outputs or isolated UI events, rather than modeling interaction as a state-driven