Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support

Devika Venugopalan, Ziwen Yan, Conrad Borchers, Jionghao Lin, Vincent Aleven · 2024

Read Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support by Devika Venugopalan, Ziwen Yan, Conrad Borchers, Jionghao Lin, Vincent Aleven (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11995.txt.

Introduction

Parents want to help with homework but often face math taught differently from their own school days. Devika Venugopalan and co-authors build a chat assistant for those parents in a 2024 study, pairing a language model with the live problem-solving context of a tutoring system. The tutoring side supplies what the child is working on and how, keeping the model's advice grounded.

Prompt trials found that few worked examples plus the live context produced the most useful suggestions. Ten middle school parents then tried the messages and prized help that explained the content and prompted children to explain their own thinking. The study maps how to merge the two technologies.

Catalog record

CREATORDevika Venugopalan, Ziwen Yan, Conrad Borchers, Jionghao Lin, Vincent Aleven
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Combining Large Language Models with Tutoring System Intelligence: A Case Study in Caregiver Homework Support Devika Venugopalan, Ziwen Yan, Conrad Borchers, Jionghao Lin, Vincent Aleven Abstract . Caregivers (i.e., parents and members of a child’s caring community) are underappreciated stakeholders in learning analytics. Although caregiver involvement can enhance student academic outcomes, many obstacles hinder involvement, most notably knowledge gaps with respect to modern school curricula. An emerging topic of interest in learning analytics is hybrid tutoring, which includes instructional and motivational support. Caregivers assert similar roles in homework, yet it is unknown how learning analytics can support them. Our past work with caregivers suggested that conversational support is a promising method of providing caregivers with the guidance needed to effectively support student learning. We developed a system that provides instructional support to caregivers through conversational recommendations generated by a Large Language Model (LLM). Addressing known instructional limitations of LLMs, we use instructional intelligence from tutoring systems while conducting prompt engineering experiments with the open-source Llama 3 LLM. This LLM generated message recommendations for caregivers supporting their child’s math practice via chat. Few-shot prompting and combining real-time problem-solving context from tutoring systems with examples of tutoring practices yielded desirable message recommendations. These recommendations were evaluated with ten middle school caregivers, who valued recommendations facilitating content-level support and student metacognition through self-explanation. We contribute insights into how tutoring systems can best be merged with LLMs to support hybrid tutoring settings through conversational assistance, facilitating effective caregiver involvement in tutoring systems. Conference: LAK25: The 15th International Learning Analytics and Knowledge Conference; March 03–07, 2025; Dublin, Ireland LAK25: The 15th International Learning Analytics and Knowledge Conference (LAK 2025), March 03–07, 2025, Dublin, Ireland DOI: 10.1145/3706468.3706516 ISBN: 979-8-4007-0701-8/25/03 CCS: Human-centered computing Human computer interaction (HCI) CCS: Applied computing Education 1. Introduction Caregiver (i.e., parents and members of a child’s caring community) involvement positively contributes to student outcomes, including academic performance and motivation ( Hoover-Dempsey and Sandler, 1997 ) . However, limited research has studied caregiver involvement in learning analytics applications, such as tutoring systems and student dashboards. While tutored students learn at approximately the same rate per practiced problem-solving step, large-scale analyses found notable variation in prior knowledge ( Koedinger et al., 2023 ) . If learning rates are near-constant and prior knowledge is variable, more practice opportunities could help close knowledge gaps. Motivational human support beyond cognitive tutoring, which caregivers can provide ( Hoover-Dempsey and Sandler, 1997 ) , is needed to help increase opportunities to practice. To do so, our field must study systems and analytics that enhance human support in tutoring system practice; a key idea we investigated was the need for caregivers to receive instructional support, along with guidance on effective tutoring strategies. One emerging topic of interest in learning analytics is hybrid tutoring, where students are supported by an intelligent system and a human tutor ( Thomas et al., 2023 ) . Hybrid tutoring provides both instructional and motivational guidance ( Thomas et al., 2023 ) ; this is similar to roles caregivers assert in homework support ( Hoover-Dempsey and Sandler, 1997 ) . However, caregivers often struggle in proving adequate instructional homework support to students ( Pelemo, 2022 ) . To address this issue, recent research has investigated novel ways in which caregivers can be involved in tutoring systems ( Nguyen et al., 2024 ) . One promising direction suggested by this line of work is providing instructional support to caregivers. This can bridge knowledge gaps that caregivers often report regarding modern math curricula ( Pelemo, 2022 ; Nguyen et al., 2024 ) . Other guidance can include suggesting effective tutoring practices for caregivers. However, while prior work suggested that caregivers appreciate conversational recommendations ( Nguyen et al., 2024 ) , they also noted a lack of personalization and contextual relevance in pre-generated messages. The current study addresses generation of caregiver message recommendations in a hybrid tutoring context, in real time. LLMs have demonstrated great potential to support learning analytics applications and learners, including conversational support for debugging during problem solving ( Ma et al., 2024 ) , contextual reflection triggers in collaborative learning ( Naik et al., 2024 ) , prediction of self-regulated learning ( Zhang et al., 2024a ) , and virtual teaching assistants ( Liu et al., 2024b ) , among others. These advancements promise to improve human learning through conversational tutoring. While promising, a fundamental challenge in the design of these tools is that foundation models are domain-general: they lack expert knowledge on domain-specific instruction. To circumvent this issue, recent research has suggested using instructional material as prompting aids that induce automated tutoring in a domain ( Schmucker et al., 2024 ) . Other work has suggested prompting foundation models with pedagogically meaningful guardrails, such as ensuring that answers are withheld, as is practiced in effective human tutoring ( Pal Chowdhury et al., 2024 ) . We study the integration of LLMs into a middle school tutoring system for equation-solving, Lynnette ( Long et al., 2018 ) , in a hybrid tutoring context, an emerging area of interest in learning analytics ( Thomas et al., 2024 ) . We designed the Caregiver Conversational Support Tool (CCST) , which provides personalized, contextually relevant message recommendations through LLMs for caregivers supporting learners in tutoring systems. To our knowledge, this study is the first to evaluate LLM-generated message recommendations based on both contextual information supplied by a tutoring system (e.g., hint use, accuracy, and next viable problem-solving steps) and tutoring principles. The current study presents two angles on developing and evaluating LLM-integrated tutoring systems. First, we describe prompt engineering experiments combining the tutoring system’s instructional model, tutoring principles, and LLM instruction. Second, we describe an evaluation study with ten middle school caregivers involved in hybrid tutoring using the CCST, including prototype feedback and perceptions of generated conversational support during live student support. We answer these two primary research questions: RQ1: How can an LLM best generate conversational recommendations for middle school caregivers based on problem-solving context? RQ2: How do middle school caregivers perceive these conversational recommendations? 2. Background 2.1. Caregiver Support in Learning Caregivers play a key role in their child’s academic success ( Hoover-Dempsey and Sandler, 1997 ) . As homework increasingly moves online ( Magalhães et al., 2020 ) , technology-integrated caregiver support offers new involvement opportunities ( Mrazek et al., 2021 ) . Yet, research on supporting caregivers with learning analytics is limited, aside from studies on AI acceptance ( Martens et al., 2024 ) . Previous work has focused on indirect support, like notification features ( Broderick et al., 2011 ) . However, these do not enable caregivers to provide direct instructional assistance during homework, missing the potential for caregivers to actively enhance academic performance ( Hoover-Dempsey and Sandler, 1997 ) . Notably, different types of caregiver support can have varying impacts on their child’s academic performance ( Doctoroff and Arnold, 2017 ) . Given the importance of effective caregiver involvement in homework ( Hoover-Dempsey and Sandler, 1997 ) , there is a research opportunity to explore how learning analytics can support caregivers in enhancing student learning. Many caregivers struggle with homework support due to unfamiliarity with modern curricula or lack of tutoring experience

Read the complete text