LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms

Ali Hamdi 1, Ahmed Abdelmoneim Mazrou 1, Mohamed Shaltout 2 · 2024

Read LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms by Ali Hamdi 1, Ahmed Abdelmoneim Mazrou 1, Mohamed Shaltout 2 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13765.txt.

Introduction

Ali Hamdi, Ahmed Abdelmoneim Mazrou, and Mohamed Shaltout introduced in 2024 a way to score student engagement on e-learning platforms from video data and comment feelings. Older methods lean on surveys with small samples or thin metadata, and student comments are often fuzzy in tone. Their metric lets large language models read comment sentiment, then blends it with normalized views and likes. Scores come out per lesson and per course.

The team compared several language models and fine-tuned one on human-labeled sentiment to sharpen its reads. Two open models were also run locally for the pipeline. Tests showed the metric is both scalable and accurate, the authors report. It aims to give course teams a live read on engagement without pestering students with forms.

Catalog record

CREATORAli Hamdi 1, Ahmed Abdelmoneim Mazrou 1, Mohamed Shaltout 2
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13765

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms Ali Hamdi 1, Ahmed Abdelmoneim Mazrou 1, Mohamed Shaltout 2 Abstract Current methods for analyzing student engagement in e-learning platforms, including automated systems, often struggle with challenges such as handling fuzzy sentiment in text comments and relying on limited metadata. Traditional approaches, such as surveys and questionnaires, also face issues like small sample sizes and scalability. In this paper, we introduce LLM-SEM (Language Model-Based Student Engagement Metric), a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement. By utilizing recent Large Language Models (LLMs), we generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes. Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels. Extensive experiments were conducted to evaluate various LLM models, demonstrating the effectiveness of LLM-SEM in providing a scalable and accurate measure of student engagement. We fine-tuned TXLM-RoBERTa using human-annotated sentiment datasets to enhance prediction accuracy and utilized LLama 3B, and Gemma 9B from Ollama. † † institute: 1 AiTech AU, Preston, Australia , 2 Mawhiba, Riyadh, KSA, 1 Introduction The proliferation of e-learning platforms has transformed the way students engage with educational content [ 22 ] . Platforms such as YouTube host numerous educational channels that cater to diverse learning needs [ 19 ] . However, assessing student engagement with these resources remains a significant challenge for educators and content creators [ 14 ] . Traditional methods, such as surveys and questionnaires, often suffer from small sample sizes, lack of scalability, and difficulty in capturing the opinions of a broader audience [ 15 ] . Automated methods, while an improvement, face their own challenges, including dealing with fuzzy sentiment in text comments and relying on limited metadata, which hampers their ability to provide a comprehensive assessment of engagement [ 23 ] . Sentiment analysis has been advancing applications across several domains [ 9 , 2 , 3 , 8 , 11 , 10 ] . Current sentiment analysis methods for student engagement, while useful, are often hindered by the ambiguity of user comments, leading to misinterpretations of student feedback. Furthermore, traditional metrics heavily rely on small-scale metadata or subjective self-reports, which do not scale well for large e-learning platforms. There is a need for a scalable, data-driven metric that can integrate both qualitative sentiment data and quantitative metadata to assess student engagement more accurately. To address these limitations, we propose LLM-SEM (Language Model-Based Student Engagement Metric), a novel quantitative method to gauge student engagement on e-learning platforms. By leveraging recent advances in Large Language Models (LLMs), LLM-SEM combines course and lesson video metadata with sentiment analysis of user comments, providing a more accurate and scalable solution. This approach overcomes the constraints of traditional methods by incorporating comprehensive metadata and high-quality sentiment predictions, allowing for a holistic evaluation of student engagement at both the course and lesson levels. Our research introduces an LLM-driven metric that addresses the challenges faced by current methods and provides a scalable solution to engagement analysis. We fine-tuned the RoBERTa model on human-annotated sentiment datasets to generate high-quality sentiment predictions that enhance the analysis of student feedback. Other state-of-the-art LLMs, such as TXLM-RoBERTa [ 17 ] , LLama 3B [ 25 ] , and Gemma 9B [ 24 ] , were evaluated without fine-tuning. Additionally, we introduce a methodology that normalizes key metadata features such as views and likes, integrating these with sentiment polarity scores to form a comprehensive engagement metric. This paper is organized as follows: Section 2 discusses related work on student engagement and sentiment analysis. Section 3 outlines our proposed methodology and the LLM-SEM model. Section 4 presents the experimental results and discussion of the findings. Finally, Section 5 concludes the paper. 2 Related Work Student engagement and feedback are crucial indicators of the effectiveness of educational content [ 20 ] . Traditionally, course evaluation methods have relied on qualitative approaches, such as surveys and questionnaires, to gather feedback on student engagement and satisfaction [ 18 ] . These methods, while valuable, face limitations in scalability, representativeness, and timeliness. With the growing volume of online educational content, there has been a shift towards automated and data-driven methods for assessing student engagement, including sentiment analysis and the use of metadata [ 6 ] . Sentiment analysis has been widely applied across various domains to gauge public opinion, including politics, marketing, and consumer reviews [ 11 , 5 , 2 , 3 , 13 , 12 ] . In the context of education, sentiment analysis has emerged as a promising tool for evaluating student satisfaction and engagement. Previous studies have demonstrated the potential of sentiment analysis in predicting student satisfaction by analyzing reviews and comments from various online learning platforms. Recent advancements in Large Language Models (LLMs) have revolutionized sentiment analysis, enabling more accurate and nuanced predictions of sentiment in textual data [ 26 ] . LLMs such as BERT [ 16 ] and its variations, including AraBERT and RoBERTa, have shown significant improvements in understanding the sentiment of complex and context-dependent text, addressing the limitations of earlier sentiment analysis methods. These models have been fine-tuned on a variety of sentiment datasets to enhance their performance in domain-specific applications. Leveraging LLMs for sentiment analysis in educational contexts can help overcome the challenges of fuzzy and ambiguous student comments, providing more accurate insights into student satisfaction [ 1 ] . Combining sentiment polarity scores with video metadata is another area of growing interest in the educational domain. Metadata such as views, likes, and the duration of student interaction with course materials can provide valuable insights into engagement levels [ 7 ] . Some studies have proposed methods that incorporate both sentiment analysis and metadata to compute engagement scores, particularly in non-educational settings like social media and entertainment platforms [ 21 ] . However, applying these techniques to educational platforms, especially on YouTube, is still in its early stages. By integrating comprehensive metadata with sentiment analysis, we can achieve a more holistic and scalable approach to measuring student engagement. In this paper, we build on these developments by proposing a novel metric, LLM-SEM , which combines sentiment analysis using LLMs with course and lesson metadata to compute student engagement scores. Our approach addresses the limitations of current methods by incorporating both the textual sentiment and the metadata features to gauge student engagement more accurately and at scale. 3 Methodology Our methodology involves several stages, each representing a distinct step in the process of calculating the Student Engagement Metric (SEM). These stages include data collection, metadata extraction, sentiment analysis, polarity scoring, feature space normalization, and finally, the calculation of a comprehensive engagement metric by integrating both sentiment and metadata. A high-level workflow is shown in Fig. 1 , which demonstrates how course lessons, metadata, and comments are processed to compute the final SEM. Figure 1: A workflow illustrating the process of calculating the Student Engagement Metric (SEM) using course lessons, metadata, comments, and sentiment analysis. Starting from (a) the course lessons, (b) metadata is extracted and normalized using Min-Max normalization (g) to contribute to the final SEM (h). Simultaneously, (c)

Read the complete text