Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education

Naiming Liu, Shashank Sonkar, Debshila Basu Mallick, Richard Baraniuk, Zhongzhou Chen · 2024

Read Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education by Naiming Liu, Shashank Sonkar, Debshila Basu Mallick, Richard Baraniuk, Zhongzhou Chen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09914.txt.

Introduction

Naiming Liu and co-authors present work from two thousand twenty-four on labeling physics questions with fine-grained learning goals. A standard syllabus states broad goals, but tutors need to know the exact skills each question tests. Their atomic system breaks goals into small units written as subject, verb, and object.

The system covers nine chapters of an introductory college physics course. The team applied it to one hundred thirty-one questions from expert banks and a standard textbook, tagging each with up to eight atomic goals. They then tested whether large language models can do the tagging as well as human experts, under several prompting styles. The analysis shows where the models match experts and where they stray, with lessons for tutoring systems and personal study paths.

Catalog record

CREATORNaiming Liu, Shashank Sonkar, Debshila Basu Mallick, Richard Baraniuk, Zhongzhou Chen
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09914

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education Naiming Liu, Shashank Sonkar, Debshila Basu Mallick, Richard Baraniuk, Zhongzhou Chen Abstract This paper introduces a novel approach to create a high-resolution “map" for physics learning: an "atomic" learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem solving in a college-level introductory physics course. Our method leverages Large Language Models (LLMs) for automated labeling of physics questions and introduces a comprehensive set of metrics to evaluate the quality of the labeling outcomes. The atomic LO system, covering nine chapters of an introductory physics course, uses a “subject-verb-object” structure to represent specific cognitive processes. We apply this system to 131 questions from expert-curated question banks and the OpenStax University Physics textbook. Each question is labeled with 1-8 atomic LOs across three chapters. Through extensive experiments using various prompting strategies and LLMs, we compare automated LOs labeling results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs, providing insight into LLMs reasoning processes for labeling LOs and identifying areas for improvement in LOs system design. Our work contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions. Our findings have significant implications for the development of intelligent tutoring systems and personalized learning pathways in STEM education, paving the way for more effective “learning GPS” systems. 1 Introduction The rapid advancement of Artificial Intelligence (AI) has opened new avenues for developing intelligent learning systems that improve student learning outcomes [ 18 , 17 , 15 ] . An ideal learning system should function as a “learning GPS,” capable of assessing a student’s current learning states and level of mastery, then guiding them towards the most suitable learning resources. However, current versions of a “learning GPS” suffers from “low-resolution maps” of knowledge domains, particularly in STEM disciplines. Existing learning objective systems, designed primarily for human labeling, often provide only broad descriptions of skills and concepts related to learning resources. For instance, in the OpenStax University Physics textbook, each sub-chapter is labeled with only 2-3 learning objectives, despite containing 20-40 end-of-chapter review questions. This low granularity limits the effectiveness of learning guidance, similar to providing vague driving directions. The fundamental reason behind the low-resolution map lies in the design of taxonomy systems designed for human domain experts. This approach not only restricts the number of objectives that can be tagged but also leads to potential degeneracy and ambiguity among learning objectives. In this paper, we present a novel approach to create a higher-resolution “map” for physics learning: 1). We introduce an “atomic” learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem-solving. 2). We use Large Language Models (LLMs) for reliable automated labeling of learning resources. 3). We propose a comprehensive set of metrics to quantitatively and qualitatively evaluate the quality of LLMs labeling outcomes. Our atomic LO system, developed for a college-level introductory physics course, covers nine chapters. It builds upon previous work on physics taxonomy systems, such as NTEO [ 19 ] . Each atomic LO describes a specific cognitive process in physics problem-solving using a constrained set of verbs and adheres to a “subject-verb-object” structure, representing “input,” “cognitive action,” and “output.” We apply the atomic LO system to a diverse set of 131 questions from expert-created problem banks for a physics course and OpenStax University-physics textbook. The questions are selected from three chapters and each question is tagged with 1-8 atomic LOs. We also conduct extensive experiments using various prompting strategies and LLMs to automate the LOs labeling process and compare the results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs in automated LOs labeling. We identify performance patterns that provide insights into LLM reasoning processes and highlight potential areas for improvement in both LLM capabilities and LO system design for AI applications. Our research contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions, which paves the foundation for designing intelligent tutoring systems and personalized learning pathways in STEM education. 2 Related Works 2.1 Learning Objectives Construction in Education Learning objectives are usually effective instructional design in education, with early approaches relying on manually developed taxonomies emphasizing on cognitive processes, such as Bloom’s taxonomy [ 2 ] . In physics education, curated systems often emphasize conceptual clarity and progression, such as the Force Concept Inventory [ 8 ] , which assesses student understanding of fundamental mechanics concepts. Other initiatives, like PhET simulations, integrate conceptual frameworks to enhance student engagement and comprehension [ 14 ] . 2.2 Automated Learning Objectives Labeling Expert labeling with learning objectives, while accurate, is impractical for large datasets due to variability in interpretation and potential fatigue. In contrast, automated methods for learning objective labeling and generation have emerged as useful tools that can facilitate human experts [ 3 ] . Previous research has explored mapping textbook materials and questions to concepts and skills using matrix factorization and VAE-based methods [ 4 , 13 ] , alongside studies focused on classifying knowledge components in mathematics and chemistry [ 12 , 20 ] . Another line of research has applied topic modeling and hierarchical clustering to automatically derive learning objectives from course content [ 7 ] . Recent advancements in natural language processing have opened new avenues for automating the process of identifying and labeling LOs. For instance, [ 21 ] uses GPT-3 to classify questions with LOs from OpenStax textbooks, while [ 9 ] explores the abilities of LLMs in tagging multilingual problem content with the appropriate skill from a taxonomy. LLMs have also shown promising results in various educational task, including automated grading, intelligent tutoring systems and student modeling [ 10 , 17 , 11 , 16 ] . However, LLM’s application specifically for labeling or generating learning objectives in physics education remains a relatively unexplored area. Our research addresses this gap by utilizing LLMs to automate the creation and labeling of LOs for physics-related questions. 3 Problem Formulation In this section, we provide the problem formulation for automated learning objective labeling using LLMs. Let P=\{p_{1},p_{2},\ldots,p_{n}\} be a set of physics problems and L=\{l_{1},l_{2},\ldots,l_{m}\} be a set of learning objectives. Define the ground truth mapping G:P\rightarrow\mathcal{P}(L) , where \mathcal{P}(L) is a power set of L . Given a LLM \mathcal{M} , define a function F_{\mathcal{M}}:A\times P\times\mathcal{P}(L)\rightarrow\mathcal{P}(L) such that F_{\mathcal{M}}(a,p_{i},S)=L^{\prime}\subseteq S where • a\in A is a prompting strategy (e.g., simple, explanation, chain of thought) • p_{i}\in P is a physics problem • S\subseteq L is a subset of learning objectives provided to the LLM • L^{\prime} is the subset of learning objectives predicted by the LLM for question p_{i} For the learning objectives labeling task, we aim to 1. Systematically assess the efficacy of various Large Language Models \mathcal{M} and prompting strategies a\in A in predicting learning objectives for physics problems. The assessment includes: • Quantitative comparison of prediction accuracy across different model architectures and sizes • Analysis of the impact of diverse prompting techniques on labeling performance • Qualitative assessment of the models’ ability to capture detailed learning objectives • Identification of patterns, strengths, and limitations

Read the complete text