Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation

Pei-Yu Tseng, Mahir Akgun, Peng Liu · 2026

Read Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation by Pei-Yu Tseng, Mahir Akgun, Peng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23629.txt.

Description

Tseng, Akgun, and Liu examine why language models misjudge the quality of pretest questions, in a 2026 paper on education technology. Pretesting asks students questions before teaching the material, and the attempt itself improves later learning. Good pretest questions need the right openness, cognitive depth, and fit to learning goals, but writing and grading them by hand is slow. The authors build an AI assisted workflow that generates questions, scores them with a rubric, and keeps the best.

Because machine scores do the filtering, the study tests how well machine and human judgments align. A two by two design varies the rubric wording and the scoring mode. Disagreements turn out to be systematic, not random noise. Revising the rubric to state quality criteria more explicitly helps alignment more than asking the model to reason first. The two fixes also stack, each adding gains the other misses.

Catalog record

CREATORPei-Yu Tseng, Mahir Akgun, Peng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23629

Browse the full library · About the Tower of Babel