Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation
Pei-Yu Tseng, Mahir Akgun, Peng Liu · 2026
Read Why Machines Misread Pedagogical Quality: Human–Machine Alignment in LLM-Based Pretest Question Evaluation by Pei-Yu Tseng, Mahir Akgun, Peng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23629.txt.
Description
Tseng, Akgun, and Liu examine why language models misjudge the quality of pretest questions, in a 2026 paper on education technology. Pretesting asks students questions before teaching the material, and the attempt itself improves later learning. Good pretest questions need the right openness, cognitive depth, and fit to learning goals, but writing and grading them by hand is slow. The authors build an AI assisted workflow that generates questions, scores them with a rubric, and keeps the best.
Because machine scores do the filtering, the study tests how well machine and human judgments align. A two by two design varies the rubric wording and the scoring mode. Disagreements turn out to be systematic, not random noise. Revising the rubric to state quality criteria more explicitly helps alignment more than asking the model to reason first. The two fixes also stack, each adding gains the other misses.
Catalog record
| CREATOR | Pei-Yu Tseng, Mahir Akgun, Peng Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.23629