CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models
Dong Wang · 2025
Read CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models by Dong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13261.txt.
Description
Dong Wang introduces a benchmark from 2025 for testing how well large language models know Chinese pedagogical grammar. Models such as ChatGPT now shape foreign language teaching, yet their grasp of grammar as taught in class is poorly measured. The benchmark is called CPG-EVAL and is built for that gap.
Five tasks check grammar recognition, fine grained distinction, category judgment, and resistance to confusing input. Small models handled single instance tasks but struggled with multiple instances and interference. Large models resisted interference better but still left much room for gains in accuracy. The author argues for better teaching alignment and stricter tests before models guide language teaching at scale.
Catalog record
| CREATOR | Dong Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13261