CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models

Dong Wang · 2025

Read CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models by Dong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13261.txt.

Description

Dong Wang introduces a benchmark from 2025 for testing how well large language models know Chinese pedagogical grammar. Models such as ChatGPT now shape foreign language teaching, yet their grasp of grammar as taught in class is poorly measured. The benchmark is called CPG-EVAL and is built for that gap.

Five tasks check grammar recognition, fine grained distinction, category judgment, and resistance to confusing input. Small models handled single instance tasks but struggled with multiple instances and interference. Large models resisted interference better but still left much room for gains in accuracy. The author argues for better teaching alignment and stricter tests before models guide language teaching at scale.

Catalog record

CREATORDong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13261

Browse the full library · About the Tower of Babel