Polishing Every Facet of the GEM : Testing Linguistic Competence of LLMs and Humans in Korean
SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee · 2025
Read Polishing Every Facet of the GEM : Testing Linguistic Competence of LLMs and Humans in Korean by SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01237.txt.
Description
SungHo Kim and co-authors build a Korean grammar test in a 2025 paper on language models. Their benchmark, KoGEM, holds about 1,500 multiple choice questions. The items span five main areas and sixteen sub areas of grammar. Both large language models and human takers face the same items, so scores can be compared head to head.
Twenty-seven models of mixed size and type took the test with no prior examples. Models did well on plain fact and rule questions. They stumbled where answers need lived knowledge of the language, like sound rules and pronunciation. The authors argue that feeding models such grounded knowledge could lift their grammar skill. The open test set gives later teams a shared yardstick for Korean language skill.
Catalog record
| CREATOR | SungHo Kim, Nayeon Kim, Taehee Jeon, SangKeun Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01237