HardTests : Synthesizing High-Quality Test Cases for LLM Coding
Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li · 2025
Read HardTests : Synthesizing High-Quality Test Cases for LLM Coding by Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24098.txt.
Description
Grading code written by AI needs test cases that catch clever wrong answers, and a 2025 paper by Zhongmou He and co-authors builds a machine for making them. For hard coding problems, only careful human edge cases expose a well-disguised bad solution, and those cases are hard to make at scale.
Their pipeline, HARDTESTGEN, uses language models to synthesize sharp tests. With it they built HARDTESTS, a set of forty-seven thousand contest problems with fresh tests. The new tests beat older ones at flagging bad code, with gains up to forty points on the hardest problems. Training models on the set also improved their later coding scores. The data set and pipeline are set for open release.
Catalog record
| CREATOR | Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24098