HardTests : Synthesizing High-Quality Test Cases for LLM Coding

Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li · 2025

Read HardTests : Synthesizing High-Quality Test Cases for LLM Coding by Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24098.txt.

Description

Grading code written by AI needs test cases that catch clever wrong answers, and a 2025 paper by Zhongmou He and co-authors builds a machine for making them. For hard coding problems, only careful human edge cases expose a well-disguised bad solution, and those cases are hard to make at scale.

Their pipeline, HARDTESTGEN, uses language models to synthesize sharp tests. With it they built HARDTESTS, a set of forty-seven thousand contest problems with fresh tests. The new tests beat older ones at flagging bad code, with gains up to forty points on the hardest problems. Training models on the set also improved their later coding scores. The data set and pipeline are set for open release.

Catalog record

CREATORZhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24098

Browse the full library · About the Tower of Babel