ArkEval: Benchmarking and Evaluating Automated Code Repair for ArkTS CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software reliability

Bang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Chenhao Ying, Yuan Luo · 2026

Read ArkEval: Benchmarking and Evaluating Automated Code Repair for ArkTS CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software reliability by Bang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Chenhao Ying, Yuan Luo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08866.txt.

Description

HarmonyOS apps are written in ArkTS, a typed cousin of TypeScript, yet the platform has lacked a solid test for automated code repair. Bang Xie and five co-authors fill that gap with ArkEval, a benchmark and evaluation workflow built from a large official Huawei repository holding over 400 ArkTS applications. Careful multi-stage filtering leaves 502 reproducible issues.

Test cases are generated by language models and voted on, including Claude, and problem statements are standardized for fair comparison. Four leading language models are then tested with a retrieval-augmented repair workflow. The scores map both the promise and the limits of today's models in this low-resource language, pointing the way for future repair tools.

Catalog record

CREATORBang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Chenhao Ying, Yuan Luo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-08866

Browse the full library · About the Tower of Babel