CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows

Rabeya Khatun Muna, Md Nakhla Rafi, Tse-Hsun (Peter) Chen · 2026

Read CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows by Rabeya Khatun Muna, Md Nakhla Rafi, Tse-Hsun (Peter) Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27148.txt.

Description

Muna, Rafi, and Chen introduce CI-Repair-Bench, a benchmark for fixing continuous integration failures at the repository level using real GitHub Actions runs. It holds 567 failure cases from 103 repositories and judges repairs only through full CI workflows, not isolated tests. Unlike older repair benchmarks, it covers non-code artifacts, environment and dependency faults, noisy logs, and workflow constraints that make CI repair harder than source-only fixes.

Catalog record

CREATORRabeya Khatun Muna, Md Nakhla Rafi, Tse-Hsun (Peter) Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73501
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27148

Browse the full library · About the Tower of Babel