Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging

Philipp Straubinger, Marvin Kreis, Stephan Lukasczyk *, Gordon Fraser · 2025

Read Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging by Philipp Straubinger, Marvin Kreis, Stephan Lukasczyk *, Gordon Fraser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08182.txt.

Description

Philipp Straubinger and three co-authors test in 2025 whether a scientific debugging loop helps language models write tests that kill mutants, or seeded code faults. Mutation testing grades test quality by fault catching. Models tend to mimic past tests rather than reason about code behavior.

In the loop, the model states a hypothesis for killing one mutant, drafts a test, and refines it with stated reasons. Rivals are direct drafting, blind retry, and a search-based tool. Model tests beat the search tool on fault catching and coverage, at higher compute cost. The stepwise loop adds the most on stubborn mutants. Equivalent mutants, which no test can kill, are also flagged with fair success.

Catalog record

CREATORPhilipp Straubinger, Marvin Kreis, Stephan Lukasczyk *, Gordon Fraser
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08182

Browse the full library · About the Tower of Babel