On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language

Sébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected] · 2025

Read On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language by Sébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19136.txt.

Description

Sébastien Salva and Redha Taguelmimt test whether language-model agents can run software tests that are written as plain English instructions. Such natural-language test cases for graphical interfaces are cheaper to write and maintain than coded scripts. But the instructions can be ambiguous, so an agent may report failures that are not real bugs, a flaw the authors call unsoundness. Rerunning the same test may also give different outcomes, which undermines trust in the results.

Their algorithm adds guardrails and specialist agents that check each step as it runs. They define measures for execution skill and consistency, plus a notion of weak unsoundness tied to industrial Six Sigma quality levels. Across eight public models, the largest, a 70-billion-parameter Llama model, performed acceptably with high consistency. Smaller models showed clear limits, mapping where the approach stands today.

Catalog record

CREATORSébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19136

Browse the full library · About the Tower of Babel