On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language
Sébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected] · 2025
Read On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language by Sébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19136.txt.
Description
Sébastien Salva and Redha Taguelmimt test whether language-model agents can run software tests that are written as plain English instructions. Such natural-language test cases for graphical interfaces are cheaper to write and maintain than coded scripts. But the instructions can be ambiguous, so an agent may report failures that are not real bugs, a flaw the authors call unsoundness. Rerunning the same test may also give different outcomes, which undermines trust in the results.
Their algorithm adds guardrails and specialist agents that check each step as it runs. They define measures for execution skill and consistency, plus a notion of weak unsoundness tied to industrial Six Sigma quality levels. Across eight public models, the largest, a 70-billion-parameter Llama model, performed acceptably with high consistency. Smaller models showed clear limits, mapping where the approach stands today.
Catalog record
| CREATOR | Sébastien Salva 1 and Redha Taguelmimt 1 1 LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, Aubière, France 2 Department of Computing, Main University, MySecondTown, MyCountry [email protected], [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19136