PromptPex: Automatic Test Generation for Language Model Prompts
Reshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn · 2025
Read PromptPex: Automatic Test Generation for Language Model Prompts by Reshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05070.txt.
Description
Reshabh Sharma and co-authors bring unit testing to the prompts that steer language models. Prompts now live inside software like code, taking inputs and returning outputs. Yet teams edit them blind: a tweak that fixes one case silently breaks others, and behavior shifts with the model underneath.
PromptPex reads a prompt's implied contract, then automatically generates test inputs and expected outputs, and grades the model's runs against them. It works with the specific model in use, since the same words behave differently across models. Test suites for prompts could do for prompt engineering what unit tests did for code: make change safe, reviewable, and repeatable.
Catalog record
| CREATOR | Reshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05070