PromptPex: Automatic Test Generation for Language Model Prompts

Reshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn · 2025

Read PromptPex: Automatic Test Generation for Language Model Prompts by Reshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05070.txt.

Description

Reshabh Sharma and co-authors bring unit testing to the prompts that steer language models. Prompts now live inside software like code, taking inputs and returning outputs. Yet teams edit them blind: a tweak that fixes one case silently breaks others, and behavior shifts with the model underneath.

PromptPex reads a prompt's implied contract, then automatically generates test inputs and expected outputs, and grades the model's runs against them. It works with the specific model in use, since the same words behave differently across models. Test suites for prompts could do for prompt engineering what unit tests did for code: make change safe, reviewable, and repeatable.

Catalog record

CREATORReshabh K Sharma, Jonathan de Halleux, Shraddha Barke, Benjamin Zorn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05070

Browse the full library · About the Tower of Babel