What You Prompt Is What You Get: Increasing Transparency Using Prompt Cards
Amandine M. Caut, Beimnet Zenebe, Amy Rouillard, David J. T. Sumpter · 2026
Read What You Prompt Is What You Get: Increasing Transparency Using Prompt Cards by Amandine M. Caut, Beimnet Zenebe, Amy Rouillard, David J. T. Sumpter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12741.txt.
Introduction
Amandine Caut and co-authors propose prompt cards as a standard way to document prompt engineering work. Prompts can be long and complex, and judging them on subjective tasks is hard. A prompt card records goals, design choices, context steps, evaluation practice, and ethical points. The authors show the idea on wordalisation, where structured numbers are turned into text. Cards, they argue, aid reproducibility and transparency. They can also serve as an alternative to benchmarks for judging text quality.
Catalog record
| CREATOR | Amandine M. Caut, Beimnet Zenebe, Amy Rouillard, David J. T. Sumpter |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54841 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What You Prompt Is What You Get: Increasing Transparency Using Prompt Cards Amandine M. Caut, Beimnet Zenebe, Amy Rouillard, David J. T. Sumpter Abstract The rapid advancement and impressive capabilities of large language models (LLMs) have given rise to the field of prompt engineering, the practice of crafting inputs to guide LLMs toward high-quality, task-relevant outputs. A critical challenge facing the field is the lack of standardised prompt documentation and evaluation practices. Prompts can be long, complex and difficult to evaluate on subjective tasks. To address this challenge, we propose the use of prompt cards, structured summaries of prompt engineering practices inspired by the concept of model cards. Through prompt cards, the specific goals, considerations and steps taken during prompt engineering can be systematically documented and assessed. We present the prompt card approach and illustrate it on a specific task called ”wordalisation”, in which structured numerical data is transformed into text. We argue that a well-structured prompt card can enable better reproducibility, transparency, improve prompt methodology and give an effective alternative to benchmarking for judging the quality of generated texts. By systemically capturing underlying model details, prompt intent, contextualisation strategies, evaluation practices and ethical considerations, prompt cards make explicit the often implicit design decisions that shape system behaviour. Documenting these choices is important as prompting increasingly involves complex pipelines with multiple moving parts. Keywords: LLM, NLP, prompt, prompt engineering Introduction Prompt engineering is the art and practice of creating and optimising requests to guide large language models (LLMs) and other forms of generative artificial intelligence (AI) to produce desired outputs [ 67 , 57 ] . Although prompt engineering existed prior to the release of ChatGPT, its launch in November 2022 triggered a significant surge in the number of articles published in this field. A search on ArXiv for articles in the fields of computer science, electrical engineering and systems science using the keywords ”prompt engineering”, filtered by year, showed a growth from 2018 and 2019, when there were just 7 and 6 articles published, correspondingly, through a sharp increase from 83 in 2022, to 476, 1228 and 2086 articles in 2023, 2024 and 2025, respectively. In the pursuit of prompts that improve, modify and tune the output of generative models, several new methods have arisen. Sahoo et al. [ 82 ] classify the applications of prompt engineering into 12 categories with a total of 41 prompt techniques, including ’new task without data’ (zero-shot and few-shot prompt techniques [ 18 ] ) and ’reduce hallucinations’ (Retrieval Augmented Generation [ 59 ] , ReAct [ 111 ] , CoVe [ 30 ] , CoN [ 113 ] and Cok [ 61 ] ). NeuroPrompts [ 81 ] , for example, are used to optimise prompts for text-to-image generation by automatically enhancing a user’s prompt. Moreover, prompt engineering is increasingly recognised as a valuable skill [ 72 ] . One of several major concerns about LLM outputs is bias, as AI systems can inadvertently reproduce or amplify societal, cultural, or demographic prejudices leading to unfair or discriminatory outcomes [ 5 , 54 , 19 , 117 ] . Another critical issue is hallucination, where AI generates information that is factually incorrect, misleading, or entirely fabricated, despite appearing plausible [ 64 , 50 ] . These phenomena can have serious consequences in high-stakes contexts, such as healthcare, legal decision-making, and scientific research, where erroneous or biased outputs may result in harm or the propagation of misinformation [ 89 , 5 ] . Addressing these risks requires rigorous evaluation, transparency, and the development of safeguards to mitigate both bias and hallucination in AI systems. The standard way to evaluate and compare the performance of models is benchmarking [ 65 , 94 , 100 ] . Benchmarks have been proposed to measure labelling, paraphrasing and inference performance of prompting methods [ 99 ] . However, a serious drawback of these benchmark tests is that, as Kaddour et al. [ 53 ] write, ”slight modifications of the benchmark prompt or evaluation protocol can give drastically different results”. Raji et al. [ 77 ] argue that focusing on benchmarks for every task is not always relevant, and not all tasks can or should be measured using standardised benchmarks. Their claim is supported by Bowman and Dahl [ 17 ] , who write ”we know of no simple test that would allow one to determine if a benchmark presents a valid measure of model ability” and Dehghani et al. [ 29 ] who state that ”in some cases, there is simply no standard benchmark or setup”. Machine learning (ML) faces significant challenges regarding reproducibility, defined as the ability to replicate results under the same conditions as the original experiment [ 41 , 43 ] . Achieving reproducibility in AI is difficult and depends on several key factors: • Access to the original dataset: Many datasets, especially in sensitive domains like healthcare, contain private information and are not publicly accessible. • Detailed methodological documentation: This includes all steps of the pipeline, such as data pre-processing and post-processing, which are often omitted or vaguely described. • Exact computational environment: Variations in hardware (e.g., CPU vs. GPU) or software (e.g., different library or framework versions) can lead to inconsistent results [ 8 , 83 ] . • Complete model configuration: Training parameters (e.g., learning rate, batch size) are sometimes hidden, left at default values, or not reported at all. • Thorough result analysis: Final performance metrics and evaluation procedures must be transparently reported to enable meaningful comparisons [ 41 ] . Even when all these conditions are met, inherent randomness in ML experiments, such as stochastic gradient descent or random weight initialisation, can cause variation in results, even under seemingly identical setups [ 8 , 83 ] . As a result, obtaining identical outcomes across different runs or environments remains a significant challenge. Similarly, the same trained generative model can generate different outputs for identical inputs because sampling-based decoding introduces inherent randomness. To tackle the ongoing challenges of reproducibility in AI, several initiatives have emerged, such as the ML Reproducibility Challenge from Princeton University [ 70 ] and platforms like Hugging Face feature the ”Trending Papers” page [ 33 ] , formerly known as Paper with Code, which evaluates the reproducibility of papers accepted at top machine learning conferences. These efforts aim to mitigate reproducibility issues by encouraging the validation and replication of published results. However, reproducing models remains challenging due to factors such as the use of proprietary or sensitive datasets, limited computational resources [ 83 ] , and incomplete or non-functional code accompanying publications. Recent work demonstrates that the behaviour of LLMs is highly sensitive to prompt phrasing, ordering, and contextual structure [ 74 , 63 , 115 , 104 ] . Unlike traditional classification and regression machine learning systems, where model behaviour is determined primarily by trained parameters, LLM outputs depend strongly on in-context examples, natural language instructions and retrieval-augmented information [ 18 , 2 , 80 ] . This sensitivity introduces reproducibility challenges: identical models can produce substantially different outputs when given slightly different prompts or contextual sequences [ 62 , 13 ] . As prompting, retrieval, and agentic orchestration now function as integral components of system