Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading
Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier · 2026
Read Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading by Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27637.txt.
Description
Sadjoli and four colleagues investigate whether prompt optimization changes large language model rankings. Current frameworks use one static prompt template for every model, unlike industry practice, which optimizes the prompt per model for best results. On public academic plus internal industry test sets, optimization greatly affects final model rankings. The authors urge practitioners to optimize prompts per model when choosing the best model for a task.
Catalog record
| CREATOR | Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75319 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27637