Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading

Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier · 2026

Read Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading by Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27637.txt.

Description

Sadjoli and four colleagues investigate whether prompt optimization changes large language model rankings. Current frameworks use one static prompt template for every model, unlike industry practice, which optimizes the prompt per model for best results. On public academic plus internal industry test sets, optimization greatly affects final model rankings. The authors urge practitioners to optimize prompts per model when choosing the best model for a task.

Catalog record

CREATORNicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, Daniel Dahlmeier
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75319
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27637

Browse the full library · About the Tower of Babel