Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index
Tyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami · 2024
Read Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index by Tyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01690.txt.
Description
Tyler McDonald, Anthony Colosimo, Yifeng Li, and Ali Emami ask whether fancier prompts are worth their price, in a 2024 paper. Prompt engineering keeps producing elaborate techniques, but evaluations usually score accuracy alone. The authors build a metric that blends accuracy with token cost, weighted by how much the user cares about cost.
They test six advanced prompting techniques on ten language models across four datasets. Heavy methods like self-consistency often add cost while giving gains too small to matter. On a top model, simple chain-of-thought scored better on their index than self-consistency once even slight cost concern was applied. The authors argue the field should rethink complex prompting where resources are tight. Their index gives end users a direct way to balance quality against the bill.
Catalog record
| CREATOR | Tyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01690