Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index

Tyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami · 2024

Read Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index by Tyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01690.txt.

Description

Tyler McDonald, Anthony Colosimo, Yifeng Li, and Ali Emami ask whether fancier prompts are worth their price, in a 2024 paper. Prompt engineering keeps producing elaborate techniques, but evaluations usually score accuracy alone. The authors build a metric that blends accuracy with token cost, weighted by how much the user cares about cost.

They test six advanced prompting techniques on ten language models across four datasets. Heavy methods like self-consistency often add cost while giving gains too small to matter. On a top model, simple chain-of-thought scored better on their index than self-consistency once even slight cost concern was applied. The authors argue the field should rethink complex prompting where resources are tight. Their index gives end users a direct way to balance quality against the bill.

Catalog record

CREATORTyler McDonald, Anthony Colosimo , Yifeng Li, and Ali Emami
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01690

Browse the full library · About the Tower of Babel