LangProBe: a Language Programs Benchmark

Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J. Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia · 2025

Read LangProBe: a Language Programs Benchmark by Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J. Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20315.txt.

Description

Shangyin Tan and co-authors present a large benchmark for programs built by chaining language model calls, in work from 2025. Such programs, with prompts tuned part by part, are now a standard way to build AI systems. Yet the trade-offs among their designs had barely been studied.

The benchmark, LangProBe, covers over two thousand mixes of tasks, designs, tuners, and models. It is the first to weigh how design and tuning choices, alone and combined, trade quality against cost. Tuned programs beat raw model calls on that trade-off by a clear margin. Still, which mixes to chase called for human judgment or trial results. The team will share the code and data in the open.

Catalog record

CREATORShangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J. Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20315

Browse the full library · About the Tower of Babel