Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment Conference: Proceedings of The Web Conference; April 28 – May 2, 2025; Sydney, Australia Proceedings of The Web Conference (WebConf ’25), April 28 – May 2, 2025, Sydney, Australia DOI: XXXXXXX.XXXXXXX Conference

Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi · 2025

Read Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment Conference: Proceedings of The Web Conference; April 28 – May 2, 2025; Sydney, Australia Proceedings of The Web Conference (WebConf ’25), April 28 – May 2, 2025, Sydney, Australia DOI: XXXXXXX.XXXXXXX Conference by Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14296.txt.

Description

Search engines are graded by relevance labels, but getting trustworthy labels at scale is slow and expensive. Julian Schnabel, Johanne Trippas, Falk Scholer, and Danula Hettiachchi propose in 2025 to split the labeling job into a pipeline of stages. Each stage gets its own prompt and a model sized to the subtask, rather than asking one big model to judge cold.

On the TREC Deep Learning benchmark, one pipeline beat GPT-4o mini's agreement with human labels by 18.4 percent while costing about twenty cents per million input tokens. Even the flagship GPT-4o improved by 9.7 percent when wrapped in the staged approach. The authors present modular pipelines as a cheaper, scalable route to relevance assessment — a reminder that how models are chained can matter as much as how big they are.

Catalog record

CREATORJulian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14296

Browse the full library · About the Tower of Babel