Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment Conference: Proceedings of The Web Conference; April 28 – May 2, 2025; Sydney, Australia Proceedings of The Web Conference (WebConf ’25), April 28 – May 2, 2025, Sydney, Australia DOI: XXXXXXX.XXXXXXX Conference
Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi · 2025
Read Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment Conference: Proceedings of The Web Conference; April 28 – May 2, 2025; Sydney, Australia Proceedings of The Web Conference (WebConf ’25), April 28 – May 2, 2025, Sydney, Australia DOI: XXXXXXX.XXXXXXX Conference by Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14296.txt.
Description
Search engines are graded by relevance labels, but getting trustworthy labels at scale is slow and expensive. Julian Schnabel, Johanne Trippas, Falk Scholer, and Danula Hettiachchi propose in 2025 to split the labeling job into a pipeline of stages. Each stage gets its own prompt and a model sized to the subtask, rather than asking one big model to judge cold.
On the TREC Deep Learning benchmark, one pipeline beat GPT-4o mini's agreement with human labels by 18.4 percent while costing about twenty cents per million input tokens. Even the flagship GPT-4o improved by 9.7 percent when wrapped in the staged approach. The authors present modular pipelines as a cheaper, scalable route to relevance assessment — a reminder that how models are chained can matter as much as how big they are.
Catalog record
| CREATOR | Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, Danula Hettiachchi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14296