Lost in the Middle, and In-Between: Enhancing Language Models’ Ability to Reason Over Long Contexts in Multi-Hop QA
2024
Read Lost in the Middle, and In-Between: Enhancing Language Models’ Ability to Reason Over Long Contexts in Multi-Hop QA (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10079.txt.
Description
George Arthur Baker, Ankush Raut, Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense study a known flaw of long-context language models, in work published in 2024. Models tend to use facts near the start or end of a long input and miss facts buried in the middle. The team asks what happens when a question needs several facts spread across many documents.
In this multi-hop setting, scores fell not only when a key fact sat far from the edges, but also when the needed facts sat far apart from each other. The team then tried fixes: trimming documents to short fact triples from a knowledge graph, summarizing them, and asking the model to reason step by step. The work maps where long inputs still trip models up and which trims help most.
Catalog record
| YEAR | 2024 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10079