Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
Amey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1 · 2025
Read Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks by Amey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12845.txt.
Description
Amey Hengle and co-authors build a tougher test in 2025 for language models that read very long texts in many tongues. Most tests just ask a model to find one fact buried in filler. Finding is not the same as thinking, and such tests can leak or be gamed.
Their test, MLRBench, asks for multi-step thought, sums across facts, and calls on missing facts. It spans seven tongues, keeps its parts in step across them, and can grow to any length. In trials, one open model did well in rich-resource tongues and far worse in poor-resource ones. It also used less than a third of its claimed read span. Search aids helped some but did not close the gap. The team shares the test for all to use.
Catalog record
| CREATOR | Amey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12845