Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks

Amey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1 · 2025

Read Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks by Amey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12845.txt.

Description

Amey Hengle and co-authors build a tougher test in 2025 for language models that read very long texts in many tongues. Most tests just ask a model to find one fact buried in filler. Finding is not the same as thinking, and such tests can leak or be gamed.

Their test, MLRBench, asks for multi-step thought, sums across facts, and calls on missing facts. It spans seven tongues, keeps its parts in step across them, and can grow to any length. In trials, one open model did well in rich-resource tongues and far worse in poor-resource ones. It also used less than a third of its claimed read span. Search aids helped some but did not close the gap. The team shares the test for all to use.

Catalog record

CREATORAmey Hengle 1, Prasoon Bajpai 1, Soham Dan 2, Tanmoy Chakraborty 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12845

Browse the full library · About the Tower of Babel