DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking

Lanni Bu, Lauren Levine, Amir Zeldes · 2025

Read DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking by Lanni Bu, Lauren Levine, Amir Zeldes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17013.txt.

Introduction

Lanni Bu, Lauren Levine, and Amir Zeldes build a harder reading test for language models in a 2025 paper. Most benchmarks ask about facts stated plainly in single sentences. Real understanding means tracking meaning across whole documents. Pronouns, salience, and unstated links all live between sentences.

Their benchmark, DiscoTrack, covers twelve languages. Tasks sit at four levels of discourse understanding. They include recognizing what stays salient and tracking mentioned entities. They also include discourse relations and bridging inference, guessing unstated links. Even the strongest current models found these tasks hard. The results expose a gap between sentence-level scores and true reading. DiscoTrack gives researchers a multilingual yardstick for that deeper skill.

Catalog record

CREATORLanni Bu, Lauren Levine, Amir Zeldes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking Lanni Bu, Lauren Levine, Amir Zeldes Abstract Recent LLM benchmarks have tested models on a range of phenomena, but are still focused primarily on natural language understanding for extraction of explicit information, such as QA or summarization, with responses often targeting information from individual sentences. We are still lacking more challenging, and importantly also multilingual, benchmarks focusing on implicit information and pragmatic inferences across larger documents in the context of discourse tracking: integrating and aggregating information across sentences, paragraphs and multiple speaker utterances. To this end, we present DiscoTrack, an LLM benchmark targeting a range of tasks across 12 languages and four levels of discourse understanding: salience recognition, entity tracking, discourse relations and bridging inference. Our evaluation shows that these tasks remain challenging, even for state-of-the-art models. 1 Introduction A growing number of benchmarks, such as GLUE ( Wang et al. 2018 ) and CLUE ( Xu et al. 2020 ) , have been developed to evaluate LLMs across a wide range of phenomena, languages, and task types. However, the majority of existing benchmarks focus on extracting explicit information, often from individual sentences, and offer limited coverage of discourse-level phenomena that require reasoning across multiple sentences or larger contexts. Some recent efforts, such as Disco-Bench ( Wang et al. 2023 ) , have begun to address discourse-level evaluation. However, this benchmark is limited to only two languages and still focuses on QA and translation, rather than tracking discourse in terms of entities mentioned or implicit relations between utterances. This gap in coverage motivates our work, DiscoTrack. In this paper, we present a multilingual benchmark designed to evaluate large language models’ capabilities in discourse tracking: the ability to gather information that is not explicitly stated from across long documents to identify relations and group information based on discourse structure. The main contributions of this paper are: • Releasing an openly available, multilingual benchmark, covering 12 typologically diverse languages: English, Chinese, Czech, French, German, Italian, Polish, Portuguese, Russian, Spanish, Thai and Turkish. • Broadening the range of phenomena for LLM benchmarking to include salience, entity tracking, discourse relations, and multiple aspects of bridging anaphora. • Presenting a thorough evaluation of several LMs and corresponding human performance specifically on tasks at the discourse level. • Providing error analysis to expose specific model strengths and weaknesses, with confusion matrices, comparing zero and few-shot settings, detailed separate scores on English and lower-resource lanugages, as well as comparisons to human errors. 2 Related work The most closely related effort to our work is Disco-Bench Wang et al. 2023 , which specifically targets some discourse phenomena. However Disco-Bench is limited to Chinese and English, and targets discourse rather indirectly: for discourse understanding, it only uses three monolingual Chinese tasks: speaker identification, zero anaphora reconstruction (which is much less relevant for languages such as English) and classic reading comprehension questions. For generation it uses a mix of machine translation, text expansion and language modeling (text filling/completion), all of which benefit from discourse understanding, but do not probe discourse processing capabilities directly. While both Disco-Bench and our work focus on evaluating discourse-aware capabilities of LLMs, the broader field of LLM evaluation encompasses a wide range of efforts that test different linguistic phenomena using diverse models, tasks, and prompting strategies. Recent benchmarking efforts reveal two notable trends: a shift from monolingual to multilingual evaluation, and a progression from simple sentence-level tasks toward more complex, cognitively demanding language understanding. This progression in task complexity is reflected in a variety of recent benchmarks. Early efforts such as GLUE and SuperGLUE ( Wang et al. 2019 ; Wang et al. 2023 ) focused on localized classification tasks like natural language inference (NLI) and coreference linking. Later benchmarks like MMLU ( Hendrycks et al. 2020 ) , LM Evaluation Harness Gao et al. 2021 and BIG-Bench ( Kazemi et al. 2025 ) expanded the scope to include general-purpose reasoning, knowledge retrieval, and problem-solving across domains. MT-bench ( Bai et al. 2024 ) and LiveBench ( White et al. 2024 ) have further shifted attention toward open-ended generation, assessing coherence and fluency in dialogue or multi-turn responses. Some efforts, such as AGIEval ( Zhong et al. 2023 ) , explicitly model human-like cognitive testing by evaluating performance on standardized exam questions. While many of these benchmarks and tasks involve reasoning over multiple sentences, they are not well suited to isolating discourse capabilities in the sense of computational discourse processing Stede 2012 , i.e. with a focus on revealing implicit structure, priorities and unexpressed relationships that form the backbone of coherent texts. 3 DiscoTrack Task Group English Czech French German Portuguese Russian Spanish Italian Turkish Chinese Thai Polish BRIDGDET 466 BRIDGTYPE 219 BRIDGANTE 195 SALTOP 207 SALREL 500 RELCLF 500 500 500 500 500 500 500 500 500 500 500 500 RELEXP 500 500 322 500 262 500 500 500 500 RELIMP 500 301 316 296 342 500 326 ENTMAX 234 500 143 221 179 500 22 500 ENTCOUNT 405 500 413 173 373 472 376 500 ENTCOREF 500 500 500 104 500 500 500 500 Table 1: Language Availability and Number of Instances per Task in the DiscoTrack Benchmark. In order to specifically evaluate LLM discourse capabilities, DiscoTrack focuses on tasks which target multi-sentential context, long distance dependencies, and whole document understanding. Our benchmark covers a wide range of languages (see Table 1 ), enabling cross-lingual evaluation. We select tasks in four areas: • Salience recognition - by focusing on the distinction between more and less important entities in a document, we force models to relativize or rank some or all entities mentioned in the discourse simultaneously. For example in Figure 1 , Kim and Mary (in bold) are more salient than Jamie (in gray). • Entity tracking - as discourse develops, entities are referred back to, and models must distinguish between newly introduced concepts and ones that tie back to the preceding discourse. For example, models should notice that there is only one Mary, and she is mentioned 3 times in Figure 1 . • Discourse relations - the relationship between multiple propositions is often left unexpressed, and here we force models to identify, interpret and complete information signaling such relations. In Figure 1 , these include temporal precedence ( first Kim pushed Mary, then she fell) and causality (Mary fell because of Kim pushing). • Bridging inference - we challenge models to explicitly identify implicit relationships between mentioned concepts, including unexpressed part-whole, set-member and other associative links which make the nature of newly introduced entities inferable in context. For example in Figure 1 we infer that the school is the one Kim and Mary regularly go to. Each area corresponds to a module in the benchmark and includes 2–3 sub-tasks, described below. Kim and Mary went to school with Jamie . bridging: entity-associative [Suddenly, Mary fell.] [ Kim pushed her !] cause + precedence Figure 1: Sample discourse illustrating associative bridging anaphora, implicit cause and precedence discourse relations, tracking the entity Mary, and relative salience, where the less salient ‘Jamie’ is shown in gray. 3.1 Salience recognition Salience in language can refer to a range of concepts (see Boswijk and Coler 2020 ), but for

Read the complete text