Un-considering Contextual Information: Assessing LLMs’ Understanding of Indexical Elements

Metehan Oğuz, Yavuz Bakman, Duygu Nur Yaldiz · 2025

Read Un-considering Contextual Information: Assessing LLMs’ Understanding of Indexical Elements by Metehan Oğuz, Yavuz Bakman, Duygu Nur Yaldiz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01089.txt.

Introduction

Oğuz, Bakman, and Yaldiz probe how language models handle words whose meaning is pure context: I, you, here, tomorrow. Such indexicals flip referent with every speaker and moment, unlike names. The team built the first English test of its kind, 1,600 multiple-choice questions, and ran four leading models through it.

Models ace 'I' yet stumble on 'you', 'here', and 'tomorrow'. Quotation marks, a strong human cue for shifted perspective, help with some words and hurt with others, suggesting the models lean on surface patterns rather than a true sense of viewpoint. The dataset and code are public, giving linguists and engineers a yardstick for a skill conversation depends on.

Catalog record

CREATORMetehan Oğuz, Yavuz Bakman, Duygu Nur Yaldiz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01089

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Un-considering Contextual Information: Assessing LLMs’ Understanding of Indexical Elements Metehan Oğuz, Yavuz Bakman, Duygu Nur Yaldiz Abstract Large Language Models (LLMs) have demonstrated impressive performances in tasks related to coreference resolution. However, previous studies mostly assessed LLM performance on coreference resolution with nouns and third person pronouns. This study evaluates LLM performance on coreference resolution with indexicals like I, you, here and tomorrow , which come with unique challenges due to their linguistic properties. We present the first study examining how LLMs interpret indexicals in English, releasing the English Indexical Dataset with 1600 multiple-choice questions. We evaluate pioneering LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek V3. Our results reveal that LLMs exhibit an impressive performance with some indexicals ( I ), while struggling with others ( you, here, tomorrow ), and that syntactic cues (e.g. quotation) contribute to LLM performance with some indexicals, while they reduce performance with others. Code and data are available at: https://github.com/metehanoguzz/LLMs-Indexicals-English 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in zero-shot and few-shot learning, excelling across a wide array of tasks such as machine translation, text summarization, and question answering OpenAI 2024 ; Ye et al. 2023 ; Bakman et al. 2024 ; Yaldiz et al. 2024 . Their versatility has led to widespread applications in diverse domains, including education, law, and medicine. Figure 1: An example for LLM misinterpreting indexical element ‘here’, uttered by a speaker in Los Angeles. As the use of LLMs continues to expand, understanding their underlying behaviors has become increasingly important. Recent studies have evaluated the performance of large language models on linguistic tasks such as coreference resolution Gan et al. 2024 ; Le and Ritter 2023 ; Brown et al. 2020 ; Yang et al. 2022 ; Agrawal et al. 2022 . Previous work on coreference resolution mostly focused on how coreference is established between two third person entities such as proper names (e.g. Andy, the mechanic ) and third person pronouns (e.g. he, him, himself ) in English and other languages ( Yang et al. 2022 ; Yang 2025 , e.g.) . In this study, we investigate how LLMs establish coreference with indexical elements (e.g. I, you, here ), which differ from third person nouns/pronouns in substantial ways and bring unique challenges for LLMs (see Figure 1 as an example). We investigate how state-of-the-art LLMs interpret the indexical elements I, you, here and tomorrow in English sentences, and whether context or grammatical constraints influence their decisions. To the best of our knowledge, this is the first study examining LLMs’ handling of indexical elements in English. Our key contributions are as follows: • We introduce the English Indexical Dataset , comprising 400 interpretation samples for each indexical element, I, you, here, tomorrow , totaling 1,600 instances. • We evaluate the performance of four frontier LLMs, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek-V3, on the interpretation of indexical elements in English. • We show that LLM performances are not uniform across different types of indexical elements: indexical I is successfully interpreted by most LLMs, other indexicals like you , here and tomorrow lead to poor performances. • We show that quotation affects LLMs performances differently: quotation reduces LLMs’ accuracies with tomorrow , but it increases accuracies with here . 2 Indexical elements Indexical pronominals like I, you, here and now are used to refer to referents of the speech-act coordinates ( Kaplan 1977 ; Schlenker 2003 , e.g.) . For instance, I refers to author (speaker) of the utterance, while here refers to the location where the utterance was made. Thus, a sentence like 2 means different things if uttered by different people and/or in different places. If John utters 2 in Los Angeles , it means that John was born in Los Angeles, but if Mary utters the same sentence in New York it means that Mary was born in New York. \ex . ˙I was born here . .̱ Andrew said that I went to Buckhead. Indexicals are interpreted inside the context of utterance, referring to the actual speech-act coordinates like the author or the location of the actual utterance. As a result, if 2 is uttered by John , the indexical I can only be interpreted as referring to John as the speaker, leading to a reading like ‘Andrew said that John went to Buckhead’. Crucially, even though Andrew’s speech/claim is reported in 2 , the indexical I cannot refer to Andrew. Direct quotation is an exception to this generalization, where reported material is interpreted as verbatim utterance/thoughts of its owner. Thus, when indexicals appear inside direct quotation, they are interpreted inside the reported context, rather than the actual context of utterance. In other words, direct quotation ‘shifts’ the interpretations of indexicals into the reported context. For example, I and here in 2 appear inside direct quotation, where Andrew’s speech is reported. \ex . While we were in Atlanta, Andrew said “ I was born here .” Regardless of who utters 2 , the sentence means that Andrew was born in Atlanta , so both I and here are ‘shifted’ into the reported context, where Andrew is the speaker and Atlanta is the location 1 1 1 Some languages (not English) allow indexical elements to ‘shift’ without quotation. See Deal 2020 for an overview. Indexicals differ from other pronominals in substantial ways. First, though syntactic and semantic factors can affect how a pronoun is interpreted (e.g. subject bias), pronouns are typically ambiguous regarding what/who they refer to. For example, the third person pronoun he in 2 is most naturally interpreted as referring to the subject John (for syntactic or contextual reasons), but the object Bill is still a possible antecedent, causing ambiguity between two different readings ( Crawley et al. 1990 ; Stewart and Pickering 1998 ; Pickering and Majid 2007 , e.g.) . In addition, he can refer to any contextually salient person that is not mentioned in the sentence (e.g. Peter ), which makes pronouns even more ambiguous and context-dependent. \ex . John hit Bill and he ran away. As a result, semantic/contextual information plays a crucial role in how pronouns are interpreted, and speakers use those cues to establish coreference with pronouns. For instance, if 2 is uttered in a context where John is a supportive and humble coworker, the most natural interpretation is that John suggests that Bill should get promoted ( he = Bill). However, if John is arrogant and jealous, the most natural interpretation is that John suggests that John should get promoted ( he = John). \ex . John told Bill that he should get promoted. Indexicals, on the other hand, unambiguously refer to the referents of the speech-act coordinates. For example, I in 2 refers to the speaker regardless of what we know about John or Bill. I refers to the speaker even if John and/or Bill are arrogant and jealous, so contextual information like this should be disregarded while interpreteting indexicals. \ex . John told Bill that I should get a promotion. In summary, indexicals are restricted by different syntactic factors than pronouns (e.g. quotation vs non-quotation) and are typically unambiguous, while pronouns are free to refer to a wide

Read the complete text