Context informs pragmatic interpretation in vision–language models

Alvin W.M. Tan*, Ben Prystawski*, Veronica Boyce, Michael C. Frank · 2025

Read Context informs pragmatic interpretation in vision–language models by Alvin W.M. Tan*, Ben Prystawski*, Veronica Boyce, Michael C. Frank (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03908.txt.

Introduction

Alvin Wei Ming Tan, Ben Prystawski, Veronica Boyce, and Michael C Frank test how context shapes pragmatic interpretation in vision-language models. They use iterated reference games, where players repeatedly pick out new referents with language. Amount, order, and relevance of context were varied across trials for both humans and models.

Without relevant context, models performed above chance but well below humans. With relevant context supplied, model performance rose sharply across trials. Few-shot games with abstract referents stayed hard for models overall. The findings show that the right conversational history can unlock much stronger context-sensitive reasoning, while also marking a remaining gap with human pragmatic skill.

Catalog record

CREATORAlvin W.M. Tan*, Ben Prystawski*, Veronica Boyce, Michael C. Frank
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context informs pragmatic interpretation in vision–language models Alvin W.M. Tan*, Ben Prystawski*, Veronica Boyce, Michael C. Frank Abstract Iterated reference games—in which players repeatedly pick out novel referents using language—present a test case for agents’ ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision–language models on trials from iterated reference games, varying the given context in terms of amount, order, and relevance. Without relevant context, models were above chance but substantially worse than humans. However, with relevant context, model performance increased dramatically over trials. Few-shot reference games with abstract referents remain a difficult task for machine learning models. \workshoptitle First Workshop on Interpreting Cognition in Deep Learning Models (CogInterp) 1 Introduction Recent advances in machine learning have produced multi-turn conversational agents, which are a public face of artificial intelligence [ 23 ] . The utility of conversational agents depends on capacities including natural language understanding, world knowledge, and instruction following [ 12 ] . Furthermore, success in multi-turn conversation requires the underlying language model to respond to the user’s message appropriately given the preceding context, which in turn requires the language model to retain relevant contextual information and use it to interpret new messages. Such multi-turn interactions are a core feature of human communication: shared conversational history supports a shared system of semantic meaning [ 9 , 11 ] . Iterated reference games are a common experimental paradigm for studying this type of communication. In these paradigms, a describer produces a description of a referent such that a matcher can correctly select the referent from a set of options. Games repeat over multiple rounds, typically resulting in the emergence of conventionalised referring expressions over repeated descriptions of the same referent [ 8 ] . These experiments demonstrate that humans dynamically adapt to their conversational partners, creating ad hoc context- and/or partner-specific meanings and conventions [ 8 , 14 , 3 ] . Conversations—whether human–human or human–AI—require their participants to use contextual information to understand the meanings of utterances. A system with the ability to do this would show two behavioural signatures: (1) the ability to interpret linguistic meaning pragmatically and (2) sensitivity to contextual information [ 15 , 4 , 19 , 13 , 16 , 7 , 25 ] . In this work, we employ iterated reference games as a minimal test case to investigate whether state-of-the-art open vision–language models (VLMs) reason pragmatically about referring expressions given varying types of prior context, and how their sensitivity to context compares to that of humans. Iterated reference games have previously been used to study AI systems for both generation and comprehension of descriptions, but this domain continues to prove challenging for AI systems, especially for generation with abstract images [ 17 , 13 , 27 ] . Figure 1 : A. Overview of the experimental structure for the original interactive games in Boyce et al. [ 3 ] . B. User interface for each trial in the experiments with naïve participants. 2 Methods All model data and code are available on Github at https://github.com/benpry/vlm-tg-context/tree/coginterp . 2.1 Dataset We used an iterated reference game dataset from Boyce et al. [ 3 ] . In each game, players saw a grid of 12 tangram images (Figure 1 A). On each trial, one player (the describer), saw one image highlighted. The describer described a target image to the other players via chatbox so that the matchers could each select the target image from the 12 options. Each game had 2–6 players, who played 6 rounds of 12 trials each (one trial for each image). Here we use ten of these games, which were subsequently used by Boyce et al. [ 4 ] to test for context sensitivity in naïve human matchers. Naïve matchers read transcripts of the conversations between the players in the original games and then made a selection of what image was being describer (Figure 1 B). Naïve matchers received feedback on whether they were correct, but were not told the correct answer if their response was incorrect. Each participant saw all trials from one game, in an order that varied based on condition. Boyce et al. [ 4 ] collected participants in the yoked condition ( N = 99) and the shuffled condition ( N = 97). We collected additional human data on Prolific in the backward ( N = 89) and random ( N = 107) conditions (described in Section 2.3 ) using the same procedure. 2.2 Experiment setup Our experiments with VLMs aimed to match the naïve human setup. We evaluated the instruction-tuned versions of four leading open-weights VLMs of different sizes: Qwen 2.5 VL 32B [ 2 ] , Gemma 3 27B [ 10 ] , Llama 3.2 11B [ 22 ] , and Kimi VL A3B [ 20 ] . Models were given a general system prompt describing the task setup, followed by the set of 12 tangram image options labeled A through L presented as a single image. The exact prompt and image are included in Appendix A . Models were then given the context of all preceding trials formatted as a chat history, with the user supplying the text written by participants and the model serving as the assistant providing the target tangram, denoted A through L; note that models were given the correct target (unlike humans). Finally, the description for the test trial was provided, and the model log probabilities for the tokens A through L were obtained. We renormalised the log probabilities using a softmax, and treated the probability assigned to the correct target as the model’s accuracy. 2.3 Context conditions Do VLMs make use of context to the same extent, and in the same way, as humans? We considered eight conditions varying the amount and type of in-context trials seen by the model (Table 1 ). Table 1: Conditions of in-context trials seen by models. Columns indicate whether the in-context trials are from the same original game as the test trial, whether the in-context trials are all from the same original game, what the order of the in-context trials is, and whether the tangram in the test trial occurs in the in-context trials. Condition N_{\textup{trials}} Original game Same game Trial order Same tangram seen Yoked 720 Y Y Original Y Shuffled 7056 Y Y Permuted Y Backward 720 Y Y Reversed Y Ablated 720 Y Y Original N Other-within 6480 N Y Original Y Other-across 6480 N N Original Y Random 7704 N N Permuted Y No context 720 N N None N The conditions varied in their fidelity to the context experienced by actual human participants in the original iterated reference games. Four conditions contained trials drawn from a single original game. The yoked , shuffled , and backward conditions presented all the trials in the original game, in the original order, a permuted order, or the reversed order respectively. The ablated condition presented the trials in the original order, but in-context trials containing the same target tangram as the test trial were removed. Two other conditions involved in-context trials sampled from different original games than the test trial. The other-within condition drew all in-context trials from a single different original game than the test

Read the complete text