Visual symbolic mechanisms: Emergent symbol processing in vision language models

Rim Assouel*, Declan Campbell*, Taylor Webb, * Equal contribution · 2025

Read Visual symbolic mechanisms: Emergent symbol processing in vision language models by Rim Assouel*, Declan Campbell*, Taylor Webb, * Equal contribution (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15871.txt.

Introduction

Rim Assouel, Declan Campbell, and co-authors ask how vision and language models keep track of which traits belong to which object. Their 2025 paper uses a simple test case: a red square with a blue circle looks much like a blue square with a red circle. To tell the two apart, a model must bind each color to the right shape.

Past work found that text only models solve this with tag like codes that do not depend on content. The team finds that vision models use a like scheme, but one based on place in the image. They trace how these tags form inside the model as it works. The findings help explain why vision models still fail at some binding tasks, and where in the model that skill sits.

Catalog record

CREATORRim Assouel*, Declan Campbell*, Taylor Webb, * Equal contribution
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual symbolic mechanisms: Emergent symbol processing in vision language models Rim Assouel*, Declan Campbell*, Taylor Webb, * Equal contribution Abstract To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this ‘binding problem’ via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by vision language models (VLMs). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a set of emergent symbolic mechanisms that support binding in VLMs via a content-independent, spatial indexing scheme. Moreover, we find that binding errors can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for addressing the persistent binding failures exhibited by these models. 1 Introduction The visual world is composed of many commonly recurring elements — shapes, colors, textures, etc. — and visual inputs can be efficiently represented by making use of this compositional structure, for instance, by representing a ‘red square’ as a combination of features for ‘red’ and ‘square’. There is growing evidence that neural networks, including language models and vision language models (VLMs), learn such compositional representations and use them to represent novel combinations of familiar features ( Lepori et al. 2023 ; Lewis et al. 2022 ; Campbell et al. 2024 ) . This compositional coding approach, however, introduces a fundamental representational challenge, sometimes referred to as the ‘binding problem’ ( Treisman and Gelade 1980 ; Greff et al. 2020 ; Frankland et al. 2021 ) . Given a set of compositional features, such as features representing basic colors and shapes, the binding problem refers to the question of how these features are bound together to represent the entities and relational configurations in a specific context. For instance, in order to represent an image that contains a red square and a blue circle, and to distinguish it from other potential combinations of the same features (e.g., an image containing a blue square and a red circle), it is necessary to form in-context associations between the features that correspond to the same entities (i.e., to bind ‘red’ to ‘square’, and ‘blue’ to ‘circle’). Recent work has begun to uncover the representations and mechanisms that support this capacity in language models. Of particular relevance is the discovery of emergent binding IDs , vectors representing content-independent indices — akin to symbolic variables — that language models use to track the assignment of entities and attributes in-context ( Feng and Steinhardt 2023 ; Feng et al. 2024 ) . Binding IDs are additively incorporated into the token embeddings corresponding to their arguments, and can be intervened upon to produce systematic binding errors. Other recent work has identified a set of emergent symbolic mechanisms that language models use to perform abstract reasoning ( Yang et al. 2025 ) . These mechanisms convert input tokens to abstract, symbol-like variables, which can then be processed independently of the tokens to which they’re bound, before eventually being converted back to their corresponding tokens to perform inference. These and other recent findings suggest that language models rely on an emergent form of symbol processing to represent and reason about entities and relations in-context, but it is an open question whether VLMs employ similar mechanisms to bind visual entities in-context. It is particularly notable that many of the most puzzling shortcomings of VLMs appear to be directly related to the binding problem. VLMs perform very poorly on many tasks that are easy for humans, such as counting or visual search ( Rahmanzadehgervi et al. 2024 ) , and many of these tasks seem to involve the need to accurately bind features in-context. Indeed, careful behavioral evaluations have found that VLMs display specific psychophysical signatures similar to those observed in human vision for conditions that interfere with binding ( Campbell et al. 2024 ) . These findings underscore the importance of understanding the mechanisms that support binding in VLMs, and especially how the failure of these mechanisms might explain the binding failures that limit the performance of these models. In this work, we present evidence for a set of emergent symbolic mechanisms that support binding in VLMs. Similar to the content-independent binding IDs that have been identified in text-only language models, we find that VLMs use visual space as a content-independent scaffold to bind features and parse multi-object scenes. We therefore refer to these indices as position IDs. We also identify a pair of visual symbolic mechanisms that VLMs use to manipulate these indices: 1) Position ID Heads compute the position ID associated with a target object; and 2) Feature Retrieval Heads use these position IDs to retrieve the features associated with that object. We present convergent evidence for these mechanisms across a set of representational, causal mediation, and intervention analyses. Finally, we analyze the role that these mechanisms play in the persistent binding failures exhibited by VLMs. We find that binding errors can be directly tied to failures of the identified mechanisms, and that the outputs of these mechanisms can be intervened upon to rescue performance in conditions that typically produce binding errors. Taken together, these results begin to uncover the mechanisms that support symbol-like binding and inference in VLMs, and illustrate how the breakdown of these mechanisms contributes to their persistent binding failures, suggesting avenues for further improvement of these models. 2 Results To investigate the mechanisms and representations involved in multi-object visual processing, we focus on a simple scene description task. In this task, an image with multiple objects is presented. Importantly, each object spans multiple image patches, so that it’s necessary to bind patch-level feature representations to form a unified object-level percept. The image is accompanied by an incomplete caption describing some of the objects in the image, and the task is to complete this caption by describing the remaining objects. This seemingly simple task can be quite challenging for VLMs, particularly for images involving larger numbers of objects, and images with significant feature overlap between objects ( Campbell et al. 2024 ) . Performance on this task has also been closely tied to the binding problem, so it offers an excellent testbed to investigate the mechanisms involved in in-context feature binding, and to study the role that these mechanisms play in binding errors. All experiments were performed on the Qwen2-VL-7B-Instruct model ( Wang et al. 2024 ) . 2.1 Representational Analyses We first sought to understand the representations that are involved in describing a target object, given an incomplete caption describing the other objects in an image. We presented the model with an image containing a 2\times 3 array of objects, and a caption describing 5 out of 6 of these objects. We analyzed the representational structure of the residual stream at the position of the final token in the caption (the position at which the model is expected to generate a description of the target object). We identified

Read the complete text