Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning

Victor Sebastian Martinez Pozos, Ivan Vladimir Meza Ruiz · 2025

Read Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning by Victor Sebastian Martinez Pozos, Ivan Vladimir Meza Ruiz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04900.txt.

Introduction

Victor Sebastian Martinez Pozos and Ivan Vladimir Meza Ruiz teach computers to turn pictures into symbol strings. Their 2025 paper borrows a trick from language: words compress a scene into parts that can be reasoned about. The team extends a self-taught vision framework so it learns from images and symbols together.

A decoder built on attention writes the symbol string for each image. Its attention maps show which image region each symbol came from, so the output can be inspected rather than taken on faith. Early tests suggest the strings capture a real level of abstraction, though polish is still needed. The authors aim at high-level scene understanding in robots and assistants.

Catalog record

CREATORVictor Sebastian Martinez Pozos, Ivan Vladimir Meza Ruiz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning Victor Sebastian Martinez Pozos, Ivan Vladimir Meza Ruiz Abstract In this paper, we explore the potential of abstracting complex visual information into discrete, structured symbolic sequences using self-supervised learning (SSL). Inspired by how language abstracts and organizes information to enable better reasoning and generalization, we propose a novel approach for generating symbolic representations from visual data. To learn these sequences, we extend the DINO framework to handle both visual and symbolic information. Initial experiments suggest that the generated symbolic sequences capture a meaningful level of abstraction, though further refinement is required. An advantage of our method is its interpretability: the sequences are produced by a decoder transformer using cross-attention, allowing attention maps to be linked to specific symbols and offering insight into how these representations correspond to image regions. This approach lays the foundation for creating interpretable symbolic representations with potential applications in high-level scene understanding. 1 Introduction In recent years ( Bengio et al. 2013 ; Krizhevsky et al. 2017 ) , advances in computer vision and machine learning have significantly improved our ability to learn from complex visual data. However, most learned representations remain continuous and unstructured ( LeCun et al. 2015 ) , making it difficult to reason about high-level abstractions and relationships in the data. Inspired by language structure, which allows us to abstract and generalize from perceptual input, we explore whether it is possible to generate discrete, structured symbolic representations from visual data through self-supervised learning (SSL). Language provides a compelling framework for abstraction ( Bisk et al. 2020 ) : it captures meaning through compositional symbols that represent and generalize complex information, enabling higher-level reasoning. Translating this capacity to machine learning could be a key step toward more interpretable and generalizable models ( Lake et al. 2016 ) . However, current approaches to visual representation learning primarily focus on learning dense, continuous features, which lack the compositional properties needed for symbolic reasoning. This gap motivates our investigation into generating symbolic sequences from visual input, where structured symbols can encapsulate the variations and complexities of visual data in a compact, interpretable form. In this work, we introduce a novel approach to generating symbolic representations from visual data using an extended version of the DINO framework ( Caron et al. 2021 ; Oquab et al. 2024 ) , which is designed to handle both visual and symbolic information. Our method leverages pre-trained visual representations from a Vision Transformer (ViT) ( Dosovitskiy et al. 2021 ) and extends them to produce symbolic sequences that can abstract the compositional properties of visual scenes. By utilizing a decoder transformer with cross-attention mechanisms, we ensure that the generated symbols can be interpreted and linked to specific regions in the input data, providing a more interpretable model. Our main contributions are as follows: (1). We propose a novel method for generating structured symbolic sequences from visual data through self-supervised learning, inspired by linguistic abstraction. (2). We extend the DINO framework to handle both visual and symbolic information, enabling the learning of compositional symbolic representations. (3). We demonstrate the interpretability of our method by linking the generated symbols to visual regions through attention maps, offering insights into how these symbols correspond to image features. (4). Initial experiments show that our approach captures a meaningful level of abstraction, though further refinement is needed. This suggests potential for high-level scene understanding and interpretability. By bridging the gap between continuous visual representations and discrete symbolic reasoning, our approach opens the door to more interpretable models. It lays the groundwork for further exploration in abstract visual understanding. Figure 1: Visualization of four sample images alongside their generated sequences and corresponding attention masks, produced by our model. The sequences are generated using a temperature-softmax discretization process with a temperature of 0.12 during training. Attention masks, associated with each sequence element, are extracted from the cross-attention layers in the deepest layer of the descriptor module. From left to right: The first column shows the input sample images, followed by the generated sequences and their corresponding attention masks. 2 Related Work Visual Representation Learning and Discrete Latent Representations Self-supervised learning (SSL) ( Chen et al. 2020 ; Radford et al. 2021 ) has led to significant progress in visual representation learning, with Vision Transformers (ViT) proving particularly effective at extracting meaningful features by attending to different regions of an image. Methods like DINO build on ViT to capture dense, continuous representations from visual data. However, while these representations are powerful, they often lack the structure necessary for symbolic reasoning and high-level abstractions. A growing body of work addresses this limitation by introducing discrete latent representations ( van den Oord et al. 2018 ; Yu et al. 2022 ) , which transform continuous visual features into discrete codes or tokens. These discrete representations offer a more structured and interpretable way to model complex visual inputs. However, while they provide compact encodings, they typically do not impose specific structural properties or constraints on the learned codes, leaving the challenge of generating meaningful, compositional symbolic sequences open. Our work builds on this by adopting discrete representations and focusing on learning structured symbolic sequences that capture the underlying compositional nature of visual data. Image Captioning and Encoder-Decoder Models Traditional image captioning models typically follow an encoder-decoder architecture ( Vinyals et al. 2015 ; Xu et al. 2016 ) , where the encoder (often a CNN or Vision Transformer) processes the image into a dense, continuous representation, and the decoder (such as an RNN or Transformer) ( He et al. 2015 ; Vaswani et al. 2023 ) generates descriptive language sequences based on these features. These models rely on explicit supervision, using labelled datasets with human-annotated captions to guide the mapping from images to text. However, our approach differs fundamentally. While we also employ a standard encoder-decoder architecture with a pretrained Transformer to capture visual features, our method does not rely on human-provided labels, predefined language targets, or prior training on linguistic models. Instead, we autonomously generate symbolic sequences directly from visual data in a self-supervised manner, discovering structured outputs without the guidance of any external symbolic or linguistic framework. This makes our task more challenging and allows for the emergence of abstract and compositional visual representations in a fully unsupervised setting. Neuro-Symbolic Learning and Symbolic Reasoning Neuro-symbolic approaches ( Susskind et al. 2021 ) aim to combine the pattern recognition capabilities of neural networks with the logical, interpretable reasoning strengths of symbolic AI. These models use neural networks to process raw data (such as images or text) and output symbolic representations or logical rules for high-level reasoning tasks. Recent advances have introduced methods for generating symbolic sequences directly from visual data, allowing models to infer perceptual features and symbolic abstractions in a unified framework. Our work aligns with this direction by generating symbolic sequences from visual input, directly integrating perception with structured symbolic reasoning without requiring external symbolic systems. Interpretability through Attention Mechanisms and Discrete Representations Attention mechanisms, particularly in Transformer architectures, have proven essential for improving the interpretability of models by highlighting which parts of the input data are most relevant for a given prediction. In visual models, attention maps make it possible

Read the complete text