DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes
Sergey Linok, Vadim Semenov, Anastasia Trunova, Oleg Bulichev, Dmitry Yudin · 2025
Read DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes by Sergey Linok, Vadim Semenov, Anastasia Trunova, Oleg Bulichev, Dmitry Yudin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03581.txt.
Introduction
Sergey Linok and co-authors present a method for answering questions about scenes that change over time, in this 2025 paper. Robots and smart agents need to reason about events, not just single snapshots. Most vision systems read images directly, which hides the objects and links between them inside opaque features. Their method instead writes each moment as a scene graph, a text map of objects and their relations.
A sequence of these graphs is then compressed and fed to a large language model. The model reasons over the clear structure to answer questions about what happened. Because the input spells out objects and their ties across time, the reasoning can be inspected, not just trusted. The approach targets settings like home robots that must explain what they saw.
Catalog record
| CREATOR | Sergey Linok, Vadim Semenov, Anastasia Trunova, Oleg Bulichev, Dmitry Yudin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03581
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes Sergey Linok, Vadim Semenov, Anastasia Trunova, Oleg Bulichev, Dmitry Yudin Abstract The analysis of events in dynamic environments poses a fundamental challenge in the development of intelligent agents and robots capable of interacting with humans. Current approaches predominantly utilize visual models. However, these methods often capture information implicitly from images, lacking interpretable spatial-temporal object representations. To address this issue we introduce DyGEnc - a novel method for Encoding a Dynamic Graph. This method integrates compressed spatial-temporal structural observation representation with the cognitive capabilities of large language models. The purpose of this integration is to enable advanced question answering based on a sequence of textual scene graphs. Extended evaluations on the STAR and AGQA datasets indicate that DyGEnc outperforms existing visual methods by a large margin of 15–25% in addressing queries regarding the history of human-to-object interactions. Furthermore, the proposed method can be seamlessly extended to process raw input images utilizing foundational models for extracting explicit textual scene graphs, as substantiated by the results of a robotic experiment conducted with a wheeled manipulator platform. We hope that these findings will contribute to the implementation of robust and compressed graph-based robotic memory for long-horizon reasoning. Code is available at github.com/linukc/DyGEnc 4 4 4 This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. . I INTRODUCTION Interpretable object maps for representing the surrounding environment for robots are an actively researched topic. These maps include descriptions — either explicit textual annotations or implicit representations in the form of extracted features — of scene elements along with their 3D positions and orientations, typically for subsequent utilization with large language models that facilitate logical analysis and reasoning of user’s textual queries. ConceptGraphs [ 1 ] , BBQ [ 2 ] , Search3D [ 3 ] and analogous approaches construct advanced graph structures from a sequence of positioned frames using fundamental visual models. This enables the identification of objects of interest through arbitrary text queries that specify diverse inter-object spatial relationships. HOV-SG [ 4 ] and Clio [ 5 ] employ a multi-level hierarchy to represent large interior spaces as layered graphs (e.g. floors, rooms), with each node preserving its unique features. This approach substantially narrows the search context for text queries and facilitates scaling the knowledge maps of intelligent agents to extensive areas. Fig. 1: DyGEnc compactly encodes a dynamic graph (sequence of textual scene graphs) of a changing environment in a few tokens. The resulting representation is then utilized by an aligned large language model for situated logical reasoning and question answering. The presented methods operate under the assumption that the observed environment is static, which significantly limits their potential deployment in real-world settings where variability is a key attribute. PSG4D [ 6 ] incorporates an object tracking procedure, allowing an observation to be represented as a graph of objects with edges that depend on the query time. However, if the resulting representation is entirely conveyed in textual form to the context of a large language model, it can lead to hallucinations during logical reasoning due to the large volume of information generated by continuous changes. G-Retriever [ 7 ] advances the idea of encoding single graph representation [ 8 , 9 ] , enabling a concise and implicit depiction of the scene description in the form of specialized input tokens with high compression rate and without information quality drop. To address the context limit for dynamic scenes, we propose the DyGEnc method, which extends the encoding concept to sequences of graphs (dynamic graph), as illustrated in Fig. 1 . Thus, our key contributions are as follows: 1. DyGEnc architecture for encoding sequence of textual scene graphs based on a parameter-efficient fine-tuning of a large language model (Sec. III ); 2. A comprehensive analysis of DyGEnc components and their impact on the model performance, evaluated on STAR [ 10 ] and AGQA [ 11 ] benchmarks (Sec. IV-D ); 3. A practical approach for deploying DyGEnc for real-world robotic applications by leveraging foundational models for extracting textual scene graphs from a sequence of images (Sec. IV-F ). Code is available at github.com/linukc/DyGEnc . II RELATED WORK II-A Dynamic Scene Graph Generation We define Dynamic Scene Graph (DSG) as a sequence of graphs in which the connections are characterized not only by spatial relations between objects but also by action connections between moving actors and objects. The prediction of such graphs from sensory data (primarily image sequences) has been extensively researched. Two broad categories of modern methods can be identified: firstly, end-to-end trainable approaches, and secondly, graph construction based on foundational models. The advent of trainable methods was precipitated by the emergence of manually labeled graph datasets such as Visual Genome [ 12 ] , GQA [ 13 ] , PSG [ 14 ] , Action Genome [ 15 ] and STAR [ 10 ] . Today, a vast set of diverse approaches exists, with some of the most state-of-the-art being PSG [ 14 ] , PVSG [ 16 ] , EGTR [ 17 ] , Reltr [ 18 ] , and OED [ 19 ] transformer image-based graph predictors. In the second group of methods, the utilization of foundational models [ 20 , 21 , 22 ] , as well as modern large language models with visual input (vLLM) [ 23 , 24 , 25 , 26 ] , is worthy of particular attention. These models are actively employed to generate uprising synthetic graph annotations [ 27 , 28 , 29 ] . The second group of approaches exhibits better generalization on new data than trainable methods on existing graph datasets, which are limited in their diversity — a key factor for the proposed algorithm’s performance across a wide range of possible scenarios — but demand human-in-a-loop to correct hallucinations and verify output. II-B Video Question Answering (VQA) with DSG In recent years, Video Question Answering (VideoQA) has emerged as one of the most rapidly developing research areas at the intersection of computer vision, natural language processing, and multimodal learning. However, as recently substantiated by studies in the field [ 30 , 31 , 32 ] , existing vLLMs encounter difficulties in accumulating perceived data in a meaningful inner representation due to their implicit encoding of input images. This feature plays a particularly critical role in dynamic scene understanding, where the relationships between entities are in constant flux. That is why alternative architectures are being developed, incorporating analogs of graph-based hierarchical representations. The authors [ 33 ] propose to represent video as a (2.5+1)D scene graph, where each node has spatio-temporal coordinates. To construct the graph, they employ detection and tracking models pre-trained on a specific domain and train their transformer model for question answering. The creators [ 34 ] assembled their graph dataset and trained a model to generate Egocentric Action Scene Graphs for video representation, while logical reasoning is performed by passing the information into the context of a large language model. HyperGLM [ 35 ] proposes a Video Scene HyperGraph, where hyperedges are depicted as polygons, encapsulating interactions through