DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement

Shaoqing Lin, Chong Teng, Fei Li, Donghong Ji, Lizhen Qu,Zhuang Li · 2025

Read DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement by Shaoqing Lin, Chong Teng, Fei Li, Donghong Ji, Lizhen Qu,Zhuang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15583.txt.

Introduction

Shaoqing Lin and five co-authors tackle graph parsing for long, multi-sentence image descriptions, in a paper from twenty twenty-five. A scene graph lists objects and their relations as triples. Parsers were built for single-sentence captions, so merging their outputs for long text drops links across sentences and breaks the graph.

The team defines the fuller task and releases a data set of caption and graph pairs, part hand-checked by experts and part machine-made. Their own parser drafts a graph, then a learned editor refines it step by step. It scores well above the merging baseline while running far faster than the big commercial model they tested. Downstream checks, such as spotting made-up content, also improve.

Catalog record

CREATORShaoqing Lin, Chong Teng, Fei Li, Donghong Ji, Lizhen Qu,Zhuang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15583

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement Shaoqing Lin, Chong Teng, Fei Li, Donghong Ji, Lizhen Qu,Zhuang Li Abstract Vision-Language Models (VLMs) now generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers originally designed for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. To address this, we introduce a new task, Disco urse-level text S cene G raph parsing ( DiscoSG ), supported by our dataset DiscoSG-DS , which comprises 400 expert-annotated and 8,430 synthesised multi-sentence caption–graph pairs for images. Each caption averages 9 sentences, and each graph contains at least 3 × \times more triples than those in existing datasets. While fine-tuning large PLMs (i.e., GPT-4o) on DiscoSG-DS improves SPICE by ∼ \sim 48% over the best sentence-merging baseline, high inference cost and restrictive licensing hinder its open-source use, and smaller fine-tuned PLMs struggle with complex graphs. We propose DiscoSG-Refiner , which drafts a base graph using one small PLM, then employs a second PLM to iteratively propose graph edits, reducing full-graph generation overhead. Using two Flan-T5-base models, DiscoSG-Refiner still improves SPICE by ∼ \sim 30% over the best baseline while achieving 86 × \times faster inference than GPT-4o. It also consistently improves downstream VLM tasks like discourse-level caption evaluation and hallucination detection. Code and data are available at https://github.com/ShaoqLin/DiscoSG . * * footnotetext: Corresponding author 1 Introduction Text scene graph parsing converts visual descriptions into semantic graphs of entities and their relations, supporting tasks like image captioning evaluation Dong et al. 2024 , hallucination detection Yu et al. 2024 , and image retrieval Johnson et al. 2015 . Traditional methods focus on single-sentence captions, converting dependency parses into graphs or fine-tuning pre-trained language models (PLMs) on sentence-level pairs. While effective for short captions, these methods are increasingly inadequate due to a paradigm shift in multimodal research. Vision-language models (VLMs) now generate detailed discourse-level, multi-sentence captions with complex inter-sentential dependencies Cheng et al. 2025 . Processing discourse-level text introduces several critical challenges for current sentence-level parsers: First , cross-sentence coreference resolution requires correctly linking entities across sentences (e.g., “a woman” later as “she”). Second , capturing long-range relations between entities in different sentences. Third , inferring implicit relationships not stated in any single sentence (e.g., (cat, near, window) from “The cat is on the mat” and “The mat is under the window”). Finally , ensuring graph coherence by producing a globally consistent and complete scene representation. However, existing methods Dong et al. 2024 typically resort to heuristically merging sentence-level parsing outputs, resulting in semantically inconsistent graphs that ignore long-range dependencies. To address the challenges, we formulate a new task, Disco urse-level text S cene G raph parsing ( DiscoSG ), which converts multi-sentence descriptions into complex scene graphs. We introduce DiscoSG-DS , a dataset of 400 expert-annotated and 8,430 synthesised caption-graph pairs for images. Each caption averages 9 sentences, and each graph contains \sim 15 \times more objects and relations than earlier datasets Li et al. 2023a . Our annotation guidelines ensure graphs capture cross-sentence coreference, long-range dependencies, implicit relations, and global semantic consistency. However, standard approaches like fine-tuning PLMs Li et al. 2023a on caption-graph pairs face challenges with DiscoSG-DS. Empirically, naively fine-tuning smaller PLMs performs poorly due to the complexity of discourse-level graph generation. While fine-tuning large PLMs (e.g., GPT-4o 1 1 1 https://openai.com/index/hello-gpt-4o/ ) yields better quality, it is impractical for open-source deployment due to high cost and licensing. Few-shot prompting of large PLMs Yang et al. 2025 also underperforms fine-tuned counterparts on this discourse task due to the lack of task-specific adaptation. To overcome these limitations, we propose DiscoSG-Refiner, a novel lightweight framework for DiscoSG via iterative graph refinement. It begins with a base graph generated by merging SOTA sentence-level parsers, then iteratively improves graph quality through complementary edit operations proposed by an encoder-decoder PLM-based “Programmer”. This module innovatively disentangles edit operations: the encoder classifies and removes erroneous and inconsistent triples, while the decoder generates new triples to enhance completeness and semantic coverage. The Programmer is trained on DiscoSG-ED, a dataset of synthetic edit operations derived from DiscoSG-DS. This approach significantly reduces generation overhead while incorporating broader inter-sentence context and task-specific knowledge. GPT-4o fine-tuned on DiscoSG-DS outperforms the SOTA sentence-merging baseline by 48% on SPICE Anderson et al. 2016 , but it is neither resource-efficient nor open-source due to licensing constraints. Our DiscoSG-Refiner, built with two Flan-T5-base Chung et al. 2024 models, achieves a 30% improvement over the same baseline while offering an efficient alternative with 86 \times faster inference. DiscoSG-Refiner also consistently improves downstream tasks , including evaluation of VLM-generated captions and hallucination detection, compared to baseline parsers. Due to the lack of resources for discourse-level hallucination detection, we introduce D-FOIL, a new benchmark inspired by FOIL Shekhar et al. 2017 , which was designed for sentence-level hallucination detection. D-FOIL consists of 200 pairs of hallucinated and corrected multi-sentence image captions, each accompanied by a human-written reference caption. Overall, our main contributions are: (i) defining DiscoSG, a novel task for discourse-level text scene graph parsing from multi-sentence visual descriptions; (ii) introducing DiscoSG-DS, a dataset of 400 expert-annotated and 8,430 synthesized examples for discourse-level parsing; (iii) proposing DiscoSG-Refiner, a lightweight iterative framework with a novel PLM-based “Programmer” trained on synthesized edits (DiscoSG-ED) derived from DiscoSG-DS; (iv) demonstrating that our 0.5B model outperforms 0.25B sentence-merging baselines by 30%, and is more resource-efficient than fine-tuned GPT-4o; and (v) establishing D-FOIL, a new benchmark for discourse-level hallucination detection. 2 DiscoSG-DS: Discourse-Level Scene Graph Dataset To study DiscoSG, we introduce the DiscoSG-DS dataset, which consists of 400 manually annotated gold-standard examples and 8,430 synthesised instances. Each instance aligns a discourse-level image description {\bm{x}}=(x_{1},\dots,x_{n}) with a corresponding scene graph {\bm{y}} . Each description {\bm{x}} contains n sentences that are contextually dependent, requiring parsers to model discourse-level semantics. Each graph ( {\bm{y}} ) captures the semantics of {\bm{x}} through a set of structured triples representing object relationships (e_{sub},r,e_{obj}) (such as (man, wear, hat) ) and attributes (e,a,v) (like (hat, has_attribute, red) ). 2.1 Source Data Selection Descriptions ( {\bm{x}} ) were sourced from the SharedGPT4V dataset Chen et al. 2024 , containing 1.2 million captions generated by GPT-4 for images. These highly descriptive captions span multiple sentences, making them suitable for discourse-level analysis. We selected 40,000 diverse image-description pairs using diversity sampling based on TF-IDF text embeddings, adapting methods from Zhuo et al. 2023 ; Li et al. 2023b . The selected descriptions were then annotated either manually or via synthesis to form the final DiscoSG-DS dataset. 2.2 Manual Annotation Pipeline To construct high-quality ground truth data for DiscoSG, we manually annotated 400 descriptions randomly selected from the pool of 40,000, using a human-in-the-loop active learning framework. Annotator Setup and Annotation Principles. We employed two expert annotators: a postdoctoral researcher with over three years of relevant task experience and a trained postgraduate student in computer science. To calibrate the postgraduate annotator, we conducted a phased training process in which the student annotated 40 examples across four batches under the postdoc’s supervision. By the final batch, the annotators had achieved 90% inter-annotator agreement. Text descriptions were

Read the complete text