Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions

Shue Shiinoki, Ryo Koshihara, Hayato Motegi, Masumi Morishige · 2025

Read Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions by Shue Shiinoki, Ryo Koshihara, Hayato Motegi, Masumi Morishige (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04389.txt.

Description

Shue Shiinoki, Ryo Koshihara, Hayato Motegi, and Masumi Morishige propose a text-driven workaround for diagram understanding in business documents. Vision-language models still struggle to extract structures and relationships from diagrams reliably. Office files, however, store diagram shapes, lines, and annotations as editable textual metadata.

The team extracted shape data from spreadsheet-based system design documents and fed it as text to a large language model. Compared with a vision-based method, the text-driven framework answered detailed structural questions more accurately. The approach extends to presentation and word-processing files with similar source data. The authors argue it can improve workflow efficiency and information analysis in real business settings.

Catalog record

CREATORShue Shiinoki, Ryo Koshihara, Hayato Motegi, Masumi Morishige
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04389

Browse the full library · About the Tower of Babel