Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insights DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06

Yifan Wu, Lutao Yan, Yizhang Zhu, Yinan Mei, Jiannan Wang, Nan Tang, Yuyu Luo · 2025

Read Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insights DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 by Yifan Wu, Lutao Yan, Yizhang Zhu, Yinan Mei, Jiannan Wang, Nan Tang, Yuyu Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10043.txt.

Introduction

Yifan Wu and co-authors help users find the right chart by typing a plain question, in a paper from 2025. Past test sets were too clean to mimic real business use, where many charts look much alike.

Their new bench holds near twenty two thousand charts and a few hundred real queries, set up as target versus close decoy tasks. Old tools that lean on looks alone scored poorly. The team auto wrote layered notes on each chart, from shape to use, and trained a finder on them. Scores rose by wide margins on both exact and fuzzy queries.

Catalog record

CREATORYifan Wu, Lutao Yan, Yizhang Zhu, Yinan Mei, Jiannan Wang, Nan Tang, Yuyu Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insights DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Yifan Wu, Lutao Yan, Yizhang Zhu, Yinan Mei, Jiannan Wang, Nan Tang, Yuyu Luo Abstract . Charts are crucial for data analysis and decision-making. Text-to-chart retrieval systems have become increasingly important for Business Intelligence (BI), where users need to find relevant charts that match their analytical needs. These needs can be categorized into precise queries that are well-specified and fuzzy queries that are more exploratory – both require understanding the semantics and context of the charts. However, existing text-to-chart retrieval solutions often fail to capture the semantic content and contextual information of charts, primarily due to the lack of comprehensive metadata (or semantic insights). To address this limitation, we propose a training data development pipeline that automatically synthesizes hierarchical semantic insights for charts, covering visual patterns (visual-oriented), statistical properties (statistics-oriented), and practical applications (task-oriented), which produces 207,498 semantic insights for 69,166 charts. Based on these, we train a CLIP-based model named ChartFinder to learn better representations of charts for text-to-chart retrieval. Our method leverages rich semantic insights during the training phase to develop a model that understands both visual and semantic aspects of charts. To evaluate text-to-chart retrieval performance, we curate the first benchmark, CRBench, for this task with 21,862 charts and 326 text queries from real-world BI applications, with ground-truth labels verified by the crowd workers. Experiments show that ChartFinder significantly outperforms existing methods in text-to-chart retrieval tasks across various settings. For precise queries, ChartFinder achieves up to 66.9% NDCG@10, which is 11.58% higher than state-of-the-art models. In fuzzy query tasks, our method also demonstrates consistent improvements, with an average increase of 5% across nearly all metrics. 1. Introduction Visualization charts (charts for short) are essential for various fields, including data analysis and business decision-making ( Wu et al. 2024 ; Luo et al. 2018a ; Shen et al. 2022 ; Qin et al. 2020 ) . In particular, Business Intelligence (BI) systems heavily rely on charts to present complex data in ways that provide users with intuitive and actionable insights ( Setlur et al. 2023 ; Luo et al. 2023 ) . Most charts are carefully designed to serve specific analytical tasks, with considerations for visual design, chart types, and data relationships. Given their quality and versatility, these well-crafted charts are valuable resources that can be reused across different tasks and contexts. Given a user-provided text query and a chart repository, the text-to-chart retrieval task is to find a set of charts that align with the user’s query. This process requires understanding both the visual characteristics of the charts and the semantic content of the user query , ensuring that the retrieved charts are not only relevant but also contextually suitable for the user’s analytical task. Figure 1. ChartFinder integrates both visual and semantic aspects of charts to improve text-to-chart retrieval. Text-to-Chart Retrieval in Real-world BI Systems. Through collaboration with a leading BI company X [Redacted], we analyzed real-world user behavior and identified two primary categories of user queries for chart retrieval: precise queries and fuzzy queries. These categories reflect the diverse needs of users, ranging from specific, data-driven queries to more exploratory or stylistic tasks. Precise queries are characterized by data-driven intent and focus on specific analytical targets. These queries aim to retrieve charts that present well-defined information, such as trends, comparisons, or aggregated data points, to answer a clear analytical question. For example, as shown in Figure 1 , a user may ask, “Find charts showing the hotel booking distribution across channels in Europe for the year 2022.” Such queries demand high semantic alignment to ensure the retrieved charts align closely with the user’s specific requirements. To achieve this, retrieval systems must effectively leverage both the data context and the visual features of chart images to establish strong semantic relationships between the query and the chart. Fuzzy queries , on the other hand, are more open-ended and often focus on exploratory tasks and abstract information. These queries are less constrained and may emphasize stylistic or structural elements of charts rather than specific data content. For example, a user might query, “Use a pie chart to compare different sale methods”, as shown in Figure 1 . Addressing such queries requires a retrieval system capable of understanding a broader intent and contextual flexibility inherent in exploratory tasks. Limitations of Existing Methods. Text-to-image retrieval has seen remarkable advancements, leveraging state-of-the-art Multimodal Large Language Models (MLLMs) ( Liu et al. 2022 ) . These approaches have demonstrated effectiveness in general image retrieval tasks by efficiently capturing visual features from images. However, when being directly adapted to text-to-chart retrieval, these methods face several challenges in addressing the unique aspects of charts. First, existing methods fail to capture the semantic depth of charts . Unlike natural images, which primarily convey visual information, charts encode complex data relationships, trends, and comparisons. Existing methods that focus solely on visual features often retrieve charts that are visually similar but semantically irrelevant. For example, two charts may share visual characteristics ( e.g., both being bar charts) but represent entirely different datasets, topics, or analytical purposes. This limitation is particularly problematic for precise queries , which require high semantic alignment between the query and the retrieved chart to meet data-driven needs. Second, current approaches fail to address both precise and fuzzy queries effectively. For precise queries, existing methods that prioritize visual features lack the ability to connect queries with the underlying data or analytical intent, leading to poor retrieval performance. For fuzzy queries, existing methods struggle to interpret broader user intents, such as stylistic preferences or exploratory goals, as they lack mechanisms to capture design elements of charts. Key Idea. To improve text-to-chart retrieval, it is essential to capture both the visual and semantic aspects of charts. Our key idea is to extract semantic insights from chart metadata ( e.g., source code) and combine this with the chart images. By training a model that incorporates both the charts ( in the form of images ) and the semantic insights ( in the form of text ), we aim to develop a system that comprehensively understands and bridges the gap between visual representation and the underlying semantic meaning of charts. However, implementing this idea presents several challenges . First, manually annotating semantic insights for charts is costly and time-consuming. Thus, we need to develop an efficient and scalable method to derive these semantic insights from chart metadata automatically. Once semantic insights are generated, the second challenge is to effectively integrate them with the visual features of the charts. We need an effective model architecture and training strategy that can seamlessly combine these two modalities, i.e., semantic insights and visual features, to generate more accurate representations of the charts. Third, there is currently no readily available corpus containing text queries and their corresponding charts to evaluate the performance of text-to-chart retrieval. Our Proposal. We propose ChartFinder, a new text-to-chart retrieval model that leverages contrastive learning techniques to align both the visual features and semantic insights of charts, based on the Contrastive Language-Image Pre-Training architecture ( Radford et

Read the complete text