Semantic Scaffolding: Augmenting Textual Structures with Domain-Specific Groupings for Accessible Data Exploration

Jonathan Zong Isabella Pedraza Pineros Mengzhu (Katie) Chen MIT CSAIL MIT CSAIL MIT CSAIL Daniel Hajas Arvind Satyanarayan Global Disability Innovation Hub MIT CSAIL · 2025

Read Semantic Scaffolding: Augmenting Textual Structures with Domain-Specific Groupings for Accessible Data Exploration by Jonathan Zong Isabella Pedraza Pineros Mengzhu (Katie) Chen MIT CSAIL MIT CSAIL MIT CSAIL Daniel Hajas Arvind Satyanarayan Global Disability Innovation Hub MIT CSAIL (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15883.txt.

Introduction

The authors present a way to help readers grasp what grouped data mean in the real world. A chart pattern may catch the eye while its meaning stays unclear, or a known idea may be hard to phrase in dataset terms. Their method asks a large language model for domain groupings and shows them as bins over a field or notes on sets of records.

The method is built into Olli, a chart tool made for blind and low-vision readers. In a study with fifteen such readers, the groupings helped people grasp data fast. Readers also stayed alert to how the notes could steer their reading.

Catalog record

CREATORJonathan Zong Isabella Pedraza Pineros Mengzhu (Katie) Chen MIT CSAIL MIT CSAIL MIT CSAIL Daniel Hajas Arvind Satyanarayan Global Disability Innovation Hub MIT CSAIL
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15883

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Scaffolding: Augmenting Textual Structures with Domain-Specific Groupings for Accessible Data Exploration Jonathan Zong Isabella Pedraza Pineros Mengzhu (Katie) Chen MIT CSAIL MIT CSAIL MIT CSAIL Daniel Hajas Arvind Satyanarayan Global Disability Innovation Hub MIT CSAIL Abstract Drawing connections between interesting groupings of data and their real-world meaning is an important, yet difficult, part of encountering a new dataset. A lay reader might see an interesting visual pattern in a chart but lack the domain expertise to explain its meaning. Or, a reader might be familiar with a real-world concept but struggle to express it in terms of a dataset’s fields. In response, we developed semantic scaffolding , a technique for using domain-specific information from large language models (LLMs) to identify, explain, and formalize semantically meaningful data groupings. We present groupings in two ways: as semantic bins , which segment a field into domain-specific intervals and categories; and data highlights , which annotate subsets of data records with their real-world meaning. We demonstrate and evaluate this technique in Olli, an accessible visualization tool that exemplifies tensions around explicitly defining groupings while respecting the agency of readers to conduct independent data exploration. We conducted a study with 15 blind and low-vision (BLV) users and found that readers used semantic scaffolds to quickly understand the meaning of the data, but were often also critically aware of its influence on their interpretation. \onlineid 1056 \vgtccategory Research \vgtcinsertpkg \teaser Data highlights generated from an example cars dataset using the semantic scaffolding technique. Each highlight is shown with a visualization of its query predicate, demonstrating that highlights correspond to areas of interest in the data. Introduction keywords Data visualization, data narrative, accessibility. When a lay reader encounters a new dataset on a website or news article, an important part of their exploratory process is to identify interesting groupings of data, and explain them in terms of that data’s real-world meaning. Broadly, the real-world meaning of data is known as its semantics ; domain-specific knowledge about data is how a reader determines whether a number “represent[s] a day of the month, or an age, or a measurement of height, or a unique code for a specific person, or a postal code for a neighborhood, or a position in space” [ 11 ] or other possible meanings. Effectively drawing connections between data and its semantics can be challenging for a lay reader. For instance, a reader might see an interesting visual pattern in a chart but lack the domain-specific knowledge to explain its meaning. In this case, difficulty arises because the reader does not know what they don’t know — it may be difficult to know how to begin to acquire the relevant context. Or, a reader might be familiar with a real-world concept but struggle to translate it into the terms of the dataset. For example, a reader may not know how to approximate a subjective concept like “sports car” in terms of fields like Horsepower and Miles_per_Gallon — or possibly other fields in the dataset that were not in the initial visualization. These challenges can affect a reader’s comprehension of the data — as prior research has shown, a reader’s interpretation of a visualization is sensitive to differences in their prior knowledge and personal background [ 6 , 13 ] . In response, we developed a technique called semantic scaffolding for using domain-specific information from large language models (LLMs) to identify, explain, and formalize semantically meaningful data groupings. Rather than use an LLM to generate summary descriptions or chat responses, we use it to return groupings as data structures that include a name , explanation , and query predicate . For instance, in the example cars dataset, an LLM might create a grouping named “Fuel Efficient Japanese Cars” with the following explanation: “This group represents cars from Japan that are known for their fuel efficiency, reflecting Japanese automotive engineering and consumer trends towards sustainable driving” ( \autoref fig:teaser). Crucially, the LLM associates this grouping with the following query predicate: Miles_per_Gallon \geq 25 \cap Origin = Japan . This allows a reader to connect the name and explanation with an explicitly-defined subset of data. We developed two types of interface elements for presenting semantic groupings to a reader: semantic bins , which segment a single field (i.e. column) into domain-specific intervals and categories; and data highlights , which annotate subsets of data records (i.e. rows) with their real world meaning. These two uses of semantic scaffolding in an interface serve distinct purposes. Semantic bins help a reader break down a single field into understandable pieces, facilitating navigation and exploration through a dataset. Data highlights help a reader quickly get an overview of a dataset, indicating potentially interesting subsets to begin exploring further. We prototyped these designs via extensions to Olli [ 1 ] , an accessible visualization system for screen reader users. Data accessibility is a rich context in which to evaluate our work because current tools for blind and low vision (BLV) screen reader users amplify the challenges lay readers of visualizations face. First, sighted readers often rely on their visual perception to identify interesting groupings in a visualization, but screen reader interfaces rarely afford a similar type of overview. Second, BLV readers highly value the agency to independently explore data, conduct an open-ended interpretive process, and arrive at their own conclusions about the data [ 10 ] . For instance, research has shown that screen reader users find descriptions less useful when they over-emphasize contextual and domain-specific information at the expense of descriptive statistics or the data values themselves [ 10 , 2 ] . These two factors require screen reader interfaces to strike a balance between making groupings available without overly prescribing a reader’s interpretation. In our preliminary user testing, we found that textual scaffolding was able to accelerate BLV readers’ understanding of a dataset’s real-world meaning, but that readers were often critically aware of AI’s influence on their interpretation process. 1 Related Work 1.1 Accessible Textual Data Representations To make data visualizations accessible to screen reader users, a designer must provide descriptions that can be read as text-to-speech. Because conventional static alt text does not afford data exploration at varying levels of detail comparable to strategies sighted readers employ, researchers have turned to structured textual descriptions [ 20 ] , which enable screen reader users to navigate along a hierarchy and move between overview and detail with textual descriptions. Accessible visualization systems that incorporate structured textual descriptions include Olli [ 1 ] , Data Navigator [ 3 ] , VizAbility [ 4 ] , Chart Reader [ 18 ] , and Umwelt [ 21 ] . This work introduces a technique for augmenting textual structures with LLM-generated groupings, and demonstrates the technique via extensions to Olli. 1.2 Generating Textual Descriptions with Language Models Natural language generation of textual descriptions for data visualization is an area of research that has received renewed attention due to recent advances in large language models (LLMs). There are generally two interface design approaches that these systems have taken. First are systems that provide a chat-like interface with which a user can query a description by inputting a question in natural language (often known as chart question answering

Read the complete text