A Case for Computing on Unstructured Data

Mushtari Sadia, Amrita Roy Chowdhury, Ang Chen · 2025

Read A Case for Computing on Unstructured Data by Mushtari Sadia, Amrita Roy Chowdhury, Ang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14601.txt.

Introduction

Mushtari Sadia, Amrita Roy Chowdhury and Ang Chen argue in 2025 for computing directly on unstructured data: text, images, audio and video that fill most of the world information. Traditional data systems compute well only on structured tables. Unstructured riches stay hard to query and transform.

Their proposed paradigm runs in three stages. Latent structure is first extracted from raw media. Structured processing tools then transform that structure. Results are finally projected back into unstructured form for people and AI to consume. The two way pipeline aims to keep human friendly richness while gaining analytic power. Position papers matter when they name an architecture: this one gives system builders a concrete loop to implement and test.

Catalog record

CREATORMushtari Sadia, Amrita Roy Chowdhury, Ang Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Case for Computing on Unstructured Data Mushtari Sadia, Amrita Roy Chowdhury, Ang Chen Abstract . Unstructured data, such as text, images, audio, and video, comprises the vast majority of the world’s information, yet it remains poorly supported by traditional data systems that rely on structured formats for computation. We argue for a new paradigm, which we call computing on unstructured data, built around three stages: extraction of latent structure, transformation of this structure through data processing techniques, and projection back into unstructured formats. This bi-directional pipeline allows unstructured data to benefit from the analytical power of structured computation, while preserving the richness and accessibility of unstructured representations for human and AI consumption. We illustrate this paradigm through two use cases, and present the research components that need to be developed in a new data system called MXFlow . 1. Introduction The majority of today’s data is unstructured—natural language text, images, audio, or video. This is not only because the world inherently produces analog signals, but also because the human brain is naturally adept at interpreting unstructured inputs. While our world and our cognition are fundamentally geared toward unstructured content, our computing machinery excels at manipulating structured formats. This mismatch means that much of the world’s data has been effectively locked away from our computing tools. Despite decades of progress, data systems require that the data already adheres to a structured format (e.g., tables, graphs, programs). Recent advances in AI, particularly large language models (LLMs) and vision models, are beginning to shift this landscape. A key insight is that real-world data is "unstructured" only in its format—not in its meaning. Such data often contains rich, latent semantic structure that is intuitive to humans. AI models, like humans, can interpret and reason over this unstructured data and, when guided appropriately, can extract and articulate its underlying structure. Capitalizing on this capability, a recent body of work has demonstrated promising results in analyzing unstructured data ( Lin et al. 2024 ; Wang et al. 2025 ; Biswal et al. 2024 ) . Notably, Madden et al. ( Madden et al. 2024 ) envision a future where we can "query all of the world’s bytes" using AI. However, existing work has largely focused on one-way translation: converting unstructured data into structured formats to just enable querying. In this paper, we argue for a broader and more ambitious vision: a new paradigm of computing on unstructured data . Our vision goes beyond querying—it involves computing in the full sense of the term. Specifically, querying is essentially a read-only operation that allows efficient information retrieval. In contrast, by computing we refer to a read/write paradigm where one can manipulate, interact with and reason over unstructured content. To operationalize this vision, we propose a holistic eXtract–Transform–Project (XTP) pipeline. The extract phase ingests raw unstructured data, and converts it into a structured internal representation- for instance, transforming text into tables or images into symbolic programs, or audio into event timelines. The transform phase applies data processing techniques (e.g., optimization, imputation, validation) over this structured representation. Finally, the project phase translates the transformed results back into unstructured formats, either the same as the original or a different one, making the outputs easier for humans and AI agents to consume and act upon. A central feature of the XTP pipeline is its bidirectional nature. While prior work has looked at each of these phases individually, XTP closes the loop and enables end-to-end computation. This mirrors how humans think: we extract latent structure to reason, then re-embed those insights in unstructured ways (e.g.,language, visuals, audio) to communicate. The intermediate structured representation serves as a computational substrate, enabling principled and verifiable transformations, while the final projection step ensures that results remain compatible with how data is consumed in practice. By tightly integrating structure and unstructured representation in a single loop, XTP unlocks the best of both worlds: it retains the expressiveness and richness of unstructured data, while unlocking the analytical power and rigor of structured computation. Consider two motivating examples. (i) Text: Doctors often work with free-form text, such as medical prescriptions. An XTP pipeline can automatically extract patient information and diagnoses to build a structured database. While querying this data (e.g., “Which patients were prescribed Drug X?”) enables retrieval, real-world tasks often require more, such as adjusting dosages or identifying at-risk patients based on complex factors. These require direct computation and data modification. The transform phase enables such updates, and the project phase generates human-readable outputs like revised prescriptions or reports. (ii) Diagrams : Engineers use complex visual artifacts (e.g., datacenter layouts, schematics, blueprints) rich in information but hard to process. XTP can extract formal representations (e.g., a symbolic program showing the components and connections) of the visual artifacts, whether circuit schematics or datacenter layouts. While simple queries (e.g., “Does this node have upstream power?”) aid inspection, tasks like validating redundancy or optimizing fault tolerance require transform operations: editing components, simulating flow, or refactoring layouts. The project phase regenerates an updated visual diagram for engineers to inspect and refine. Figure 1 . A sketch of our envisioned design. ① Shows operator patterns, where operators can be neural, symbolic, or hybrid, with possible self-loops or dynamic routing. ② Depicts an operator for which the query planner selects neural or symbolic tools based on criteria such as service-level objectives (SLOs). These objectives apply to the full end-to-end task and must be considered holistically during tool selection. Operators (e.g., ExtractGraph ) consume data of enforced types (e.g., type: Image ) and produce outputs in specific formats. ③ Shows an example query, where the query planner orchestrates a concrete sequence of XTP operators to extract structure, apply transformations, and project an updated diagram and bill of materials report. Achieving this vision requires addressing a range of research challenges. Unstructured data is inherently more ambiguous and fuzzy than what conventional data systems can deal with. A naïve approach is to “throw everything at the LLM,” but LLMs are non-deterministic, prone to hallucination, and often make subtle errors. Additionally, they are difficult to integrate as computing modules due to their unpredictability and variability in performance, cost, and failure modes. We argue that tackling these challenges requires rethinking data system design, specifically, through a new system specialized for unstructured data management, which we call MXFlow . MXFlow allows users to express tasks as a dataflow graph, where nodes represent operators and edges encode the flow of data. This structure supports a principled decomposition of tasks into subtasks, each of which can be independently scheduled, executed, and validated using tools tailored to the specific operation. Yet, MXFlow departs from traditional dataflow systems in several ways. The first is the use of hybrid tools. MXFlow supports both neural and symbolic operators. While large language and vision models offer powerful tools for handling unstructured inputs, they need to be coupled with symbolic methods that are deterministic and come with strong correctness guarantees. MXFlow enables seamless composition of these tool types—leveraging the flexibility and expressiveness of neural models when needed, while falling back on symbolic techniques, built on decades of data management research, wherever possible. Moreover, MXFlow enables dynamic dataflow. Unlike traditional dataflow systems with fixed control paths, MXFlow supports dynamic

Read the complete text