Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior
Marcus Kessel · 2025
Read Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior by Marcus Kessel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02795.txt.
Introduction
Marcus Kessel argues that code generating AI should learn from watching software run, not just reading it. Today's models train on static artifacts: source code, comments, and specifications. They rarely see recorded behavior, so they can absorb buggy or mislabeled code as if it were truth.
Deep facts about a program cannot be decided in general, so observed runs are the practical ground truth. Kessel proposes observation lakehouses: living, interactive archives of software behavior. Feeding models real execution traces could teach them what code does, not just what it looks like, cutting confident errors. Runs leave a trail that reads can check, and that trail is the part most code tools throw out too soon.
Catalog record
| CREATOR | Marcus Kessel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02795
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior Marcus Kessel Abstract Code-generating LLMs are trained largely on static artifacts (source, comments, specifications) and rarely on materializations of run‑time behavior. As a result, they readily internalize buggy or mislabeled code. Since non‑trivial semantic properties are undecidable in general, the only practical way to obtain ground-truth functionality is by dynamic observation of executions. In prior work, we addressed representation with Sequence Sheets, Stimulus-Response Matrices (SRMs), and Stimulus-Response Cubes (SRCs) to capture and compare behavior across tests, implementations, and contexts. These structures make observation data analyzable offline and reusable, but they do not by themselves provide persistence, evolution, or interactive analytics at scale. In this paper, therefore, we introduce observation lakehouses that operationalize continual SRCs: a tall, append‑only observations table storing every actuation (stimulus, response, context) and SQL queries that materialize SRC slices on demand. Built on Apache Parquet + Iceberg + DuckDB, the lakehouse ingests data from controlled pipelines (LASSO) and CI pipelines (e.g., unit test executions), enabling n‑version assessment, behavioral clustering, and consensus oracles without re‑execution. On a 509‑problem benchmark, we ingest ≈ \approx 8.6M observation rows ( < < 51MiB) and reconstruct SRM/SRC views and clusters in < < 100ms on a laptop, demonstrating that continual behavior mining is practical without a distributed cluster of machines. This makes behavioral ground truth first‑class alongside other run‑time data and provides an infrastructure path toward behavior‑aware evaluation and training. The Observation Lakehouse, together with the accompanying dataset, is publicly available as an open‑source project on GitHub: https://github.com/SoftwareObservatorium/observation-lakehouse I Introduction Most code-generating LLMs are trained on static artifacts – source code, comments, and specifications – rather than on materializations of run‑time behavior [ 6 ] . Such models inevitably internalize patterns from buggy or mislabeled code as readily as from correct implementations, because they are rarely, if ever, exposed to trustworthy evidence of functional run-time behavior (i.e., functionality) [ 8 , 10 ] . Improving reliability, therefore, requires exposing models to ground‑truth behavior captured under execution. Rice’s theorem rules out deciding non‑trivial semantic properties of programs in general [ 18 ] , so for real systems the only practical way of doing this is through dynamic observation – that is, to execute software where it is expected to meet its advertised functionality and record what it actually does, either during controlled tests (i.e., software testing [ 1 ] ) or during real usage (i.e., monitoring and tracing). Current technology makes this much harder than it should be. Unit‑testing frameworks typically discard most observations (observable outputs to calls), preserving only pass/fail outcomes, and burying stimuli in scaffolding code. Execution tracing captures the opposite extreme – massive, low‑level call/state data – from which functional responses must be mined laboriously. In short, testing captures too little, while (extensive) tracing captures too much, and both obscure the functionality that is important. In earlier work, we addressed the representation gap by introducing three data structures tailored to the observation and recording of run‑time functional behavior [ 12 ] : Sequence Sheets to encode the precise invocations and the responses they elicit, the Stimulus-Response Matrices (SRMs) to compare many implementations against many tests, and Stimulus-Response Cubes (SRCs) (formely SRHs) to analyze ensembles of SRMs across additional dimensions such as repetitions, environments, and metrics. In an SRM, rows are tests and columns are implementations of the same functional abstraction (i.e., coding problem). Each cell stores the actuation (operation, inputs, outputs) for that test-implementation pair. SRCs bundle related SRMs into a multi‑dimensional structure for navigation and analysis. These structures make behavioral data analyzable offline through data-driven analytics, and reusable (replication, reproduction, repurposing) [ 12 ] . While the efficacy of these data structures has been demonstrated, also at large scales [ 12 , 11 ] , their current implementation in the LASSO platform assumes semi-sanitized scenarios where the tests and implementations that populate SRCs are “curated” for specific test-driven software experiments, or for particular diffential GAI applications [ 14 ] . At present, they are not supported by any infrastructure that can collect this information, at scale and at high speed, from mainstream development environments and execution platforms, and they were not developed to allow SRC data to live alongside all the other data that is generated during software execution. More specifically, SRCs currently lack an environment/infrastructure that i) keeps every actuation, not just pass/fail verdicts; (ii) evolves without friction as new tests, implementations, environments, measurements and context appear; and (iii) answers interactive queries that reconstruct SRC views and support higher-level analyses (e.g., behavioral clustering, n-version assessment) without a distributed cluster of machines, and (iv) allows SRC data to live alongside other forms of structured and unstructured data within a broader record of run-time software execution. In short, we need an infrastructure that realizes the vision of continual SRCs – motivated recently in our research agenda for behavior-aware LLMs [ 13 ] – that grow as code is tested and make it integrable in and accessible from the current software engineering data tool landscape. To this end, we introduce the notion of an Observation Lakehouse that operationalizes the continual SRCs concept. Realized as a Python application based on a modern data lakehouse architecture [ 4 ] , it uses Apache Iceberg [ 3 ] as a transactional table format over columnar Apache Parquet files [ 17 ] , with DuckDB for high-performance querying [ 7 ] . Our approach unifies behavioral observations from two complementary streams – controlled software experiments and real-world CI pipelines – into a single, append-only data model. This foundation makes large-scale differential analysis practical and efficient, allowing users to perform n-version assessment of code variants, use behavioral clustering to identify equivalence classes, and compute consensus oracles without re-executing historical runs. The feasibility of this approach is demonstrated on a benchmark of over 8 million observation rows, where complex analytical queries – using SQL – complete in under 100ms on a standard laptop with a storage footprint of less than 51MiB. This paper contributes a lakehouse infrastructure that provides a practical foundation for mining software behavior at scale, addressing a key obstacle to creating more behavior-aware GAI. The remainder of the paper is organized as follows. Section II details the lakehouse architecture and the data-modeling choices that enable a continual SRC. Section III reports ingestion throughput, query latency, and the cost of behavioral clustering that underpins consensus‑oracle construction. Section IV discusses extensions (semantic search, Model‑Context‑Protocol server, federation) and outlines current limitations, before concluding with a call to the research community to adopt and extend the observation lakehouse. II Observation Lakehouse Architecture In this section, we describe the design and implementation of the Observation Lakehouse, a platform that materializes the continual SRC [ 13 ] . The architecture is designed to satisfy four core requirements: (1) support for concurrent, incremental ingestion of both experimental and practitioner data; (2) unrestricted schema evolution for dimensional extensibility; (3) efficient on-the-fly reconstruction of arbitrary SRMs; and (4) fast, expressive analytics over massive observation sets. II-A Conceptual Model: A Continual and Unified SRC An SRC is a collection of SRMs (see Figure 1 ), one for each functional abstraction (either a coding problem described in some way or a concrete unit under test like