LakeVilla: A Modular and Non-Invasive Toolbox for Lakehouse Transactions

Tobias Götz, Daniel Ritter, Jana Giceva · 2025

Read LakeVilla: A Modular and Non-Invasive Toolbox for Lakehouse Transactions by Tobias Götz, Daniel Ritter, Jana Giceva (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20768.txt.

Introduction

Tobias Götz, Daniel Ritter, and Jana Giceva present a toolbox for data lakehouses, in a paper published in 2025. Lakehouses store business data in cheap cloud files while offering table-like queries. Their open table formats, though, lack features databases take for granted, like changes that span several tables and safe recovery when a job aborts midway.

Their system, LakeVilla, adds those features on top of two popular formats without changing them. It supplies recovery, complex transactions, and isolation between readers and writers, even for long jobs. Tests showed small costs: about two percent slower writes and two and a half percent slower reads on standard benchmarks. The authors argue lakehouses can gain database safety while keeping their open, mix-and-match design.

Catalog record

CREATORTobias Götz, Daniel Ritter, Jana Giceva
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LakeVilla: A Modular and Non-Invasive Toolbox for Lakehouse Transactions Tobias Götz, Daniel Ritter, Jana Giceva Abstract Data lakehouses (LHs) are at the core of current cloud analytics stacks by providing elastic, relational compute on data in cloud data lakes across vendors. For relational semantics, they rely on open table formats (OTFs). Unfortunately, they have many missing features inherent to their metadata designs, like no support for multi-table transactions and recovery in case of an abort in concurrent, multi-query workloads. This, in turn, can lead to non-repeatable reads, stale data, and high costs in production cloud systems. In this work, we introduce LakeVilla, a modular toolbox that introduces recovery, complex transactions, and transaction isolation to state-of-the-art OTFs like Apache Iceberg and Delta Lake tables. We investigate its transactional guarantees and show it has minimal impact on performance (2% YCSB writes, 2.5% TPC-DS reads) and provides concurrency control for multiple readers and writers for arbitrary long transactions in OTFs in a non-invasive way. ∎ 1 Introduction Table 1: Existing approaches with their supported features and the respective design area Multi-Query/Table: (i) – (iii) Design Decisions System/Extension Single- query Txn Reco- very (i) Complex Txn (ii) Isola- tion (iii) Built into the OTF structure (D1) Modular solution for OTF transactions (D2) Apache Iceberg [ 61 ] Delta Lake [ 8 , 32 ] Apache Hudi [ 59 ] Project Nessie [ 6 ] XTable [ 7 ] DuckLake [ 44 ] ? LakeVilla (this paper) Provided: Limited/Planned: Unknown due to technical limitations (Section 7 ): ? Not provided: Lakehouses for analytical workloads. The past few years have seen the rise of a new type of data management system, called lakehouse (LH), which combines the benefits of low-cost, open-format data lakes and transactional data warehouses [ 22 , 35 , 38 , 76 ] . With the data gravity leading towards data lakes, LHs for analytical query processing became crucial for various tasks [ 2 , 9 , 10 , 28 , 29 , 24 , 36 , 38 , 55 , 76 ] . LHs use open table formats (OTFs) such as Delta Lake [ 68 ] , Apache Iceberg [ 61 ] , or Apache Hudi [ 60 ] to define separate metadata layers on the object store for each table [ 8 , 76 ] . Those formats implement transactions for single tables and operations, indexes, and time travel [ 8 , 76 ] , but also lack support for critical features [ 8 , 31 ] . Missing Features in OTFs. This paper addresses the problem of how to support multi-query and multi-table transactions for LHs while providing strong ACID guarantees prominently discussed in the community [ 32 , 44 ] . More specifically, we provide native support for the following three missing features: (i) Recovery , (ii) Complex Transactions , and (iii) Isolation . First, the conflict resolution of OTFs cannot track concurrent changes, often resulting in multiple aborts and higher transaction costs [ 31 ] . We refer to this feature as (i) Recovery . Second, OTFs today do not support cross-operation dependencies, thereby only allowing simple business logic of single query transactions [ 8 , 38 , 67 , 69 , 76 ] . We refer to it as (ii) Complex Transactions . Third, OTFs treat tables separately, leading to inconsistencies and dirty reads under concurrency [ 8 ] , and we refer to it as (iii) Isolation . Unfortunately, the lack of support for these features prohibits concurrent writes in a system and can not guarantee properties like referential integrity across tables for concurrent reads. As a result, one cannot execute multi-query and multi-table transactions, forcing use cases for elastic compute or similar [ 56 , 58 , 71 , 75 ] to rely on synchronization methods that block the whole LH [ 8 , 76 ] . For example, concurrent analytics could observe a reference to a bank account that does not exist yet (see our running example in Section 2 ). State of the Art. Multi-query and multi-table transaction support is a highly requested feature for LHs [ 8 , 32 , 44 , 76 ] that recently got attention from DuckDB [ 44 ] and Databricks [ 32 ] . In Table 1 , we summarize the most notable approaches for multi-query and multi-table transaction support in LHs and their specific design decisions. In summary, most systems do not adapt to concurrent changes (cf. (i); Recovery) [ 33 , 65 , 66 , 69 ] . At the same time, solutions for cross-operation dependency detection (ii) and global transaction isolation (iii) are either not present or currently in development for some systems. Despite known techniques like the SAGA pattern [ 23 ] or MVCC [ 12 , 14 , 30 , 39 , 42 ] that ensure similar features in traditional database systems, all previous approaches listed in Table 1 , add auxiliary components external to the object store but specific to a single OTF format [ 6 , 7 , 32 , 44 ] (D1). At the time of writing, only DuckLake [ 44 ] and Delta Lake 4.0 [ 32 ] support multi-query and multi-table transactions in OTFs, but they diverge significantly from OTF principles by offloading all metadata to DuckDB [ 44 ] and/or heavily modifying the OTF structure [ 32 ] . Additionally, they are facing technical limitations, making their position in Table 1 unclear (see Section 7 ). However, relying on these external services raises concerns for interoperability across vendors, potentially reenabling vendor-lock in, contradicting the original idea of LHs as an open data warehouse [ 22 , 76 ] . Hence, we aim to support multi-table and multi-query transactions without any external structures required. LakeVilla (LV) is our toolbox that implements the three features (i) recovery, (ii) complex transactions, and (iii) isolation to provide multi-table and multi-query transaction support under concurrency. LV defines a modular set of modules that are non-invasive and compatible with existing OTFs (D2). In contrast to prior work [ 6 , 7 , 8 , 44 ] , LV’s modules #1–#4.2 are integrated into the original OTF protocol and enable each transaction to choose its desired guarantees individually (D1). We ensure LV’s correctness by guaranteeing that modules result in a valid state of the raw OTF. This allows a simple integration into existing LH systems and compatibility with clients running the raw OTF. More specifically, LV composes #1–#4.2 into three modular features: First, LV[Recovery] (LV[R]) uses logical snapshot version markers (#1) with physical and logical transaction sublogs (#2) defining a greedy transaction model to support recovery (i). Second, LV[Complex Transactions] (LV[CT]) introduces a marker-based conflict and deadlock detection model (#3) among all tables to support complex transactions with an arbitrary number of queries/operations (ii). Lastly, LV[Isolation] (LV[I]) introduces a global version log (#4.1 and #4.2) that checks global relations among tables and acts as a new layer on the object store above all tables (iii). We also demonstrate how each LV feature strengthens the ACID guarantees of LHs and discuss its implications. Key insights. We show that LV guarantees serializability and linearizability for OTFs on the entire LH, not just individual tables or

Read the complete text