Create Benchmarks for Data Lakes
Yi Lyu, Pei-Chieh Lo, Natan Lidukhover · 2026
Read Create Benchmarks for Data Lakes by Yi Lyu, Pei-Chieh Lo, Natan Lidukhover (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19176.txt.
Introduction
Yi Lyu, Pei-Chieh Lo, and Natan Lidukhover build a benchmark for data lakes. Data lakes store vast mixed data for later use, and their use keeps growing. Yet few standard benchmarks exist to grade them, so claims about speed and cost rest on uneven tests.
A shared benchmark would give a fair, comparative view of rival systems. Buyers could see which lake design fits their loads, and builders could prove gains on common ground. The team sets out the problem and their plan for filling that gap in how these systems are judged.
Catalog record
| CREATOR | Yi Lyu, Pei-Chieh Lo, Natan Lidukhover |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Create Benchmarks for Data Lakes Yi Lyu, Pei-Chieh Lo, Natan Lidukhover Abstract 1 Introduction The problem that we are trying to solve is to build a new benchmark for the data lakes. Although there is an increasing need for data lakes these days, there are not many standardized benchmarks for data lakes. By doing that, we are hoping to get a more objective and comparative understanding of different data lake implementations. There are many users requiring a database system that offers more flexibility on storage and analytics than traditional data warehouses [ 7 ] [ 17 ] [ 4 ] [ 3 ] [ 18 ] [ 23 ] [ 24 , 25 , 11 , 12 , 10 , 15 , 19 , 20 ] . A data lake is a large repository for storing and analyzing data without the restriction on the data structure, type, and size. It is an increasingly popular way to store data because it allows users to manipulate different data types from different sources in a centralized repository. Therefore, having standard benchmarks for data lake systems can assist in the process of developing better data lakes that take performance metrics into account. The data lake market was valued at $3.74 billion in 2020 [ 8 ] . By 2026, this market is expected to reach $17.6 billion, meaning the compound annual growth rate of this market over the period from 2021 to 2026 is 29.9%. This is a rapidly growing market, and according to the O’Reilly Data Scientist Salary Survey, the speed of data retrieval is one reason why data lakes are preferred and growing more when compared to a traditional data warehouse. Given that speed is vital for data lakes, it is increasingly important to have a benchmarking suite that is capable of determining the performance of a given data lake implementation to see which ones are faster and more performant. The top data lakes in the market are owned by the biggest cloud computing platforms: Amazon Web Services, Microsoft Azure, Google Cloud Platform, and Oracle Cloud Infrastructure [ 13 ] . It will be useful to compare at least some of these to one another in terms of performance and will provide some immediate use for our benchmark. Furthermore, as mentioned in Related Work, DLBench is currently a data lake benchmark that claims to be the first one, and it ran its testing on an open-source data lake called AUDAL. That would be another useful data lake to compare to as a baseline for how our suite compares to DLBench. Currently, there has not been a lot of work on data lake benchmarks. Existing benchmarks for other big data systems cannot be directly applied here. The traditional benchmarks such as the TPC standard specification [ 5 ] only provide SQL workloads and only work for structured data stores. However, data lakes support structured data, unstructured data, and semi-structured data. Also, there is one operation that is seldom tested in existing benchmarks: finding similarity. This is a common operation in data lakes, so a benchmark should also do performance testing for this operation. Because data lakes can accept many types of structures for data, it is important to understand the differences between structured, semi-structured, and unstructured data. Structured data has a pre-defined structure and is formatted that way before being placed in storage. An example of this is tabular data inputted into a relational database such as SQL. SQL data follows a set schema and is therefore structured. Unstructured data, on the other hand, is not processed until it is taken out of the data store. An example of this is raw text data or video files. Semi-structured data is a bit more confusing to understand. It is a middle state between raw data and strictly typed fields such as those in a SQL schema. An HTML file is an example of this type of data, as tags and anchors provide a rigid structure that can be fed into a parser and have the explicit structure extracted; however, many HTML pages contain text and images, and other raw data that cannot be parsed out. For these reasons, it is not entirely processed on input into a data store nor on reading out, but rather both times, and it is semi-structured data [ 1 ] . We plan to have more comprehensive benchmarks for data lakes by (1) extending the target data structure type to structured, semi-structured, and unstructured, (2) measuring time for different workload model examples including data retrieval, data aggregation, and data queries, and (3) observing the performance with a variety of data access operations including query execution time, metadata generation time, and metadata size. The approach we are planning to take is more inclusive and will take more aspects into account in order to figure out what will be the most critical part for data lake systems in storing data and accessing it. The expected results will contain scripts to generate data with different scale factors which others can run on their own to get the performance metrics from a given data lake implementation. We would also run this script on our own to get the measuring number for each metric under different types of structured data. We would like to get some insights from the statistics and highlight our new findings for data lakes benchmarks. We will use CloudLab for our experiments. 2 Related Work Currently, there are still no standardized benchmarks to benchmark the performance of data lakes. TPC-H [ 5 ] and TPC-DS [ 6 ] issued by the Transaction Processing Performance Council are still widely adopted by the big data system community. BigBench [ 14 ] is also another popular benchmark that executes SQL queries to evaluate the performance of data systems. In addition to only structured data, it also includes more complex big data analysis tasks—namely sentiment analysis over short texts. All the benchmarks mentioned above are not a great fit to benchmark data lakes. For TPC-H [ 5 ] and TPC-DS [ 6 ] , they are only for traditional databases. That means they only have SQL workloads. Also, they only do testing for structured data. But data lakes store structured data, unstructured data, and semi-structured data at the same time. These are not enough to test all functionalities of data lakes. AdBench is a benchmark for modern end-to-end data pipelines that integrate with data lakes [ 2 ] . AdBench notes that with the emergence and increasing utilization of Apache Hadoop Distributed File System for data lake implementation and construction, data pipelines are being built on top of this platform as well. AdBench has expanded on previous standalone task benchmarks such as ones for Analytical SQL queries. AdBench specifically runs on ad-serving and streaming data, but the paper notes the characteristics of these workloads are analogous to those in Internet of Things (IoT) infrastructure, healthcare, and financial services, meaning the concept can be extended to those industries if needed. AdBench does not directly serve as a data lake benchmark, but it provides a useful set of metrics that compares across scale factors, just as we wish to do, in a closely related area to