A Benchmark for Databases with Varying Value Lengths

Danushka Liyanage, Shubham Pandey, Joshua Goldstein, Michael Cahill, Akon Dey, Alan Fekete, Uwe Röhm · 2025

Read A Benchmark for Databases with Varying Value Lengths by Danushka Liyanage, Shubham Pandey, Joshua Goldstein, Michael Cahill, Akon Dey, Alan Fekete, Uwe Röhm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07551.txt.

Introduction

Danushka Liyanage and co-authors build a database test that lets record sizes grow, in a 2025 paper. Standard tests use near fixed record lengths, yet real stores in key value, document, and graph systems hold values of wildly unlike size. Big popular records can slow a system in ways fixed tests miss.

The team added an extend step to a known cloud benchmark, which appends data to a record to mimic growth. They ran three database backends through rounds of growth and queries. Results split sharply by storage engine design and how each handles varied sizes. The authors urge tests that mirror how real data changes over time.

Catalog record

CREATORDanushka Liyanage, Shubham Pandey, Joshua Goldstein, Michael Cahill, Akon Dey, Alan Fekete, Uwe Röhm
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07551

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Benchmark for Databases with Varying Value Lengths Danushka Liyanage, Shubham Pandey, Joshua Goldstein, Michael Cahill, Akon Dey, Alan Fekete, Uwe Röhm Abstract The performance of database management systems (DBMS) is traditionally evaluated using benchmarks that focus on workloads with (almost) fixed record lengths. However, some real-world workloads in key/value stores, document databases, and graph databases exhibit significant variability in value lengths, which can lead to performance anomalies, particularly when popular records grow disproportionately large. Existing benchmarks fail to account for this variability, leaving an important aspect of DBMS behavior underexplored. In this paper, we address this gap by extending the Yahoo! Cloud Serving Benchmark (YCSB) to include an “extend” operation, which appends data to record fields, simulating the growth of values over time. Using this modified benchmark, we have measured the performance of three popular DBMS backends: MongoDB, MariaDB with the InnoDB storage engine, and MariaDB with the MyRocks storage engine. Our experiments alternate between extending values and executing query workloads, revealing significant performance differences driven by storage engine design and their handling of variable-sized values. Our key contribution is the introduction of a novel benchmarking approach to evaluate the impact of growing value sizes and isolate the effect of querying data with a distribution of data sizes from any cost associated with accessing data after a history of updates. This highlights the need for more representative benchmarks that capture the dynamic nature of real-world workloads, providing valuable guidance for both practitioners and researchers. 1 Introduction Benchmarks, such as the TPC suite, have served the database field well; they have helped practitioners with system and configuration choice, and have been a spur for researchers to improve platform design. In this paper, we consider a gap in prior work on benchmarking: the neglect of cases where some data items (records in tables, documents in a document store, values in a key value store) get longer and longer as the system evolves, eventually becoming quite long (say megabyte size). This can happen particularly when an item has an attribute whose type is array and where the application uses this array to store information about the item’s history. In that case, some operations of the application append extra information to that attribute; if the item is a popular one and the database lasts a long time, the item may grow very long. This situation has not been well covered in previous benchmarks. Some (e.g., YCSB) have fixed size for all items; while TPC-C does include some variation in record size (due to VARCHAR fields such as C_DATA), this variation is limited, with no record longer than 1KB. The existence in real applications of items that grow in size over time can impact the performance of the system, depending on choices made in the implementation of the storage layer. These situations can cause fragmentation of the data on disk or in memory, or may lead to frequent re-arrangements if the storage uses an update-in-place approach. Some systems store deltas to optimize the case where only a small part of an item is modified in an update. If these design aspects are not exercised in a benchmark, we will not know how well the choices work in practice. Thus, the focus of this paper is to propose a benchmark that examines what happens when items have lengths that increase significantly over time. When the data includes some items that are growing steadily, there are several consequential changes to the data, and we aim to disentangle the performance impact of these. One impact of some records growing longer through history is that at any given time, the data items in the database will vary in length among themselves. Another consequence is that the total volume of data in the database will grow. YCSB-IVS is designed to measure performance with a database where some items keep growing, but we also compare this with performance when the item sizes vary, but not as a result of a history of growth. Also, we look at the performance of two baselines with the same total volume and items of uniform size: one baseline has many small records, and the other has fewer records, which are all equally long. We propose a benchmark called YCSB-IVS which builds on the framework provided by Yahoo! Cloud Serving Benchmark (YCSB), a widely used benchmarking tool originally for key/value stores, but also with bindings for a wide variety of platforms including SQL ones. While the original YCSB workload has fixed size items (by default, 1000 bytes exactly), which can be read, updated, etc., we introduce an “extend” operation that appends data to one or more fields of a record, causing the length to grow over time. In contrast to the original YCSB, where data is loaded initially, and then a single measured run executes a mix of read, update, etc. operations, YCSB-IVS loads data, and then proceeds in a sequence of epochs, each of which performs some number of extend operations applied to keys randomly chosen according to some distribution, and then we measure the metrics (throughput, latency, etc.) for standard YCSB workloads against the state at the end of the extension phase. As we run successive epochs, some of the data items become longer and longer, and the standard measurements show the slowdown that this causes. For comparison, we also measure for each epoch the read performance on a database with the same logical state (and thus the same distribution of item lengths) but no history of increase. We report the performance metrics for each epoch, and also report on two baselines which execute the queries against two datasets with equal-sized items and the same total volume as the given epoch; one baseline has the same number of items as in our benchmark (and so each item has length, which is the average length found after extensions). In the other baseline, all records have the unextended length, and so there are many more items than in the benchmark execution. We have applied YCSB-IVS to measure the performance of several popular DBMS platforms – MongoDB, MariaDB with the InnoDB storage engine, and MariaDB with the MyRocks storage engine. Here we show a few examples of the results; full details are in our YCSB-IVS Tech Report . Our findings reveal significant performance differences between systems, driven by their storage engines and handling of large, variable-sized values. For example, some systems demonstrate robust performance even as values grow, while others exhibit surprising slowdowns. These results underscore the importance of incorporating varying value sizes into benchmarking methodologies to uncover hidden performance characteristics and better inform DBMS design and optimization. In Section 2 we discuss in more detail some real-world cases where items grow in length substantially over time. Section 3 provides a critical analysis of prior work relevant to this research domain. In Section 4 we describe the detailed operation of our new benchmark. In Section 5 we illustrate its use by reporting some measurements. Our key contribution is to propose a novel benchmarking approach for databases/key-value stores. We extend YCSB to simulate realistic scenarios where value sizes grow over time, reflecting issues that occur in real-world workload patterns but have not been effectively benchmarked before. Through

Read the complete text