ParquetDB: A Lightweight Python Parquet-Based Database

Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero · 2025

Read ParquetDB: A Lightweight Python Parquet-Based Database by Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05311.txt.

Introduction

Logan Lang and three co-authors present a lightweight database built on the Parquet file format, in a 2025 paper. Standard databases add friction to fast-moving research code. Their Python tool stores data in columnar files instead. Filtering pushes down into the file reads, reducing index needs. Nested and complex data types are supported natively. Portability comes free with file-based storage.

Benchmarks beat two common databases on large data volumes. Arrow-compatible formats showed the biggest wins. A real test used a materials database with millions of records. About four point eight million complex entries were handled smoothly. The records describe three-dimensional material structures. The authors position the tool for data-driven science teams. Rapid iteration, not enterprise features, is the design goal. Updates continue as user needs evolve.

Catalog record

CREATORLogan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE134 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05311

Browse the full library · About the Tower of Babel