ParquetDB: A Lightweight Python Parquet-Based Database
Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero · 2025
Read ParquetDB: A Lightweight Python Parquet-Based Database by Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05311.txt.
Introduction
Logan Lang and three co-authors present a lightweight database built on the Parquet file format, in a 2025 paper. Standard databases add friction to fast-moving research code. Their Python tool stores data in columnar files instead. Filtering pushes down into the file reads, reducing index needs. Nested and complex data types are supported natively. Portability comes free with file-based storage.
Benchmarks beat two common databases on large data volumes. Arrow-compatible formats showed the biggest wins. A real test used a materials database with millions of records. About four point eight million complex entries were handled smoothly. The records describe three-dimensional material structures. The authors position the tool for data-driven science teams. Rapid iteration, not enterprise features, is the design goal. Updates continue as user needs evolve.
Catalog record
| CREATOR | Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo H. Romero |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 134 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05311