Data Structure Alchemy
Idreos, Stratos · 2025
Read Data Structure Alchemy by Idreos, Stratos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/doe-osti-2571755-data-structure-alchemy.txt.
Description
Final project report for the Data Structure Alchemy project, led by Stratos Idreos and covering 2019 to 2024, argues that the building blocks of data systems should be designed by machines. Data structures sit at the heart of databases, AI systems, and operating systems, and small inefficiencies grow into large costs at scale. The project charted the full design space of these structures, then taught software to search it.
The method maps each design's knobs, such as fan-out, node layout, and hashing family, into one large combinatorial map. Learned cost models predict speed, size, and energy for each candidate, and a two-tier optimizer picks the best fit for a workload and budget. A generator then writes the winning design as working code. Spin-offs include the Rosetta range filter, up to 130 times faster than prior art, and storage engines that design themselves, beating standard systems twenty-fold on benchmarks.
Catalog record
| CREATOR | Idreos, Stratos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| LICENSE | Public Domain |
| RIGHTS | PUBLIC DOMAIN |
| SOURCE | DOE OSTI |