Data Structure Alchemy

Idreos, Stratos · 2025

Read Data Structure Alchemy by Idreos, Stratos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/doe-osti-2571755-data-structure-alchemy.txt.

Description

Final project report for the Data Structure Alchemy project, led by Stratos Idreos and covering 2019 to 2024, argues that the building blocks of data systems should be designed by machines. Data structures sit at the heart of databases, AI systems, and operating systems, and small inefficiencies grow into large costs at scale. The project charted the full design space of these structures, then taught software to search it.

The method maps each design's knobs, such as fan-out, node layout, and hashing family, into one large combinatorial map. Learned cost models predict speed, size, and energy for each candidate, and a two-tier optimizer picks the best fit for a workload and budget. A generator then writes the winning design as working code. Spin-offs include the Rosetta range filter, up to 130 times faster than prior art, and storage engines that design themselves, beating standard systems twenty-fold on benchmarks.

Catalog record

CREATORIdreos, Stratos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE9 KB
LICENSEPublic Domain
RIGHTSPUBLIC DOMAIN
SOURCEDOE OSTI

Source: https://www.osti.gov/biblio/2571755

Browse the full library · About the Tower of Babel