PDX: A Data Layout for Vector Similarity Search Conference: The 2025 International Conference on Management of Data; June 22–27, 2025; Berlin, Germany

Leonardo Kuffo, Elena Krippner, Peter Boncz · 2025

Read PDX: A Data Layout for Vector Similarity Search Conference: The 2025 International Conference on Management of Data; June 22–27, 2025; Berlin, Germany by Leonardo Kuffo, Elena Krippner, Peter Boncz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04422.txt.

Description

Leonardo Kuffo, Elena Krippner, and Peter Boncz turn the storage of vectors sideways to make similarity search run faster. Vector databases store each embedding as a row, but searching row by row wastes the processor. Their 2025 paper proposes PDX, a block layout that stores dimensions vertically across many vectors, so a search can sweep one dimension of thousands of vectors at once in tight loops.

On ordinary storage, PDX beat highly optimized distance kernels by about forty percent using only plain code that the compiler vectorizes itself. Dimension-pruning algorithms, which skip most coordinates, had paradoxically been losing to brute scans on the old layout; on PDX their advantage returned, two to seven times over. A new pruning strategy, PDX-BOND, works on data as stored, with no preprocessing, suiting databases that update constantly. The paper is a reminder that layout is an algorithm.

Catalog record

CREATORLeonardo Kuffo, Elena Krippner, Peter Boncz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04422

Browse the full library · About the Tower of Babel