SOCKET: SOft Collison Kernel EsTimator for Sparse Attention

Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava · 2026

Read SOCKET: SOft Collison Kernel EsTimator for Sparse Attention by Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06283.txt.

Description

Long documents make attention, the core operation of transformer models, very expensive. Sahil Joshi and 7 co-authors present SOCKET, a method for picking which past tokens deserve attention. It reworks locality-sensitive hashing, a classic trick that buckets similar items together.

Classic hashing gives a yes-or-no collision signal, which ranks tokens poorly and needs many hash tables. SOCKET instead adds up graded collision evidence across tables, producing a soft similarity score that closely tracks the true top tokens while using far less memory. Custom chip kernels handle scoring, selection, and attention. Across long-context benchmarks and several models, SOCKET matched or beat rival sparse attention methods, with much faster processing at the top end.

Catalog record

CREATORSahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06283

Browse the full library · About the Tower of Babel