SOCKET: SOft Collison Kernel EsTimator for Sparse Attention
Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava · 2026
Read SOCKET: SOft Collison Kernel EsTimator for Sparse Attention by Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06283.txt.
Description
Long documents make attention, the core operation of transformer models, very expensive. Sahil Joshi and 7 co-authors present SOCKET, a method for picking which past tokens deserve attention. It reworks locality-sensitive hashing, a classic trick that buckets similar items together.
Classic hashing gives a yes-or-no collision signal, which ranks tokens poorly and needs many hash tables. SOCKET instead adds up graded collision evidence across tables, producing a soft similarity score that closely tracks the true top tokens while using far less memory. Custom chip kernels handle scoring, selection, and attention. Across long-context benchmarks and several models, SOCKET matched or beat rival sparse attention methods, with much faster processing at the top end.
Catalog record
| CREATOR | Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger, Amar Kanakamedala, Ekam Singh, Hoang Anh Duy Le, Aditya Desai, Anshumali Shrivastava |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06283