SPKLIP: Aligning Spike Video Streams with Natural Language

Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen · 2025

Read SPKLIP: Aligning Spike Video Streams with Natural Language by Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12656.txt.

Description

Spike cameras report only sudden brightness changes, as a sparse rain of events, which makes their footage hard to pair with words. Yongchang Gao and co-authors introduce SPKLIP, the first system built to align spike video with natural language. A layered extractor reads the event stream at several time scales at once, and contrastive training pulls matching video and text together in a shared space.

The alignment supports learning from few examples, handy where labelled spike data is scarce. A fully spiking version of the visual encoder runs leaner still, suiting brain-inspired hardware. Tests set state-of-the-art marks on existing spike benchmarks and generalised well to a new real-world dataset the team contributed. Code and data are public.

Catalog record

CREATORYongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12656

Browse the full library · About the Tower of Babel