Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing

Miao Cao, Siming Zheng, Lishun Wang, Ziyang Chen, David Brady, Xin Yuan · 2025

Read Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing by Miao Cao, Siming Zheng, Lishun Wang, Ziyang Chen, David Brady, Xin Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08228.txt.

Introduction

Future cameras may shoot gigapixel video at a thousand frames per second, but today's sensors would melt the power budget. Miao Cao and 5 co-authors tackle that wall in a 2025 research paper. They work on video snapshot compressive imaging, which encodes many frames into one measurement at the sensor. Their ultra-sparse sampling sets exactly one sub-frame active at each pixel location.

Recovering the video from such sparse data is the hard part. The team built a real encoding system with a digital micromirror device to prove the sampling works. They then designed BSTFormer, a sparse transformer using block, global, and temporal attention. Tests on simulated and real data beat all previous methods clearly. The sampling strategy also offers a higher dynamic range than random sampling, and suits on-chip camera systems.

Catalog record

CREATORMiao Cao, Siming Zheng, Lishun Wang, Ziyang Chen, David Brady, Xin Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing Miao Cao, Siming Zheng, Lishun Wang, Ziyang Chen, David Brady, Xin Yuan Abstract Digital cameras consume ∼ 0.1 \sim 0.1 microjoule per pixel to capture and encode video, resulting in a power usage of ∼ 20 \sim 20 W for a 4K sensor operating at 30 fps. Imagining gigapixel cameras operating at 100-1000 fps, the current processing model is unsustainable. To address this, physical layer compressive measurement has been proposed to reduce power consumption per pixel by 10-100 × \times . Video Snapshot Compressive Imaging (SCI) introduces high frequency modulation in the optical sensor layer to increase effective frame rate. A commonly used sampling strategy of video SCI is Random Sampling (RS) where each mask element value is randomly set to be 0 or 1. Similarly, image inpainting (I2P) has demonstrated that images can be recovered from a fraction of the image pixels. Inspired by I2P, we propose Ultra-Sparse Sampling (USS) regime, where at each spatial location, only one sub-frame is set to 1 and all others are set to 0. We then build a Digital Micro-mirror Device (DMD) encoding system to verify the effectiveness of our USS strategy. Ideally, we can decompose the USS measurement into sub-measurements for which we can utilize I2P algorithms to recover high-speed frames. However, due to the mismatch between the DMD and CCD, the USS measurement cannot be perfectly decomposed. To this end, we propose BSTFormer , a sparse TransFormer that utilizes local Block attention, global Sparse attention, and global Temporal attention to exploit the sparsity of the USS measurement. Extensive results on both simulated and real-world data show that our method significantly outperforms all previous state-of-the-art algorithms. Additionally, an essential advantage of the USS strategy is its higher dynamic range than that of the RS strategy. Finally, from the application perspective, the USS strategy is a good choice to implement a complete video SCI system on chip due to its fixed exposure time. Code is available at https://github.com/mcao92/BSTFormer. Fig. 1: Principle of the video SCI system: a) In the encoding process, a dynamic scene occurring within a single exposure time (from t_{1} to t_{T} ) is first modulated by different masks. The modulated scene is then integrated over this time period. Finally, a low-speed camera captures the integrated scene as a snapshot compressed measurement. b) In the decoding process, the captured snapshot compressed measurement and the corresponding modulation masks are fed into a reconstruction algorithm to obtain the desired video frames. I Introduction Data bandwidth and electrical power are significant barriers to achieving high-speed, high-resolution imaging. To address this, physical layer compression and computational imaging have been proposed [ 1 ] . A key challenge in computational imaging arises from the mismatch between 2D sensor arrays and the multidimensional data volumes of objects. In this work, we focus on video Snapshot Compressive Imaging (SCI) [ 2 ] , which uses a low-speed camera to capture high-speed scenes. As illustrated in Fig. 1 , there are two steps in the video SCI system: hardware encoding and software decoding. In the hardware encoding process, the dynamic scene (occurring within a single exposure time, from t_{1} to t_{T} ) is modulated by random binary masks. These masks are composed of 0s (blocking light) and 1s (transmitting light), with Random Sampling (RS) commonly used for modulation. The modulated scene is then integrated over this period. Finally, a low-speed camera captures the integrated scene as a snapshot compressed measurement. In the software decoding stage, the captured snapshot measurement, along with the corresponding modulation masks, are fed into a reconstruction algorithm to recover the desired video frames. While one goal of video SCI is to capture high-speed effects, another important goal is to reduce bandwidth and data throughput . To achieve this, coding strategies include the following methods: a) Sampling low spatial-resolution high-speed scenes and applying image/video super-resolution algorithms to retrieve high spatial-temporal scenes. A challenge here is that we still need a high-speed camera. b) Different from case a), randomly sub-sampling a high spatial-resolution image at a high speed. Afterward, image inpainting (I2P) methods [ 3 ] are used to recover high spatial-temporal scenes. c) Both of the above cases require a high-speed camera. Inspired by I2P, we propose an Ultra-Sparse Sampling (USS) strategy for video SCI. Specifically, we can use a low-speed camera to capture a “measurement”, which is a combined sub-image at different time slots. By setting the missing pixels in each sub-frame differently, an ultra-sparse mask set can be employed, where at each spatial location, only one sub-frame is set to 1 and all others are set to 0. Ideally, we can easily decompose the measurement into sub-measurements for which we can utilize I2P algorithms to recover high-speed frames. This is also the main goal of video SCI. Video SCI uses code division multiplexing to capture multiple temporal frames in a snapshot. Dynamic range is a critical issue in such multiplexing systems. Assuming T frames normalized into [0,1] are modulated and collapsed into a single SCI measurement, the dynamic range can be [0,T] . However, the measurement can only represent the range [0,1] . Some pixels will be over-exposed and lead to degraded estimation. In contrast, if the USS strategy is used, the dynamic range of the measurement will still be [0,1] . While intuitively, we may think that USS strategy is better, there are some cases, such as dark or low-light environment where we do need a longer exposure time. Therefore, it will be the applications that determine which kind of sampling strategy should we use. This paper explores dynamic range performance using experimental data captured by our video SCI system. For practical implementation of video SCI, one must miniaturize the imaging system , ideally on a chip. One challenge to hinder this is the modulation device, such as digital micro-mirror device (DMD), commonly used in previous systems. The principle of video SCI is to randomly access each pixel at high speed and also to randomly set the exposure time for each pixel . Therefore, this modulation device is not really necessary if we can access each pixel in a high speed. However, random exposure time is difficult. Towards this end, the USS strategy provides an effective solution to implement video SCI on a chip. This paper aims to compare the above-mentioned two sampling strategies (RS and USS) in video SCI applications. More importantly, it seeks to bridge the gap between these strategies and explore how to combine them to advance the high-throughput imaging techniques. Another motivation of this paper is to move one step further to miniaturize video SCI systems. The specific contributions of this paper can be summarized as follows: • We introduce I2P into video SCI via specially designing the modulation patterns. Specifically, at each spatial location, only one sub-frame is set to 1, and all others are set to 0. • Due to the mismatch between the DMD and CCD in real-built video SCI systems, the USS measurements cannot be ideally decomposed. To address this, we propose a sparse Trans Former architecture ( BSTFormer ), which includes local B lock attention, global S parse attention, and global T emporal attention to better

Read the complete text