Exploring Spatiotemporal Feature Propagation for Video-Level Compressive Spectral Reconstruction: Dataset, Model and Benchmark Lijing Cai, Zhan Shi, Chenglong Huang, Jinyao Wu Abstract Recently, Spectral Compressive Imaging (SCI) has achieved remarkable success, unlocking significant potential for dynamic spectral vision. However, existing reconstruction methods, primarily image-based, suffer from two limitations: (i) Encoding process masks spatial-spectral features, leading to uncertainty in reconstructing missing information from single compressed measurements, and (ii) The frame-by-frame reconstruction paradigm fails to ensure temporal consistency, which is crucial in the video perception. To address these challenges, this paper seeks to advance spectral reconstruction from the image level to the video level, leveraging the complementary features and temporal continuity across adjacent frames in dynamic scenes. Initially, we construct the first high-quality dynamic hyperspectral image dataset (DynaSpec), comprising 30 sequences obtained through frame-scanning acquisition. Subsequently, we propose the Propagation-Guided Spectral Video Reconstruction Transformer (PG-SVRT), which employs a spatial-then-temporal attention to effectively reconstruct spectral features from abundant video information, while using a bridged token to reduce computational complexity. Finally, we conduct simulation experiments to assess the performance of four SCI systems, and construct a DD-CASSI prototype for real-world data collection and benchmarking. Extensive experiments demonstrate that PG-SVRT achieves superior performance in reconstruction quality, spectral fidelity, and temporal consistency, while maintaining minimal FLOPs. Project page: https://github.com/nju-cite/DynaSpec . 1 Introduction Compared to RGB images, Hyperspectral Images (HSIs) offer a unique capability to detect spectral properties between various materials 27 , making them highly promising for classification 20 ; 49 ; 31 , detection 25 ; 39 , tracking 55 ; 14 ; 32 ; 57 , and autonomous driving 2 ; 46 , etc. Traditional hyperspectral imaging systems typically require scanning along either the spatial or spectral dimension, which limits their applicability in dynamic scenes. To overcome this limitation, Spectral Compressive Imaging (SCI) 12 ; 1 has gained significant attention in recent years. As shown in Fig. 1 (a), SCI employs spatial-spectral encoding to compress 3D data into a 2D measurement, enabling snapshot acquisition. Reconstruction algorithms then leverage sparsity priors to recover the spatial-spectral information. Despite its merits in bandwidth efficiency and acquisition speed, SCI faces two fundamental limitations, as displayed in Fig. 1 (b): (i) Mask-based encoding inevitably leads to spatial-spectral information loss, making the reconstruction of occluded content inherently uncertain; (ii) Frame-by-frame reconstruction is prone to temporal isolation—manifested as poor temporal continuity. Figure 1 : Spectral compressive imaging and reconstruction. (a) SCI principle. (b) Image-based methods, with issues of uncertain reconstruction and temporal inconsistency (flickering intensity curves). (c) Video-based reconstruction, where information complementarity enhances completeness and temporal consistency (smooth intensity curves). Fortunately, in the era of video perception, the relevant information from temporal measurement sequences offers a promising solution. As shown in Fig. 1 (c), a fixed encoding pattern has the potential to differentially capture complementary features across adjacent frames, thereby improving the propagation reconstruction of masked information and enhancing temporal coherence. In light of this, achieving video-level HSIs reconstruction from a sequence of compressed measurements emerges as a compelling yet challenging research problem, primarily due to two key obstacles: (i) Data scarcity is a fundamental bottleneck. Existing datasets are primarily collected for image-level reconstruction 58 ; 17 . While slicing images to generate pseudo-sequences is an optional workaround 42 , such data fails to exhibit high degrees of freedom in motion. In addition, some video-level datasets developed for downstream tasks 55 suffer from limited spectral resolution and data reliability, rendering them unsuitable as ground truth for reconstruction-oriented tasks. (ii) Existing algorithms have limited capacity for spatiotemporal modeling. HSIs reconstruction methods can be broadly categorized into model-based 3 ; 5 ; 62 ; 34 and learning-based approaches 27 ; 36 ; 38 ; 35 . Model-based methods require extensive parameter tuning, making it difficult to achieve stable and high-quality reconstruction. Learning-based methods have recently achieved SOTA performance; however, their high computational cost and limited ability to capture spatiotemporal dependencies pose significant obstacles to video-level reconstruction. As a first step toward addressing these gaps, we construct a high-quality dynamic hyperspectral image dataset, named DynaSpec. Each frame is captured individually using a push-broom hyperspectral camera, covering the 400–700 nm spectral range with a spectral resolution of 2 nm. Diverse motions are then manually introduced to emulate the high degrees of freedom encountered in real-world scenarios. The dataset comprises 30 video sequences (totaling 300~\text{HSIs} ), facilitating the exploration of video-level reconstruction and downstream tasks. Furthermore, we propose a video-level compressive spectral reconstruction algorithm, PG-SVRT, which consists of three key components: Mask-Guided Degradation Perception (MGDP), Cross-Domain Propagated Attention (CDPA), and Multi-Domain Feed-Forward Network (MDFFN). MGDP models the degradation process to aid in decoupling intra-frame encoded information. CDPA facilitates progressive cross-domain feature propagation via spatial-then-temporal attention. Inspired by linear attention 28 ; 22 ; 24 ; 23 , we introduce bridged tokens to reduce computational complexity while maintaining high-quality spatiotemporal feature extraction. Additionally, MDFFN allows for the independent extraction of spatial and temporal features while promoting their effective integration. Finally, we conduct comparative simulation experiments across four SCI systems 48 ; 21 ; 11 ; 13 . The results show that DD-CASSI, benefiting from its high spectral sampling efficiency and clear structural representation, exhibits significant superiority in video-level reconstruction. Building on this insight, we construct a prototype to capture real-world measurements for validation and benchmarking. In summary, our contributions are as follows: • We construct the DynaSpec dataset to address the scarcity of high-quality dynamic HSIs data. • We propose a novel method, PG-SVRT, for efficient video-level compressive spectral reconstruction, which achieves over 41 dB PSNR and minimal computational cost, without additional hardware modifications. • We conduct comparative simulations across representative SCI systems, and construct a lab prototype for real-world imaging. Simulations and experiments demonstrate that our method achieves superior performance. 2 Related Work Compressive Spectral Reconstruction. Model-based approaches typically rely on handcrafted priors such as sparsity 45 , low-rankness 34 , and total variation 5 ; 62 , which struggled with high-dimensional data and complex degradations. Recently, learning-based methods have emerged, including end-to-end models 8 ; 9 ; 56 ; 50 ; 26 , plug-and-play (PnP) methods 60 ; 61 ; 40 , and deep unfolding networks (DUNs) 10 ; 37 ; 63 ; 30 ; 18 . PnP methods embed pre-trained denoisers into the optimization process, but the lack of joint training limits their flexibility 53 . DUNs offers interpretability by mimicking iterative solvers with learnable modules, yet suffers from high computational cost 50 . These methods only consider the reconstruction of single-frame measurements, while video-level spectral reconstruction remains largely unexplored. In this work, we extend the learning paradigm to the video domain, enabling efficient modeling of spatiotemporal dependencies. Figure 2 : The proposed DynaSpec dataset. (a) Dynamic HSIs sequences acquired frame by frame to simulate the diverse motion of real-world scenarios. (b) A display of the 30 scenes. Spectral Compressive Imaging. SD-CASSI 48 and DD-CASSI 21 employ random binary masks that satisfy compressed sensing requirements but suffer from complicated reconstruction due to severe spectral aliasing. 12 . PMVIS 11 employs spatially sparse sampling to suppress interference, thereby reducing the ill-posedness of inversion, albeit at the cost of spatial resolution and light throughput. In contrast, NDSSI 13 uses a notch-coded mask to maximize optical throughput and incorporates dual-disperser architecture 52 to maintain the fidelity of spatial structures, although its spectral sampling density remains limited. Each system has its strengths and limitations; therefore, we conduct simulation-based evaluations to compare their performance and identify the architecture that delivers superior results for video-level spectral reconstruction. 3 Spectral Compressive Imaging Model In this section, we present a unified mathematical formulation of compressive spectral imaging. • Single-Disperser Architecture (SD). As illustrated in the top portion of Fig. 1 (a), SD systems such as SD-CASSI and PMVIS first modulate the incident light using a coded mask, followed by spectral shearing through a dispersive element. Let X_{i}\in\mathbb{R}^{H\times W\times C} denote the transient 3D spectral data, where H and W are spatial dimensions and C is the number of spectral channels. The measurement at spatial location (h,w) and time i is modeled as: Y_{i}(h,w)=\sum_{c=1}^{C}\Phi(h,w)\cdot X_{i}(h,w-\sigma(c),c) (1) where c indexes the spectral coordinate, \Phi denotes the mask modulation function, and \sigma(\cdot) represents the dispersion function introduced by the disperser. • Dual-Disperser Architecture (DD). As shown in the lower part of Fig. 1 (a), in dual-disperser systems (e.g., DD-CASSI and NDSSI), the incoming light is first dispersed, then modulated by a coded mask in the spectrally sheared domain, and finally recombined by a symmetric disperser to reverse the spectral shift. We model the measurement at spatial location (h,w) and time i as follows: Y_{i}(h,w)=\sum_{c=1}^{C}\Phi(h,w-\sigma(c))\cdot X_{i}(h,w,c) (2) For convenience, the imaging processes of both architectures are unified and formulated as: Y_{i}=\Psi X_{i}+\Theta (3) where Y_{i}\in\mathbb{R}^{H\times W^{\prime}} denotes the measurement, \Psi:\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{H\times W^{\prime}} denotes the overall encoding operator (e.g., spectral dispersion and mask modulation), and \Theta represents noise. In SD systems, due to spectral shearing, the measurement width is W^{\prime}=W+\sigma(C{-}1) . In contrast, DD systems perform symmetric inverse dispersion, restoring the spatial dimension such that W^{\prime}=W . Since both architectures can be reformulated under a unified framework, prior SD-based reconstruction methods remain valid and applicable in the context of DD systems. Building upon this, our work is devoted to extending image-level reconstruction to the video level, aiming to recover the spectral video X\in\mathbb{R}^{T\times H\times W\times C} from the compressed measurement sequence Y\in\mathbb{R}^{T\times H\times W^{\prime}} . 4 DynaSpec dataset Current datasets for HSIs reconstruction are mainly image-based 58 ; 17 . Although pseudo-video sequences can be synthesized by cropping images 42 , such methods only mimic rigid motions, such as camera motion, leaving real-world in-scene dynamics unaccounted for. Some downstream tasks have started exploring spectral data at the video level 55 ; 2 ; 59 ; however, due to task-specific considerations, the datasets used in these works often suffer from limited spectral resolution and reduced data fidelity, making them unreliable as ground truth for reconstruction tasks. Given this, it is imperative to construct a dataset consisting of high-quality spectral sequences featuring dynamic scenes. Figure 3 : Illustration of PG-SVRT. (a) and (c) The components of MGDP and CDBP. (b) PG-SVRT framework and key components. Considering the inherent challenges of acquiring ground truth HSIs towards dynamic scenes, we employ the GaiaField push-broom hyperspectral camera 19 to capture controllable objects frame-by-frame. By manually designing actions, we simulate diverse and complex motions, such as translation, rotation, and articulated movements, as shown in Fig. 2 (a). To ensure the authenticity and reliability of the acquired data, we adhere to the following principles: (1) Object motion between consecutive frames is continuous and adheres to physical laws. (2) Long integration times are used to mitigate noise interference. (3) Spectral correction is applied based on the camera’s spectral response. (4) The spectral properties of the illumination are excluded to ensure the data approximates reflectance values, preventing the network from fitting illumination-specific information. (5) Intensity calibration is performed using invariant objects within the sequence, minimizing the impact of temperature drift introduced by prolonged system operation. As illustrated in Fig. 2 (b), the DynaSpec dataset encompasses 30 scenes, totaling 300 HSIs. Each frame’s data cube features a spatial resolution of 1280\times 1280 pixels, a spectral resolution of 2~nm , and a wavelength range from 400~nm to 700~nm (151 spectral channels). The dataset’s continuity of motion, coupled with its high spectral and spatial resolution, provides a robust foundation for the reconstruction task. 5 Propagation-Guided Spectral Video Reconstruction Transformer (PG-SVRT) To efficiently extract redundant spatiotemporal features for video-level spectral reconstruction, we present PG-SVRT, as illustrated in Fig. 3 (b). It utilizes the U-Net-based architecture, primarily composed of the MGDP module and the Cross-Domain Propagated Block (CDPB). The shuffle operation aligns degradation features with measurements across the spectral dimension. The CDPB consists of the CDPA and MDFFN, as shown in Fig. 3 . 5.1 Mask-Guide Degradation Perception (MGDP) Since all measurement sequences follow the same optical encoding paradigm, the degradation prior is essential for reconstruction. Inspired by the widespread use of mask-based degradation learning in SOTA methods 9 ; 50 ; 63 ; 18 , we construct the MGDP to perceive the compression degradation process before feeding it into the main architecture. The mask matrix \Phi\in\mathbb{R}^{H\times W\times C} in this section represents the mask pattern corresponding to the same spatial region across all spectral channels. According to Sec. 3 , for the SD architecture, \Phi for each channel is \Phi(m,n) ; for the DD architecture, the mask for each channel is given by \Phi(m,n-\sigma(c)) . As shown in Fig. 3 (a), we first compress \Phi along the spectral dimension according to the SD or DD architecture to obtain \Phi_{s}\in\mathbb{R}^{H\times W^{\prime}} , and then crop or replicate \Phi_{s} to form the \Phi_{p}\in\mathbb{R}^{H\times W\times C} . \Phi_{p} represents the spatial intensity distribution of each channel after degradation. The intensity distribution difference between \Phi and \Phi_{p} is learned through a Conv_{1\times 1} , with the sigmoid function used to compute the weight distribution W_{\Phi} , which perceives the degradation features at different spectral and spatial positions. Similar operations are then applied to the measurement sequence Y , and element-wise feature weighting is performed by dot-multiplying with W_{\Phi} . Finally, a Conv_{1\times 1\times 1} convolution is used to extract the mask-guided degradation perception features, which are then concatenated with the measurements along the channels and serve as the input to the main architecture, denoted as Y_{in} , expressed as: Y_{in}=Concat(Conv(W_{m}(\Phi,\Phi_{p})\odot F_{m}(Y)),Y) (4) Figure 4 : Details of the CDPB, which consists primarily of CDPA and MDFFN. (a) CDPA is a spatial-then-temporal attention mechanism, where the blue line represents spatial feature processing and the red line indicates temporal feature processing. (b) Illustration of MDFFN. 5.2 Cross-Domain Propagated Attention (CDPA) Spatiotemporal feature extraction 41 ; 7 is commonly used in video-level tasks. However, our task also demands the reconstruction of high-dimensional spectral data, posing a critical challenge in designing low-complexity attention mechanisms to ensure computational efficiency. We consider the following two aspects: (1) Mainstream video attention mechanisms, such as G-MSA 16 and F-MSA 4 , suffer from high computational complexity, while W-MSA 33 can reduce the complexity to approximately linear without compromising feature extraction capability. However, spectral reconstruction task involves high-dimensional features ( C ), and overly small spatial windows ( H_{win}W_{win} ) are unable to fully capture the dispersion features, creating a bottleneck in complexity reduction. (2) For multi-dimensional feature processing, joint extraction and separated extraction are the two main approaches 4 . Joint extraction is resource-intensive 4 , while fully discrete processing of different dimensions will restrict the ability for feature interaction. Given this, we design the CDPA, as shown in Fig. 4 (a). CDPA is a spatial-then-temporal progressive attention that uses a shared value to propagate features across different domains, alleviating the issue of discrete multi-domain feature interactions. Additionally, inspired by linear attention mechanisms 23 , we introduce a bridged token to further reduce computational complexity. To simplify the description, we use the CDPA from the first CDPB as an example. The query Q_{N1} , key K_{N1} , and value V_{N1} are computed from the input feature Y_{N1}\in\mathbb{R}^{T\times H\times W\times C} as follows: \small Q_{N1}=Y_{N1}W_{q},\hskip 9.24994ptK_{N1}=Y_{N1}W_{k},\hskip 9.24994ptV_{N1}=Y_{N1}W_{v}, (5) where Q_{N1},K_{N1},V_{N1}\in\mathbb{R}^{T\times H\times W\times C} , and W_{q,k,v}\in\mathbb{R}^{C\times C} are learnable projection matrices. In spatial information processing, we divide the input into non-overlapping blocks Q_{s},K_{s},V_{s}\in\mathbb{R}^{Thw\times H_{win}W_{win}\times C} , where h={H}/{H_{win}},w={H}/{W_{win}} , and H_{win} , W_{win} denote the window size. Additionally, we introduce a bridged token B_{s}\in\mathbb{R}^{Thw\times N_{B}\times C} , where N_{B} denotes the number of tokens. This token serves as a bridge, enabling indirect interaction between Q_{s} , K_{s} , and V_{s} . Although many advanced methods can effectively generate new tokens 54 ; 6 , since B_{s} essentially represents the information of Q_{s} , we directly pool Q_{s} to generate B_{s} , avoiding additional computational cost. The final spatial attention can be expressed as: \small Y_{s}^{out}=GConv\left(A(Q_{s},B_{s},A(B_{s},K_{s},V_{s},\tau_{1}),\tau_{2})\right)+Y_{N1} (6) where A(Q,K,V,\tau) represents Softmax\left({QK^{T}}/{\tau}\right)V , with \tau being learnable parameters 63 . In the temporal information processing, we rearrange Q_{N1},K_{N1} , and Y_{s}^{out} into Q_{t},K_{t},Y_{t}\in\mathbb{R}^{HW\times T\times C} for temporal attention computation, which not only reduces the resource consumption caused by additional projections but also enhances feature propagation across different domain through the shared value, directly derived from Y_{s}^{out} . This computation process is expressed as: Y_{t}^{out}=A(Q_{t},K_{t},Y_{t},\tau_{3}) (7) Based on the above process, the computational complexity of CDPA is displayed as follows: \small O(CDPA)=4THWC^{2}+4THWN_{B}C+2T^{2}HWC (8) When 2N_{B}