Diffusion Autoencoders with Perceivers for Long, Irregular and Multimodal Astronomical Sequences

Yunyi Shen, Alexander Gagliano · 2025

Read Diffusion Autoencoders with Perceivers for Long, Irregular and Multimodal Astronomical Sequences by Yunyi Shen, Alexander Gagliano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20595.txt.

Introduction

Yunyi Shen and Alexander Gagliano build a diffusion autoencoder with Perceivers for star data that come as long, lopsided, mixed sets. Self-taught nets are most tried on even sets like stills or clips. Star data break that mold. Their net splits mixed reads to tokens, packs them with a Perceiver coder, and rebuilds them with a Perceiver-IO diffusion decoder.

A mate base net, a masked autoencoder in the same Perceiver frame, sets a strong bar. On both light-curve and spectra star sets, the diffusion net posts less rebuild miss, more split latent codes, and more fine grain kept than the base net and a VAE. The frame aims at fields with lopsided, mixed streams.

Catalog record

CREATORYunyi Shen, Alexander Gagliano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusion Autoencoders with Perceivers for Long, Irregular and Multimodal Astronomical Sequences Yunyi Shen, Alexander Gagliano Abstract Self-supervised learning has become a central strategy for representation learning, but the majority of architectures used for encoding data have only been validated on regularly-sampled inputs such as images, audios. and videos. In many scientific domains, data instead arrive as long, irregular, and multimodal sequences. To extract semantic information from these data, we introduce the Diffusion Autoencoder with Perceivers (daep). daep tokenizes heterogeneous measurements, compresses them with a Perceiver encoder, and reconstructs them with a Perceiver-IO diffusion decoder, enabling scalable learning in diverse data settings. To benchmark the daep architecture, we adapt the masked autoencoder to a Perceiver encoder/decoder design, and establish a strong baseline (maep) in the same architectural family as daep. Across diverse spectroscopic and photometric astronomical datasets, daep achieves lower reconstruction errors, produces more discriminative latent spaces, and better preserves fine-scale structure than both VAE and maep baselines. These results establish daep as an effective framework for scientific domains where data arrives as irregular, heterogeneous sequences. 1 Introduction Self-supervised learning (SSL) has emerged as a powerful paradigm for representation learning, driving major advances in language, vision, and audio domains ( Jing and Tian, 2020 ) . In the canonical framing of SSL techniques, however, data are defined on regular grids such as pixels in images or fixed-rate samples in audio and video. In many scientific and real-world applications, data instead arrive as long sequences sampled irregularly in time. Biomedical records contain patient measurements obtained at uneven intervals ( Krishnan et al., 2022 ) , sensor networks report readings based on event triggers or battery constraints, and in astrophysics, photometry is obtained with uneven temporal sampling due to observational constraints. Developing SSL methods that can natively handle these irregular, multimodal inputs is an important and open challenge across domains. Astronomical surveys provide a particularly valuable context for investigating SSL. Spectroscopic surveys like the Sloan Digital Sky Survey ( York et al., 2000 ) and the Dark Energy Spectroscopic Instrument Survey ( Abareshi et al., 2022 ) produce spectra for millions of sources, while time-domain imaging surveys like the Zwicky Transient Facility Bright Transient Survey ( Bellm et al., 2018 ) , the Asteroid Terrestrial-impact Last Alert System ( Tonry et al., 2018 ) , and the Young Supernova Experiment ( Jones et al., 2021 ) produce similar volumes of irregularly-sampled photometry. SSL methods that combine these data modalities are now beginning to emerge ( Zhang et al., 2024 ; Rizhko and Bloom, 2025 ) . Reconstruction-based methods have become a popular SSL paradigm, following their success in natural language processing and vision ( Gui et al., 2024 ; Chen et al., 2021 ; Rani et al., 2023 ) . Diffusion models achieve state-of-the-art sampling in the image domain ( Ho et al., 2020 ; Dhariwal and Nichol, 2021 ) , but operate directly in pixel space without learning compressed representations ( Kwon et al., 2022 ) . Diffusion autoencoders ( Preechakul et al., 2022 ) address this by coupling a deterministic encoder with a diffusion decoder, simultaneously enabling both representation learning and high-fidelity generation. Diffusion autoencoders have achieved impressive performance in the image domain, but an architecture does not yet exist to extend the capabilities to to long, irregular, and multimodal sequences. In this paper, we introduce the Diffusion AutoEncoder with Perceiver (daep), an architecture designed for self-supervised learning of long, irregular, and multimodal sequences. Our proposed architecture combines three components: (i) a Perceiver encoder that flexibly accommodates variable-length and tokenized inputs across modalities; (ii) a compact latent bottleneck for representation learning; and (iii) a Perceiver-IO diffusion decoder that reliably reconstructs the original sequence regardless of the regularity of its sampling. This design allows daep to scale to datasets with millions of heterogeneous samples, while producing both high-fidelity reconstructions and semantically-structured latent spaces. To contextualize our approach, we also develop a masked autoencoder baseline using the same Perceiver backbone (maep), enabling a controlled comparison between masking-based and diffusion-based objectives in the irregular-sequence regime. Across unimodal and multimodal spectroscopic and photometric astronomical datasets, daep achieves comparable or lower reconstruction error compared to VAE and maep baselines, with particular improvements in reconstructing critical high-frequency data features. While motivated by astronomical data, the proposed architecture is domain-agnostic and can be used for representation learning in healthcare, finance, and other areas where irregular and multimodal sequences are obtained. 2 Background Diffusion models. Diffusion models are score-based generative models that achieve state-of-the-art performance in image and video generation. These models learn a denoising process that transforms random noise into samples drawn from a target data distribution. Generation proceeds by iteratively denoising samples drawn from a Gaussian prior x_{T}\sim\mathcal{N}(0,I) into a clean data sample x_{0} after T denoising steps. Ho et al., 2020 proposed to learn a noise model \epsilon_{\theta}(x_{t},t) that predicts the noise added at diffusion time t to the corrupted data x_{t} . The model is trained by minimizing ||\epsilon_{\theta}(x_{t},t)-\epsilon_{t}||_{2}^{2} , where \epsilon_{t} is the true noise added to clean data x_{0} to produce x_{t} . During generation, the corruption process is inverted to produce a trajectory x_{T},x_{T-1},\dots,x_{0} , typically with large T . Song et al., 2020 introduced a deterministic variant, the denoising diffusion implicit model (DDIM), which enables generation in fewer steps using the trained noise prediction model. When conditioning variables z are available, the noise model can be extended as \epsilon_{\theta}(x_{t},z,t) and trained with the same loss ||\epsilon_{\theta}(x_{t},z,t)-\epsilon_{t}||_{2}^{2} . Diffusion autoencoders. Diffusion autoencoders were originally proposed for the image domain by Preechakul et al., 2022 . They encode data and reconstruct it using a conditional diffusion model. Because the encoding guides every denoising step, diffusion autoencoders capture fine-grained detail more effectively than, for example, variational autoencoders ( Kingma and Welling, 2013 ) . However, the original design relied on U-Nets ( Preechakul et al., 2022 ; Dhariwal and Nichol, 2021 ) , which are better suited to regular modalities such as images. Formally, the model encodes tokenized data into a latent representation z=\text{Enc}_{\theta}(x) , and a conditional score model \epsilon_{\theta}(x_{t},z,t) decodes data via a diffusion process. Training minimizes the score-matching loss ||\epsilon_{\theta}(x_{t},\text{Enc}_{\theta}(x),t)-\epsilon_{t}||_{2}^{2} . Perceiver. Perceiver and Perceiver-IO ( Jaegle et al., 2021b ; Jaegle et al., 2021a ) provide a general framework to (1) encode irregularly sampled sequences into a latent representation and (2) query outputs from this latent space. This makes them a natural fit for integration with diffusion transformers ( Dhariwal and Nichol, 2021 ) . Perceiver-IO is designed to accommodate diverse data settings for both encoding and decoding, and scales linearly with the input data. We provide additional details on the perceiver architecture in Section 3 below. Masked autoencoders. Masked autoencoders (MAEs) ( He et al., 2022 ) are an alternative approach to representation learning. Instead of reconstructing the full data from the latent alone, MAEs mask a subset of the input and predict the masked values conditioned on both the latent representation and the unmasked portion. Formally, the data are split into a masked portion x_{\text{m}} and an unmasked portion x_{\text{u}} . The latent representation is obtained by encoding the unmasked portion, z=\text{Enc}_{\theta}(x_{\text{u}}) , while the decoder learns a function \text{Dec}_{\theta}(x_{\text{u}},z) by minimizing the masked reconstruction loss ||x_{\text{m}}-\text{Dec}_{\theta}(x_{\text{u}},\text{Enc}_{\theta}(x_{\text{u}}))||_{2}^{2} .

Read the complete text