DiffATS: Diffusion in Aligned Tensor Space

Jinhua Lyu, Tianmin Yu, Brian Kim, Lizhuo Zhou, Chanwook Park, Naichen Shi · 2026

Read DiffATS: Diffusion in Aligned Tensor Space by Jinhua Lyu, Tianmin Yu, Brian Kim, Lizhuo Zhou, Chanwook Park, Naichen Shi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09275.txt.

Introduction

Lyu and team build DiffATS, a setup training diffusion models on aligned tensor primitives, without pretrained autoencoders. Direct diffusion modelling of high resolution fields is costly, and compact primitives tackle this by packing high dimensional data tightly. Their build starts from Tucker decomposition, with low rank multilinear form via a core tensor and mode wise factors, but Tucker factors are non-unique: one tensor admits other rotated factors, blocking generation.

Orthogonal Procrustes match fixes it: medoid anchors from the data align factors, fixing gauge freedom, yielding compact, data aware matrix and tensor primitives decoded by direct multilinear reconstruction. The team proves the primitive maps give homeomorphisms between low rank tensors and matching primitive spaces, proving faithful forms. On images, videos, and PDE cases, DiffATS gets strong results with and without guidance while shrinking raw data 3 point 9 to 210 fold, with no trained autoencoders.

Catalog record

CREATORJinhua Lyu, Tianmin Yu, Brian Kim, Lizhuo Zhou, Chanwook Park, Naichen Shi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09275

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffATS: Diffusion in Aligned Tensor Space Jinhua Lyu, Tianmin Yu, Brian Kim, Lizhuo Zhou, Chanwook Park, Naichen Shi Abstract Direct diffusion modeling of high-resolution spatiotemporal fields is computationally challenging. Parameter-efficient primitives address this by representing high-dimensional data with a compact set of parameters. In this paper, we construct data-dependent tensor primitives without pretrained compression autoencoders. Our construction starts from Tucker decomposition, which captures low-rank multilinear structure through a core tensor and mode-wise factors. However, Tucker factors are non-unique: the same tensor can be represented by different rotated factors, which complicates generative modeling. We address this issue with orthogonal Procrustes (OP) alignment. Specifically, we select medoid anchor matrices from the data and align the factor matrices to resolve the gauge ambiguity. This yields matrix Grassmannian primitives and tensor Grassmannian primitives that are compact, data-adaptive, and directly decodable by explicit multilinear reconstruction. Theoretically, we prove that the proposed primitive maps are homeomorphisms between low-rank tensors and their corresponding primitive spaces, certifying that the representations are non-degenerate and topologically faithful. Building on these primitives, we propose Diffusion in Aligned Tensor Space (DiffATS), a generative framework that trains diffusion models directly on aligned tensor primitives. Across images, videos, and PDE solutions, DiffATS achieves strong unconditional and conditional generation performance while compressing original data by 3.9 × 3.9\times to 210 × 210\times , without relying on any pretrained deep compression autoencoders. † † footnotetext: ∗ All authors are with Northwestern University. † Corresponding authors: [email protected] . Figure 1 : Generation results of DiffATS on CelebA-HQ 1024, Moving MNIST, Kármán vortex street and 2-d Burgers’ equation. “GT” denotes the ground truth. 1 Introduction Diffusion models (DMs) [ 49 , 51 , 18 , 52 , 50 ] have become central tools for learning and sampling from complex unknown distributions, with success in generating images [ 18 , 50 ] , audio [ 28 , 34 ] , videos [ 20 , 35 ] , protein structures [ 1 ] , inorganic materials [ 63 ] , and flow fields [ 48 ] . However, standard DMs often require huge computational resources to model high-resolution spatiotemporal data that are naturally represented as large tensors, such as videos, weather observations, fluid states, and parametric PDE solutions. Direct diffusion in the original tensor space is computationally demanding as training and sampling require repeated function evaluations in the full data dimension [ 45 ] . This cost is particularly severe for scientific tensors. For instance, GenCast [ 43 ] , a diffusion model for global weather forecasting, trains on tensors of size 84\times 720\times 1440 and requires 3.5 days on 32 TPUs. To mitigate the computational cost, a common strategy is to perform diffusion in a lower-dimensional representation space. Latent diffusion models (LDMs) [ 45 ] first train a regularized variational autoencoder [ 25 , 44 ] to compress data into latent embeddings, then learn a diffusion model in the latent space. Many subsequent image synthesis methods build on this framework, such as transformer-based architectures [ 41 ] and high-resolution text-to-image models [ 12 , 42 ] . The same approach extends to video generation [ 16 , 3 , 37 ] , with the latent space coming from a pretrained or task-specific video autoencoder. LDMs have become the de facto standard for natural images and videos, where large-scale pretrained autoencoders are available. Scientific data poses a different challenge. For the generative modeling of scientific problems, such as weather forecasting [ 43 ] and turbulent flow generation [ 33 ] , off-the-shelf autoencoders pretrained on large-scale datasets are often not available. Training task-specific autoencoders is expensive: the cost scales with both network size and data dimensionality. Training also requires sophisticated optimization techniques [ 60 , 11 , 12 ] . This motivates autoencoder-free primitives that compress data through explicit transforms. Recent examples include primitives based on discrete cosine transforms [ 39 ] and tensor factorizations [ 4 , 15 ] . These methods reduce dimension efficiently, yet they rely on predefined cosine or tensor factor bases, which can limit their ability to approximate the raw tensors. In light of these challenges, this paper asks the following question: Can we construct data-dependent parameter-efficient primitives for high-resolution spatiotemporal data by exploiting their inherent low-rank structure to improve generative modeling? The starting point is the Tucker decomposition [ 58 ] , which represents a tensor \mathcal{X}\in\mathbb{R}^{n_{1}\times\cdots\times n_{d}} by a core tensor and d factor matrices. This decomposition gives a natural data-adaptive and low-rank parametrization. However, as evidenced in Fig. 3 , training DMs on this parametrization yields inferior performance. We attribute this to the degeneracy of the standard Tucker decomposition: infinitely many combinations of core tensors and factor matrices yield the same tensor \mathcal{X} . This arbitrariness breaks the continuity of the primitive map, and complicates downstream modeling. Our primary contribution lies in resolving this degeneracy problem through the orthogonal Procrustes (OP) alignment of factor matrices. Under proper conditions, OP selects a unique representative from each Grassmannian equivalence class. Building on this technique, we define two families of primitives: matrix Grassmannian primitives (MGPs) for low-rank matrices and tensor Grassmannian primitives (TGPs) for low-multilinear-rank tensors. They preserve the parameter efficiency of Tucker representations while removing the rotational ambiguity that destabilizes DM training. Our theoretical analysis further shows that the primitive map \Phi is a homeomorphism under an anchor-overlap condition, which guarantees that the primitive space preserves the local topological structure of the original data space. By combining these primitives with diffusion modeling, we propose Diffusion in Aligned Tensor Space (DiffATS). The pipeline is shown in Fig. 2 . During training, DiffATS maps each high-dimensional tensor (or matrix) to its TGP (or MGP) and trains DMs in the aligned primitive space. During inference, the generated primitive is mapped back to the original data space through the multilinear reconstruction map. Figure 2 : Pipeline overview of DiffATS. We conduct experiments on images, videos, and PDE solutions, with representative samples shown in Fig. 1 . Across all settings, DiffATS outperforms DCTDiff [ 39 ] , SDIFT [ 4 ] , FNO [ 32 ] , and average pooling (AvgPool) in generation quality, despite using compression ratios that match or exceed those of the baselines. Our main contributions are summarized as follows: • We introduce MGPs and TGPs as data-adaptive, parameter-efficient representations for low-rank matrices and tensors. These primitives retain the expressive power of Tucker decompositions while resolving the rotational ambiguity through anchored OP alignment. • Theoretically, we prove the continuity and bijectivity of the primitive map \Phi under overlapping conditions. To our knowledge, this is the first low-rank primitive family with a formal homeomorphism guarantee. • We propose DiffATS, an end-to-end autoencoder-free diffusion framework that trains and samples directly in the MGP and TGP spaces, and reconstructs generated samples via the multilinear reconstruction map. • We conduct extensive experiments on images, videos, and PDE solutions, and show that DiffATS outperforms all baselines on generation quality given a matching or higher compression ratio. 2 Related Works We briefly present the overview of the related work in the main paper, while the comprehensive review is available in the supplementary material. DCTDiff [ 39 ] reduces image dimension with cosine bases; SDIFT [ 4 ] and Tucker-based diffusion [ 15

Read the complete text