multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data

Andrew J. Loza, Jun Yup Kim, Shangzheng Song, Yihang Liu, Joseph J. Y. Sung, R Andrew Taylor, Dennis L. Shung · 2025

Read multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data by Andrew J. Loza, Jun Yup Kim, Shangzheng Song, Yihang Liu, Joseph J. Y. Sung, R Andrew Taylor, Dennis L. Shung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21680.txt.

Description

Andrew Loza and six co-authors present a transformer model for mixed data streams, in a 2025 paper. Real processes produce sequences that mix categories, numbers, and even text, sampled at uneven times that themselves carry meaning. Token-only models blur numbers, while equation-based models balk at categories and odd sampling.

Their model, multivariateGPT, predicts the next item in a sequence as a joint guess of its class and its value, with matching embeddings and a matching training loss. It learned patterns in simple physical systems and handled complex records such as heart traces and hospital data with many measures at once. The authors argue the design widens transformer models to the messy, mixed streams that science and medicine actually produce.

Catalog record

CREATORAndrew J. Loza, Jun Yup Kim, Shangzheng Song, Yihang Liu, Joseph J. Y. Sung, R Andrew Taylor, Dennis L. Shung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21680

Browse the full library · About the Tower of Babel