Overcoming Semantic Dilution in Transformer-Based Next Frame Prediction

Hy Nguyen, Srikanth Thudumu, Hung Du, Rajesh Vasa, Kon Mouzakis · 2025

Read Overcoming Semantic Dilution in Transformer-Based Next Frame Prediction by Hy Nguyen, Srikanth Thudumu, Hung Du, Rajesh Vasa, Kon Mouzakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16753.txt.

Description

Hy Nguyen and four co-authors tackle a flaw in video prediction models in a paper from two thousand twenty-five. Transformers can predict the next frame of a video, a skill used in driving, tracking, and motion work. Their attention step, though, splits each input into chunks across heads, which dilutes meaning.

The authors propose a new attention design that concentrates semantic content instead of splitting it thin. They also add a loss function that works in the latent space, matching how the model actually outputs predictions. Past models trained on rebuilt frames while outputting embeddings, a mismatch their loss removes. Tests showed better performance than the original transformer predictors. The fix targets representation quality, not just scale.

Catalog record

CREATORHy Nguyen, Srikanth Thudumu, Hung Du, Rajesh Vasa, Kon Mouzakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-16753

Browse the full library · About the Tower of Babel