Causal Masking on Spatial Data: An Information-Theoretic Case for Learning Spatial Datasets with Unimodal Language Models
Jared Junkin, Samuel Nathanson · 2025
Read Causal Masking on Spatial Data: An Information-Theoretic Case for Learning Spatial Datasets with Unimodal Language Models by Jared Junkin, Samuel Nathanson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27009.txt.
Description
Jared Junkin and Samuel Nathanson test a core habit of language models on data that is not a sequence. Causal masking means each position can only look at earlier ones, a natural fit for text but an odd fit for spatial data. The pair use chess, which offers both board layouts and move lists for the same games, to study the trade-off directly.
Models trained on board states, even with causal masking, play more strongly than models trained on move sequences. The result holds across both attention styles tested. Though the test bed is chess, the lesson is broader for method design. Masking spatial data can work well, and in some domains it beats forcing the data into a sequence first.
Catalog record
| CREATOR | Jared Junkin, Samuel Nathanson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27009