When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks

Chung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang · 2026

Read When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks by Chung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27272.txt.

Introduction

Lo and six colleagues study serialization friction, the mismatch when tasks that live in two directions are flattened into flat text for language models. Many symbol problems live in two directions by nature, with relations such as row-column matching and spatial nearness defined by layout position rather than sequence order. Preserving the same entries in a 1D sequence may not preserve the relations computation needs. A controlled made-up test set of three tasks, including matrix flipping, measures how layout-dependent relations go implicit under flattening.

Catalog record

CREATORChung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38918
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27272

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks Chung-Hsiang Lo, Lu Li, Diji Yang, Tianyu Zhang, Yunkai Zhang, Yoshua Bengio, Yi Zhang Abstract Large language models (LLMs) conventionally process structured inputs as 1D token sequences. While natural for prose, such linearization may introduce additional representational burden for tasks whose computation depends directly on explicit 2D structure, because row–column alignment and local neighborhoods are no longer directly expressed in the input. We study this setting, which we refer to as serialization friction , on a small diagnostic testbed of synthetic tasks with explicit 2D structure: matrix transpose, Conway’s Game of Life, and LU decomposition. To examine this question, we compare a text-only language pathway over serialized inputs with a vision-augmented pathway, built on the same language backbone, that receives the same underlying content rendered in task-faithful 2D layout, yielding a system-level comparison between two end-to-end input pathways. Across the tasks and settings we study, the visual pathway consistently outperforms the textual pathway; the gap often widens at larger dimensions, and error patterns under serialization become increasingly spatially structured. These findings indicate that the relationship between input representation and model performance on such tasks warrants further investigation, and suggest that preserving task-relevant 2D layout is a promising direction for structured 2D tasks. 1 Introduction Large language models (LLMs) have made rapid progress on reasoning over natural language ( 1 ) , yet their dominant interface to input remains a 1D token sequence ( 19 ) . While this interface is natural for prose, some tasks depend directly on explicit 2D relations such as row–column alignment and local neighborhood interaction. This raises a concrete question: when a task’s computation depends on explicit 2D structure, does presenting the same underlying instance in its original 2D layout change model behavior relative to 1D serialization? We use serialization friction as a descriptive lens for this setting. When inputs with explicit 2D structure are cast into 1D form, task-relevant relations are no longer directly expressed in the input and must instead be recovered from token order, delimiters, and formatting cues. Under this lens, part of the difficulty in such tasks may arise from representation in addition to the underlying computation. Prior work on tables, documents, graphs, and other structured inputs suggests that model behavior can depend strongly on how structure is represented or serialized ( 9 ; 11 ; 14 ; 17 ; 7 ; 24 ; 5 ) . Recent visual-text compression methods further show that the same symbolic content can be processed through a vision-augmented pathway by rendering it into 2D layout rather than presenting it directly as text ( 21 ; 3 ; 29 ) . Existing work mainly uses this paradigm for compression or context scaling; here we use it instead as a diagnostic comparison for tasks with explicit 2D structure. In this paper, we use structural isomorphism to refer to cases where the input representation preserves the relational structure that governs the task, so that task-relevant relations such as row–column correspondence or local neighborhood structure remain explicit. This notion is related to recent mechanistic interpretability work suggesting that transformers can internally organize state in ways that mirror the structure of the underlying problem domain, as seen in Othello-based world-model studies ( 2 ; 28 ) . To study this question, we build a small diagnostic testbed spanning three synthetic tasks with explicit 2D structure: matrix transpose, Conway’s Game of Life ( 4 ) , and LU decomposition. We compare a text-only pathway over serialized inputs against a vision-augmented pathway that receives the same underlying content rendered in task-faithful 2D layout, yielding a system-level comparison between two end-to-end input pathways. Across this testbed, the visual pathway is consistently stronger than the textual pathway, with the largest gaps often emerging as task size grows and with more spatially structured error patterns under serialization. This pattern is noteworthy in light of recent evidence that multimodal models often reason worse over visualized text than over the corresponding textual inputs ( 18 ; 15 ) . Because the two pathways differ beyond layout alone, we interpret these results as diagnostic evidence rather than proof of a unique mechanism. To narrow that interpretation, we also include a limited within-visual comparison in which disrupting the 2D layout worsens performance. Taken together, these results suggest that serialization friction is a useful lens for understanding the behavior we observe, and that preserving task-relevant 2D layout is a promising direction within the scope of tasks studied here. In summary, our contributions are as follows: • We build a small diagnostic testbed of three synthetic tasks with explicit 2D structure, matrix transpose, Conway’s Game of Life, and LU decomposition, and compare a text-only pathway over serialized inputs with a vision-augmented pathway over task-faithful 2D renderings of the same underlying content. • Across this testbed, the visual pathway consistently outperforms the textual pathway, with gaps that often widen as task size grows and with more spatially structured errors under serialization. We also provide a limited within-visual comparison showing that disrupting the 2D layout can worsen performance. 2 Related work Visual-Text Compression for Context Scaling. A recent alternative to token-based long-context scaling is visual-text compression , which renders long sequences into images and relies on a vision encoder to produce a compact representation for downstream decoding. Glyph exemplifies this direction by treating long-context modeling as a multimodal problem and showing substantial compression while maintaining accuracy on long-context benchmarks, without requiring task-specific fine-tuning for each downstream use case ( 3 ) . DeepSeek-OCR similarly studies optical 2D mapping as a compression mechanism and demonstrates that high compression ratios can still preserve strong decoding accuracy, reinforcing that vision token budgets can be traded against text length in a controllable way ( 21 ) . In contrast to these largely task-agnostic pipelines, we instead use task-faithful 2D rendering as a controlled probe of a representational hypothesis. Layout-Aware Modeling and 2D Inductive Biases. Prior work in Document AI and structured input modeling suggests that preserving 2D structure can improve understanding by making alignment, locality, and grouping explicit. Work on table understanding highlights the importance of representation for row–column structure ( 9 ; 20 ; 25 ; 17 ) , while layout-aware and OCR-free approaches such as LayoutLMv3, Donut, and Pix2Struct further show the value of exploiting page structure end to end ( 11 ; 13 ; 14 ) . Related work also suggests that explicit visual structure and serialization choices can affect reasoning over structured inputs, including in vision-language and graph settings ( 12 ; 22 ; 7 ; 24 ) . Our setting differs in goal: rather than optimizing for document parsing per se, we use layout as a controllable inductive bias for general reasoning tasks where 1D serialization is brittle. 3 Concept This section introduces the conceptual lens that motivates our study and the comparison setup used to operationalize it. We first define serialization friction, the representational burden that can arise when inherently 2D tasks are flattened into sequential form, and then describe the testbed through which we compare textual and visual representations on tasks with explicit 2D structure. 3.1 Serialization Friction in Structured 2D Tasks We instantiate this comparison between textual serialization

Read the complete text