Structure over Pixels: Learning Variable-Length Visual Programs

Piotr Wyrwiński · 2026

Read Structure over Pixels: Learning Variable-Length Visual Programs by Piotr Wyrwiński (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27696.txt.

Introduction

Piotr Wyrwinski, Kacper Dobek, and Krzysztof Krawiec propose STROP, a discrete tokenizer that learns variable-length visual programs. A length head trained with a four-phase curriculum calls the active prefix in one forward pass, without post-hoc search or preset compression rates. At about 250 bits per crop, the adaptive model beats fixed-length baselines on segmentation by about 2 to 3 points.

Catalog record

CREATORPiotr Wyrwiński
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65813
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structure over Pixels: Learning Variable-Length Visual Programs Piotr Wyrwiński Abstract Discrete visual tokenizers translate images into ordered sequences of codes, providing a natural representation for structural description of scenes. Yet existing adaptive tokenizers either require post-hoc search or select among a discrete set of pre-trained rates, rather than learning a continuous per-image sequence length coupled to the model and scene, and they typically train against pixel reconstruction, emphasizing texture rather than structure. We propose STROP, a discrete visual tokenizer architecture that forms structural scene representations and simultaneously learns how long an image’s visual program should be. Using a four-phase curriculum supervised by local rate–distortion probes against frozen DINOv3 features, STROP optimizes a dedicated length head that estimates the active prefix length in a single forward pass. By bypassing pixel-level reconstruction gradients, the codebook is shaped entirely by the quality of higher-level latent representations. Program length grows with scene complexity, and signs of compositional structure emerge both in downstream dense-prediction transfer and in direct inspection of the learned code vocabulary. 1 Introduction Figure 1 : Images as discrete adaptive programs. Each image is encoded as a variable-length sequence of discrete code IDs. For each highlighted code position, we intervene by erasing that token from the program and reinterpreting the remaining sequence. The attribution panels show the resulting change in the DINO-aligned patch field: red denotes higher token influence and blue denotes lower influence within that panel. Localized red regions suggest that individual program tokens act as semantic handles over the scene. Heatmaps are normalized per panel for visualization; token scores are raw attribution magnitudes. Deep neural networks prove unmatched at most computer vision (CV) tasks, mainly thanks to efficient representation learning. Their learned representations remain however verbose, distributed, convoluted, and thus hard to inspect and validate. These deficiencies are critical for many use cases that require robust scene interpretation, like robotics, planning, autonomous driving or reliable medical diagnosing. To address these limitations, models need to reason about scenes in terms of objects, their properties, and their spatial relations. Such structural visual descriptions remain hard to learn, especially when supervision is scarce. Object-centric learning and neural program induction pursue such descriptions in different ways ( 26 ; 38 ; 35 ; 34 ) . Discrete visual tokenizers (DVTs) ( 18 ) offer a concrete starting point, since they already translate images into quasi-symbols. In those architectures, a scene is mapped to an ordered sequence of discrete codes ( program in the following), which can be usually interpreted back into the spatial feature field. The model learns to generate such sequences without hand-designed predicates, object labels, or a-priori helper structures like scene graphs. Enforcing latent representations to discrete sequences of tokens from a fixed codebook facilitates inspection and, most importantly, opens the door to asking crucial research questions, like signs of code reuse, symbol grounding, or compositional structure. Most DVTs break the link between scene content and program length. VQ-VAE and VQGAN-style models produce a fixed spatial grid of codes, and recent 1D tokenizers replace that grid with a fixed number of latent slots ( 40 ; 12 ; 48 ) . The architecture chooses the sequence length before the image is seen. A smooth sky patch and a crowded street can change which codes are used, but both receive the same program length (sometimes referred to as code budget ). This sits uneasily with cognitive principles like Gestalt principles, minimum description length, not to mention the Occam’s razor. The number of tokens should correlate with the ‘amount’ of visual structure in the observed scene – this is the first step, if not a prerequisite, for forming principled structural representations. Admittedly, recent discrete tokenizers make the budget adjustable. Some train decoders to tolerate prefixes; others use inference-time search, recurrent allocation, routing among fixed rates, or information-theoretic scores of input compressibility ( 1 ; 28 ; 43 ; 11 ; 36 ; 46 ) . The findings elaborated in those methods suggest that length control should be tightly linked with visual tokenization, rather than being delegated to some external mechanism. Another prevailing characteristic in this area is the emphasis on fidelity of reconstruction. Indeed, previous works suggest that training DVTs with pixel-level reconstruction loss serves compression well. However, it can waste budget on capturing textural characteristics that carry little semantic or geometric structure. Feature-distillation tokenizers like BEiT v2 ( 32 ) , DINOSAUR ( 35 ) , REPA 49 , and MAETok 7 show that guiding training with teacher features and representation-aligned objectives can produce useful latent spaces and object-sensitive structure. Motivated by the above findings, we propose STROP, a DVT architecture that comprises a pair of encoder-only transformers, generating and interpreting a causal VQ sequence; a learned length head decides how much of the sequence to expose. DINOv3 feature distillation supplies the training signal, and a detached pixel decoder serves as an inspection layer. We measure rate–quality curves and dense-prediction transfer, then inspect code reuse and spatial grounding. This gives four concrete contributions: (i) Variable-length visual programs. We supervise per-image program length with local rate–distortion probes of the trained tokenizer and use a prefix curriculum: random truncation first, predicted prefixes later. (ii) A simple generator–interpreter tokenizer. Two encoder-only transformers with learned queries write and read causally ordered VQ codes. DINOv3 feature distillation provides the main supervision. (iii) Pixel probes without pixel training. The convolutional decoder receives detached feature grids, so reconstructions are an inspection surface and pixel gradients do not shape the codebook. (iv) Evaluation of representation and structure. We report rate–quality tradeoffs and downstream dense-prediction performance, then analyze code reuse and spatial grounding as tests for a learned visual grammar. 2 Proposed approach 2.1 Architecture Figure 2 : STROP architecture overview. Given an input image x\in\mathbb{R}^{3\times H\times H} , we seek a compact discrete program \mathbf{c}=(c_{1},\dots,c_{L}) with c_{k}\in\{1,\dots,N\} from which a DINO-aligned patch field can be recovered, where L varies per image and |CB| is the codebook size. The model (Figure 2 ) comprises five stages: a frozen visual encoder producing patch-wise features, a program generator that maps those features into a sequence of tokens, a vector-quantization bottleneck, an interpreter that expands the program back to a spatial feature grid, and a convolutional decoder . An auxiliary length head predicts L from the encoder features, enabling variable-length programs at inference without autoregressive sampling. Encoder. We extract token-level features with a frozen DINOv3 ViT encoder ( 37 ) . For an H{\times}H image with patch size p , the encoder produces S=(H/p)^{2} tokens of dimension d_{\text{enc}} . A layer norm followed by a linear projection maps every token from d_{\text{enc}} to a shared model dimension d , yielding source features Z_{e}\in\mathbb{R}^{S\times d} . Program generator. To map the patch-based visual features into a sequence of tokens describing the scene, we adopt an encoder-only transformer that processes Z_{e} with an appended sequence of K learnable query tokens. The concatenation [Z_{e};\,Q]\in\mathbb{R}^{(S+K)\times d} is processed by M standard transformer encoder layers. A causal mask prevents query token k from attending to query tokens k^{\prime}>k , and source tokens from attending to any query, ensuring that no future-query information leaks through intermediate source representations. After encoding, only the K query outputs are retained as the pre-quantized, raw

Read the complete text