Light Cones for Vision: Simple Causal Priors for Visual Hierarchy
Manglam Kartik, Neel Tushar Shah · 2026
Read Light Cones for Vision: Simple Causal Priors for Visual Hierarchy by Manglam Kartik, Neel Tushar Shah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24753.txt.
Introduction
Kartik and Shah introduce Worldline Slot Attention, a vision model treating objects as trajectories through spacetime. Standard models treat objects as lone points and miss part-whole hierarchy. Each object holds several slots at different hierarchy levels, sharing position but differing in time. Euclidean worldlines fail badly, scoring below chance. Lorentzian worldlines score far higher across three datasets, a sixfold gain over many runs. The result suggests visual hierarchy needs causal structure, which Lorentzian light cones supply with 11,000 parameters.
Catalog record
| CREATOR | Manglam Kartik, Neel Tushar Shah |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39616 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24753
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Light Cones for Vision: Simple Causal Priors for Visual Hierarchy Manglam Kartik, Neel Tushar Shah Abstract Standard vision models treat objects as independent points in Euclidean space, unable to capture hierarchical structure like parts within wholes. We introduce Worldline Slot Attention, which models objects as persistent trajectories through spacetime worldlines, where each object has multiple slots at different hierarchy levels sharing the same spatial position but differing in temporal coordinates. This architecture consistently fails without geometric structure: Euclidean worldlines achieve 0.078 level accuracy, below random chance (0.33), while Lorentzian worldlines achieve 0.479–0.661 across three datasets: a 6 × \times improvement replicated over 20+ independent runs. Lorentzian geometry also outperforms hyperbolic embeddings showing visual hierarchies require causal structure (temporal dependency) rather than tree structure (radial branching). Our results demonstrate that hierarchical object discovery requires geometric structure encoding asymmetric causality, an inductive bias absent from Euclidean space but natural to Lorentzian light cones, achieved with only 11K parameters. The code is available at: https://github.com/iclrsubmissiongram/loco . 1 Introduction How can a model learn that a wheel is part of a car, not merely near it? This seemingly simple question reveals a fundamental limitation of object-centric learning: existing methods treat all entities as independent points in Euclidean space, unable to capture hierarchical part-whole relationships. Slot Attention ( Locatello et al. 2020 ) and its variants discover objects by competitive grouping, but cannot distinguish wholes from parts or parts from subparts; a car and its wheel receive equivalent geometric treatment. Appendix B Prior work addresses hierarchy through hyperbolic embeddings, which encode tree structure via radial distance from an origin ( Nickel & Kiela 2017 ) . However, visual hierarchies exhibit causal dependency rather than tree branching: a wheel’s identity as “part of a car” arises from the car’s existence, not from branching like child nodes in a taxonomy. This mismatch motivates our central question: what geometric structure naturally encodes visual part-whole relationships? We propose Lorentzian spacetime, where light cones provide directional causal structure distinguishing past from future. Our method, Worldline Slot Attention, embeds slots in this geometry with a key architectural innovation: worldline binding . Instead of treating slots independently, we constrain slots at different hierarchy levels (object, part, subpart) to share spatial positions while occupying different temporal coordinates. This creates worldlines persistent vertical trajectories through spacetime hence enabling each object’s spatial position to aggregate information across all abstraction levels simultaneously. Challenging the assumption that architectural constraints alone suffice for hierarchy discovery, our systematic ablation demonstrates geometry is essential. The same worldline architecture collapses to 0.078 level accuracy in Euclidean space, below random chance (0.33), while achieving 0.479–0.661 in Lorentzian spacetime ( p<0.0001 ), a qualitative transformation from failure to functional discovery. Lorentzian geometry also outperforms hyperbolic embeddings, validating this geometric distinction empirically across three datasets and 20+ independent runs. Our contributions are: (1) Worldline binding, an architectural constraint enabling multi-scale information aggregation by sharing spatial positions across hierarchy levels; (2) empirical proof that geometry is essential- Euclidean worldlines fail catastrophically (0.078) while Lorentzian worldlines succeed (0.479–0.661); (3) evidence that Lorentzian outperforms hyperbolic embeddings because visual hierarchies are causal rather than tree-like; and (4) a lightweight method (11K parameters) demonstrating these findings across diverse benchmarks. 2 Related Work Object-centric learning. Slot Attention ( Locatello et al. 2020 ) discovers objects via competitive grouping, extended to video ( Kipf et al. 2022 ) , transformers ( Singh et al. 2022 ) , and scene composition ( Burgess et al. 2019 ; Greff et al. 2019 ) . Recent work addresses structure through various inductive biases: invariant representations ( Biza et al. 2023 ) , hierarchical pipelines ( Yuan et al. 2023 ) , adaptive slot selection ( Yuan et al. 2024 ) , and bottom-up clustering ( Yang et al. 2024 ) . However, these rely on architectural constraints or learned mechanisms rather than geometric priors. Hierarchical representations. Capsule Networks ( Sabour et al. 2017 ) use part-whole voting via dynamic routing but require predefined hierarchy depth. Hyperbolic networks encode tree hierarchies via radial distance ( Nickel & Kiela 2017 ; Ganea et al. 2018 ) . Lorentzian geometry appears in physics-inspired networks ( Brehmer et al. 2024 ) but not visual reasoning. Our contribution. Unlike prior work optimizing geometry for representation quality, we investigate whether geometry is necessary . Our ablation proves Euclidean worldlines fail consistently while Lorentzian succeeds, establishing directional geometric structure as essential, not optional. 3 Method Worldline Slot Attention operates in (d{+}1) -dimensional Lorentzian spacetime, where d spatial dimensions encode object identity and one temporal dimension encodes hierarchy level. 3.1 Lorentzian Geometry Lorentzian spacetime. We embed features and slots in \mathbb{R}^{d+1} with the Minkowski metric \langle x,y\rangle_{L}=x_{0}y_{0}-\sum_{i=1}^{d}x_{i}y_{i} , where x_{0} is temporal and x_{1},\ldots,x_{d} are spatial. The signed proper time distance is d_{L}(x,y)=\text{sign}(\langle\Delta,\Delta\rangle_{L})\sqrt{|\langle\Delta,\Delta\rangle_{L}|} where \Delta=x-y . Light cone structure. For slot s=(t_{s},\mathbf{s}) , the future light cone \mathcal{C}^{+}(s)=\{x:x_{0}>t_{s},\langle x-s,x-s\rangle_{L}\geq 0\} defines causal influence. This asymmetry (past \neq future) encodes hierarchy: abstract slots (low t ) see broad future cones; specific slots (high t ) see narrow cones. 3.2 Worldline Binding Architecture. We learn N object centers \mu_{i}\in\mathbb{R}^{d} and construct K=N\times L slots by replicating each center across L levels with fixed times \{t_{0},t_{1},t_{2}\} : s_{i,j}=(t_{j},\mu_{i})\quad\text{for }i\in[N],\,j\in[L] (1) This creates N worldlines \mathcal{W}_{i} : vertical trajectories through spacetime sharing spatial position. Updates aggregate across levels before GRU, enabling robust multi-scale estimation. 3.3 Scale-Adaptive Attention Cone membership. We compute adaptive horizons h_{j}(f)=w_{j}+\alpha\cdot(\rho(f)-0.5) where \rho(f) is k -NN distance, w_{j}\in\{0.9,0.6,0.3\} , and \alpha=0.3 . For temporal gap \tau=f_{0}-s_{0} and spatial distance r=\|\mathbf{f}-\mathbf{s}\| : \text{cone}(f,s,h)=h-\frac{r}{|\tau|+\epsilon}-10\cdot\text{ReLU}(-\tau)-5\cdot\text{ReLU}(r-|\tau|) (2) Attention. Combining proper time distance and cone membership: \text{attn}_{k,n}=\text{softmax}_{k}\left[\frac{-|d_{L}(f_{n},s_{k})|+\lambda\cdot\tanh(\text{cone}(f_{n},s_{k},h_{k}))}{\tau_{\text{temp}}}\right] (3) Slots update via GRU on weighted feature aggregation ( \lambda=0.5 , \tau_{\text{temp}}=0.1 ). 4 Experiments 4.1 Datasets We evaluate on three datasets with density-based hierarchies where sparsity correlates with abstraction level: Toy Hierarchical: 3 objects/scene with 3 levels (1 center, 4–5 parts, 2–4 subparts/part), 10% noise, 50–70 points. Fresh random data each epoch, 10 seeds, 300 epochs. Sprites: Similar structure, sprite layout (body, limbs, joints), 60–80 points, 10 seeds, 300 epochs. CLEVR: Hierarchies from CLEVR annotations ( Johnson et al. 2017 ) . Each object: 1 center (L0), 3–5 parts (L1), 8–15 subparts (L2). 3000 scenes, 5 seeds, 300 epochs. Details in Appendix J 4.2 Baselines & Metrics Models: (1) LoCo (ours): Lorentzian worldlines with scale-adaptive horizons; (2) Hyperbolic WL : Poincaré ball geometry with worldline binding; (3) Euclidean WL : Worldlines without geometric structure; (4) Euclidean Std : 9 independent slots (baseline). All models use identical architectures (11K parameters), learning rate 0.003, 3 attention iterations. Metrics: Object ARI uses Hungarian matching (permutation-invariant clustering). Level Accuracy uses fixed slot-to-level mapping: slots with t=1.0 (indices 0,3,6) \to L0, t=2.5 (indices 1,4,7) \to L1, t=4.0 (indices 2,5,8) \to L2. This mapping is non-permutation-invariant by design -worldline binding architecturally binds slots to temporal coordinates, removing permutation freedom. The Euclidean collapse to 0.078 indicates inability to maintain this structural constraint, not an evaluation artifact. Table 1: Main results across three datasets. Euclidean worldlines consistently collapse to 0.078 level accuracy (below random 0.33), while Lorentzian achieves 0.479–0.661. Format: Object ARI / Level Accuracy. {}^{**}p<0.0001 vs Euclidean WL on Level Acc; {}^{***}p<0.0002 vs LoCo on Level Accuracy Model Toy (10 seeds) Sprites (10 seeds) CLEVR (10