Emergent Compositional Communication for Latent World Properties

Tomek Kaszyński · 2026

Read Emergent Compositional Communication for Latent World Properties by Tomek Kaszyński (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03266.txt.

Introduction

Kaszyński shows in a 2026 study that teams of agents can invent compositional codes for hidden physical properties from frozen video features alone. Communicating through a narrow bottleneck with iterated learning, four agents converged on codes separating elasticity, friction, and mass ratio without labels. The learned code also supported planning and reached 85.6 percent mass-comparison accuracy on real camera footage.

Catalog record

CREATORTomek Kaszyński
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80140
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03266

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Emergent Compositional Communication for Latent World Properties Tomek Kaszyński Abstract What physical knowledge do video foundation models encode, and can it be extracted into discrete, compositional form? We show that multi-agent communication pressure, combined with a discrete Gumbel-Softmax bottleneck and iterated learning, induces compositional representations of world properties that are invisible in any single observation—elasticity, friction, mass ratio—from frozen pretrained features alone. The backbone determines what is communicable. In a controlled 2 × 2 2{\times}2 factorial comparison on physics simulations, DINOv2 ( Oquab et al., 2023 ) dominates on spatially-visible physics (ramp: 98.3% vs. 95.1%), while V-JEPA 2 ( Assran et al., 2025 ) dominates on dynamics-only collision physics where properties are recoverable only from temporal velocity differences (87.4% vs. 77.7%, d = 2.74 d{=}2.74 ). Scale-matched and frame-matched controls definitively attribute this gap to video-native pretraining: DINOv2 ViT-L at matched parameters performs worse ( d = 3.37 d{=}3.37 ), and DINOv2 with matched frame count degrades further ( d = 6.53 d{=}6.53 ). This extends the findings of Garrido et al., 2025 , who showed V-JEPA representations encode intuitive physics: our results demonstrate that this physical knowledge can be compressed into discrete, compositional codes under communication pressure. Multi-agent structure drives compositionality. With 4 agents, 100% of seeds converge to near-perfect positional disentanglement (PosDis = 0.999 =0.999 , holdout 98.3%; n = 80 n{=}80 ), while 2 agents produce compositional protocols stochastically (54%). Targeted controls confirm the driver is multi-agent structure—not bandwidth or temporal coverage. Causal intervention via position-zeroing shows surgical property disruption ( ∼ \sim 15% drop on targeted property, < < 3% on others), and the frozen protocol transfers to cross-property reasoning (93.8%), outcome prediction (88.7% at 25 × \times compression), and action-conditioned planning with counterfactual velocity reasoning (91.5%, r = 0.780 r{=}0.780 ). Validation on Physics 101 real camera footage ( Wu et al., 2016 ) confirms mass-comparison accuracy of 85.6% on unseen objects, with temporal dynamics contributing +11.2% beyond static appearance, agent-scaling compositionality replicating at 90% for 4 agents, and the causal intervention extending to real video (zeroing the mass-relevant agent reduces accuracy by 7.8pp while the other causes only 2.1pp disruption; p = 0.022 p{=}0.022 , d = 1.87 d{=}1.87 ). The discrete communication channel functions as an empirical instantiation of the latent variable z z in LeCun, 2022 ’s ( LeCun, 2022 ) cognitive architecture—our results provide evidence that discrete latent structure within JEPA-style world models supports compositional, causally addressable physical reasoning. 1 Introduction Physical understanding requires inferring properties that are never directly visible. Elasticity, friction, and damping cannot be observed in a photograph—they reveal themselves only through temporal dynamics: how objects bounce, slide, and decelerate. Humans infer these properties effortlessly and communicate about them using compositional language (“the bouncy, slippery ball”). Can artificial agents develop similar capabilities from perceptual experience, driven by the need to communicate? Understanding how agents develop structured representations of latent physical properties has implications for robotics (communicating material properties between manipulators), interpretable world models (discrete, inspectable representations of physical concepts), and multi-agent coordination in partially observable environments. We study this question at the intersection of emergent communication ( Lazaridou et al., 2017 ; Havrylov & Titov, 2017 ; Lazaridou & Baroni, 2020 ) and intuitive physics ( Battaglia et al., 2013 ; Piloto et al., 2022 ) . Prior work in emergent communication has focused on referential games with directly visible attributes—color, shape, position ( Choi et al., 2018 ; Chaabouni et al., 2020 ; Ren et al., 2020 ) . Work on physical understanding from video has focused on prediction tasks with continuous representations ( Wu et al., 2017 ; Ye et al., 2018 ) . We are not aware of prior work that studies emergent discrete communication about latent physical properties inferred from temporal dynamics, with explicit compositional generalization holdouts. Our setup is simple. Two agents each observe video of a ball interacting with a ramp. Each ball has invisible physical properties—elasticity and friction—drawn from a 5{\times}5 grid. The agents must cooperate on a pairwise comparison task: determine which ball has higher elasticity and which has higher friction. They communicate through a discrete bottleneck: a short message of categorical symbols transmitted via Gumbel-Softmax ( Jang et al., 2017 ) . Each agent observes only its own scene—the bottleneck forces compression of physical information into discrete symbols. Senders never observe property values and never receive supervision on message structure; learning is driven entirely by task loss at the receiver. The key question is whether the structure of the message reflects the structure of the world. Do agents develop compositional protocols where different message positions encode different properties? We find that given three conditions—(1) a perceptual backbone capable of extracting relevant properties (DINOv2; Oquab et al., 2023 ), (2) a factored message structure providing positional slots, and (3) iterated learning with population-based receiver pressure ( Kirby et al., 2014 ; Rita et al., 2022 ) —over half of 2-agent training runs spontaneously develop positionally disentangled protocols ( n{=}80 , 54% compositional, with significantly better generalization, p{=}0.011 ). Scaling to 4 agents yields compositionality in all 80 seeds tested (PosDis =0.999 , holdout 98.3%). Causal intervention confirms this is genuine compositionality: zeroing a single message position selectively disrupts the property it encodes while leaving others intact (§ 4.2 ). Our five main contributions are: 1. Compositional communication about invisible physics. Agents develop positionally disentangled protocols from raw video (54% of 80 seeds with 2 agents; 100% with 3 and 4 agents across all seeds tested). Targeted controls show that this transition is driven by multi-agent structure, not temporal coverage or bandwidth: randomizing frame assignments preserves the effect, while matched-bandwidth single senders do not (§ 4.6 ). Compositionality provides a significant generalization advantage ( p{=}0.011 , Cohen’s d{=}0.59 ), and a frozen compositional sender enables cross-property reasoning on a novel task at 93.8% accuracy. An alternative compositionality pressure (LazImpa) achieves 0% compositionality, confirming iterated learning is necessary in our experiments (§ 4.1 – 4.2 ). 2. Perception determines what is communicable. In a controlled 2{\times}2 factorial comparison (two backbones \times two datasets, matched 4-agent configurations), V-JEPA 2 significantly outperforms DINOv2 on collision dynamics (87.4% vs. 77.7%, p{<}0.0001 ), while DINOv2 significantly outperforms V-JEPA 2 on ramp physics (98.3% vs. 95.1%, p{=}0.001 ). A scale-matched control (DINOv2 ViT-L, 304M parameters matching V-JEPA 2) confirms the collision gap is driven by video pretraining, not model capacity: DINOv2 ViT-L achieves only 74.6% ( d{=}3.37 vs. V-JEPA 2). At 2 agents, no backbone difference is detectable on either dataset, revealing that the discrete bottleneck equalizes performance until sufficient communication bandwidth exposes backbone-level encoding differences (§ 4.7 ). 3. Consistent mechanism across domains with information-driven specialization. The identical communication architecture produces compositional protocols across four domains. Agent specialization patterns reorganize to match each domain’s information structure, ranging from near-perfect disentanglement (specialization ratio 0.95 on spring-mass) to distributed encoding (0.20 on 6-property vision). Agents allocate bandwidth proportional to property extractability ( r{=}0.964 , n{=}6 ), consistent with rate-distortion principles ( Cover & Thomas, 2006 ) (§ 4.3 – 4.4 ). 4. Boundary condition: continuous vs. categorical. The mechanism succeeds on continuous property variation but fails on categorical visual recognition, where agents memorize class codes. This identifies a

Read the complete text