TailOR : Trimming the Long-Tail of Visual World Modeling Evaluation
Bingxuan Li, Yining Hong, Cheng Qian, Hyeonjeong Ha, Jiateng Liu, Zhenhailong Wang, Yue Guo, Yunzhu Li, Heng Ji · 2026
Read TailOR : Trimming the Long-Tail of Visual World Modeling Evaluation by Bingxuan Li, Yining Hong, Cheng Qian, Hyeonjeong Ha, Jiateng Liu, Zhenhailong Wang, Yue Guo, Yunzhu Li, Heng Ji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24256.txt.
Introduction
Li and colleagues build a benchmark that stresses world models on the rare, awkward physics they usually skip. Visual world models, systems that simulate how scenes evolve, ace common interactions, because training data is dominated by them. Physical life follows a long tail, though. Most events are regular, while a broad spectrum of irregular interactions stays rare. TailOR, introduced in 2026, challenges image and video generation models to simulate those irregular cases.
Scenarios demand reasoning about object attributes and physical affordances: what an object allows to be done with it. The benchmark's central question is diagnostic. Have world models learned underlying physical principles, or are they replaying statistical patterns? By trimming evaluation toward the long tail, Li's team exposes failures that headline benchmarks, saturated with ordinary interactions, hide. The suite gives model builders a harder, more honest yardstick for physical understanding.
Catalog record
| CREATOR | Bingxuan Li, Yining Hong, Cheng Qian, Hyeonjeong Ha, Jiateng Liu, Zhenhailong Wang, Yue Guo, Yunzhu Li, Heng Ji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TailOR : Trimming the Long-Tail of Visual World Modeling Evaluation Bingxuan Li, Yining Hong, Cheng Qian, Hyeonjeong Ha, Jiateng Liu, Zhenhailong Wang, Yue Guo, Yunzhu Li, Heng Ji Abstract Figure 1 : TailOR challenges world models (e.g. image generation models, video generation models) to simulate long-tail scenarios (i.e irregular physical interactions) that require reasoning about object attributes and physical affordances beyond the typical training data distribution. Our benchmark aims to investigate: Do world models truly internalize the underlying physical principles of object interactions, or do they primarily rely on statistical regularities observed in training data? Abstract Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a broad spectrum of rare and irregular interactions remains underrepresented. Although recent world models achieve impressive realism on existing benchmarks, they primarily focus on simulating common physical interactions. This raises a central question: Do current world models internalize the underlying physical principles of object interactions, or do they primarily rely on statistical regularities observed in training data? In this work, we introduce TailOR , a benchmark that challenges world models to simulate irregular physical interactions. To enable systematic evaluation, we design three scenario modes that progressively challenge their reasoning: Regular scenarios reflect common tool–task pairs, unconventional scenarios replace regular tools with attribute-compatible substitutes to test affordance generalization, and impossible scenarios introduce attribute-violating tools to probe constraint awareness. Additionally, we design two complementary settings under a unified evaluation protocol: predictive generation requires inferring outcomes without guidance, while descriptive generation specifies the target outcome for faithful realization. Our experimental results reveal a clear long-tail gap in physical world modeling: Performance consistently degrades from Regular to Unconventional and Impossible scenarios, indicating limited generalization beyond common interactions. The largest drops occur in Interaction Accuracy and Physical Realism, pointing to weak affordance-level reasoning rather than perceptual issues. Failure analysis further shows that models rely on superficial visual patterns, where image models fail to realize correct state changes, and video models additionally suffer from temporal inconsistencies. These limitations persist even with explicit outcome descriptions: Models often revert to familiar patterns in descriptive settings and fail to infer correct outcomes in predictive settings. By exposing systematic failures under long-tail scenarios, TailOR highlights the need for models that understand object attributes, causal dynamics, and constraint-aware interactions, rather than relying on statistical co-occurrence. 1 Introduction Physical interactions in the real world do not occur uniformly. Instead, they follow a long-tailed distribution : A set of common interactions, such as cutting bread with a knife, a ball shattering glass under gravity, hammering a nail with a hammer, dominates both human experience and large-scale visual data. These familiar patterns form the head of the distribution, and we refer to these cases as head scenarios . Beyond them lies a vast and diverse long tail : physical interactions that are less typical, improvised, or context-dependent, yet physically valid. We refer to these cases as long-tail scenarios . Tool-using interactions in everyday environments provide concrete illustrative examples. As illustrated in Figure 1 , a coin can substitute for a screwdriver if its rigid edge transfers sufficient torque, and a heavy hardcover book can function as a hammer to crack a walnut. These cases are rare in isolation, but collectively they occupy a large portion of the possible physical interaction space in the real world. Recent world models, such as image and video generation models, have advanced rapidly in producing realistic scenes to simulate real-world physical interactions ( DeepMind 2025a , OpenAI 2025 , Qwen 2024 , OpenAI ) . Current evaluations on these models largely operate under standard world assumptions (i.e., the head scenarios) ( Bansal et al. 2024 , Gu et al. 2025 , Cai et al. 2025 ) : objects behave in familiar ways, tools are used for their intended purposes, and physical outcomes conform to everyday experience. Under such head-distribution scenarios, current models perform impressively. For example, on MMGR ( Cai et al. 2025 ) dataset, Sora-2 ( OpenAI 2025 ) achieves 86% physical accuracy and 92% visual realism, suggesting strong capability in simulating common physical interactions. But what if the object behaves in unfamiliar ways? What happens when objects interact in unconventional ways? Can visual generation models simulate these long-tailed scenarios? Success in head scenarios is often indistinguishable from genuine mastery of physical principles because performance in common settings can be achieved through pattern recognition and co-occurrence matching. Long-tail scenarios probe a fundamentally different capability. Rather than testing whether a model recalls frequent physics patterns, they require reasoning about why an interaction works, which is largely overlooked in current evaluation benchmarks. This motivates our central question: Do world models internalize and generalize the physical principles? To answer this question, we introduce TailOR , a benchmark designed to evaluate world models under irregular physical interactions. Our framework consists of three key components. First, we carefully define the problem and propose a multi-dimensional benchmark design that isolates key evaluation axes. We construct scenarios that progressively depart from familiar interactions. We begin with Regular scenarios that mirror common tool–task pairs frequently seen in visual data. We then move to Unconventional scenarios, where canonical tools are replaced with attribute-compatible substitutes, forcing models to go beyond surface-level associations and rely on object properties. Finally, Impossible scenarios use attribute-violating tools to test whether models can respect physical constraints and recognize when an interaction should fail. To further understand how world models infer physical outcomes versus realize specified results, we evaluate each scenario under two complementary settings: Predictive generation , which withholds the outcome to test outcome prediction, and Descriptive generation , which specifies the desired result to test controllability and physical consistency. Second, we develop a scalable data generation pipeline to construct diverse evaluation instances. Starting from action-driven tool-use tasks, we generate unconventional tools that satisfy required attributes and impossible tools that violate them, enabling controlled long-tail scenarios. Third, we design a diagnostic evaluation protocol with rubric-based questions measuring instruction adherence, interaction correctness, physical realism, and perceptual quality. We implement an automatic evaluation pipeline using a vision-language-model as a judge and verify that its scores strongly align with human annotations. Together, these components provide a controlled testbed for diagnosing the physical principle knowledge and physical reasoning capabilities of current world models. We systematically evaluate state-of-the-art image models (Z-Image, Qwen-Image, GPT-Image-1, Nano-Banana-2) and video models (HunyuanVideo-1.5, Wan2.2, Sora-2, Veo-3.1) on TailOR . Our experiments results trim down the long-tail of physical world modeling, revealing a fundamental limitation of current visual generative models: a pronounced performance gap under long-tail interactions. Across both image and video generation, performance degrades consistently from Regular to Unconventional and Impossible scenarios, highlighting limited generalization beyond common, head-distribution patterns. The drop is most severe in Interaction Accuracy and Physical Realism, suggesting that failures stem from weak affordance-level understanding rather than perceptual quality. Failure analysis shows that models rely on superficial visual patterns instead of physically grounded reasoning. Image models often produce plausible scenes but fail to realize correct state changes or object attributes, while video models suffer from additional temporal failures such as implausible dynamics and cascading inconsistencies over time. These limitations persist even when the desired outcome is explicitly specified. In descriptive settings, models frequently ignore