Time Blindness: Why Video-Language Models Can’t See What Humans Can?

Ujjwal Upadhyay, Mukul Ranjan, Zhiqiang Shen, Mohamed Elhoseiny · 2025

Read Time Blindness: Why Video-Language Models Can’t See What Humans Can? by Ujjwal Upadhyay, Mukul Ranjan, Zhiqiang Shen, Mohamed Elhoseiny (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24867.txt.

Introduction

Ujjwal Upadhyay and co-authors expose a time blindness in video-language models in this 2025 paper. Modern models read space and time in videos well when scenes look normal. Their skill collapses when all meaning sits in the order of frames alone.

Their benchmark, SpookyBench, hides shapes and text inside sequences of noise-like frames. People spot the patterns with near perfect accuracy. Top models scored zero on the same task. The gap points to a heavy lean on single-frame spatial cues. When training data had weak spatial signal, model time sense decayed faster than human sight. The authors release the set to push new designs.

Catalog record

CREATORUjjwal Upadhyay, Mukul Ranjan, Zhiqiang Shen, Mohamed Elhoseiny
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Time Blindness: Why Video-Language Models Can’t See What Humans Can? Ujjwal Upadhyay, Mukul Ranjan, Zhiqiang Shen, Mohamed Elhoseiny Abstract Recent advances in vision-language models (VLMs) have made impressive strides in understanding spatio-temporal relationships in videos. However, when spatial information is obscured, these models struggle to capture purely temporal patterns. We introduce SpookyBench , a benchmark where information is encoded solely in temporal sequences of noise-like frames, mirroring natural phenomena from biological signaling to covert communication. Interestingly, while humans can recognize shapes, text, and patterns in these sequences with over 98% accuracy, state-of-the-art VLMs achieve 0% accuracy. This highlights a critical limitation: an over-reliance on frame-level spatial features and an inability to extract meaning from temporal cues. Overcoming this limitation will require novel architectures or training paradigms that decouple spatial dependencies from temporal processing. Our systematic analysis shows that this issue persists across model scales and architectures. We release SpookyBench to catalyze research in temporal pattern recognition and bridge the gap between human and machine video understanding. Dataset and code are available on the project website. 1 Introduction Large multimodal models have revolutionized visual understanding in both images [ 54 , 82 , 5 , 20 , 28 , 26 ] and videos [ 101 , 56 , 3 , 87 , 85 ] . Recent Video-Vision Language Models (Video-VLMs) demonstrate impressive capabilities across action recognition [ 91 , 40 , 102 ] , visual question answering [ 98 , 60 , 104 , 4 , 63 ] , dense captioning [ 65 , 95 , 93 , 41 , 17 , 18 , 13 ] , and temporal grounding [ 15 , 81 , 94 ] . Yet even in tasks labeled as temporal, strong per-frame spatial cues often allow models to succeed without genuinely reasoning over time [ 25 , 10 , 48 ] . This paper introduces SpookyBench , a benchmark that closes this loophole. Every stimulus is constructed so that individual frames appear as noise; the signal exists only in the temporal relationship between frames. Existing temporal benchmarks [ 10 , 49 , 97 , 46 ] still entangle spatial and temporal cues, making it difficult to diagnose where a model’s understanding breaks down. SpookyBench eliminates this confound entirely. In spirit it parallels ARC-AGI [ 23 ] , which uses synthetic puzzles to isolate core reasoning; here, controlled stimuli isolate a single fundamental capability, extracting meaning from change over time. Figure 1 : Illustration of the current video-language models’ limitations: over reliance on spatial visual features within individual frame. Frame sampling results in temporal information loss, while the visual encoder exhibits a strong spatial bias. This creates a coherence gap ( \times ) between well-represented spatial features (objects, scene layouts) and under-represented temporal features (motion patterns, causality), limiting video understanding capabilities. Current approaches to video understanding [ 77 , 61 ] typically follow a hierarchical paradigm: extract frame-level features using ViTs [ 6 , 67 , 29 ] , integrate these features temporally, and fuse them with language for downstream tasks [ 100 , 47 , 90 , 83 ] . This paradigm has yielded significant advances in general video understanding [ 45 , 31 , 77 , 61 ] . However, our findings reveal a critical blind spot: when information exists purely in the temporal domain without reliable frame-level features, state-of-the-art models fail catastrophically (Figure 1 ). The inability to decode temporal patterns has significant implications for real-world applications. In nature, organisms such as fireflies communicate through precise temporal sequences of bioluminescence [ 11 , 62 , 68 ] , encoding information exclusively through timing rather than spatial arrangements. Similarly, in medical domains, pre-ictal EEG patterns for epileptic seizure prediction emerge only through temporal analysis [ 75 ] . These natural examples demonstrate how temporal patterns can carry rich information even when individual observations contain minimal static content. Similarly, various human technologies from Morse code to digital communication protocols rely on temporal encoding, yet current Video-VLMs lack the fundamental mechanisms to process such information. The human visual system has evolved mechanisms for processing temporal information without relying solely on spatial cues [ 78 , 39 , 1 ] . Neuroscience research has revealed that temporal processing is distributed across neural structures rather than centralized in a single area [ 58 ] , and the brain uses intrinsic network dynamics to perform temporal computations [ 64 ] . Areas such as the parietal cortex integrate temporal information along with spatial and numeric magnitudes [ 8 ] . Our experiments confirm humans’ remarkable temporal perception: participants achieve over 98% accuracy on SpookyBench tasks without training. In stark contrast, our evaluation of 15 state-of-the-art Video-VLMs, including closed-source commercial systems such as GPT-4o [ 37 ] , and Gemini 2.0 Flash [ 27 ] , reveals near-zero accuracy on these same tasks. This performance gap persists across model architectures, parameter scales, and pre-training strategies, from compact systems like VideoLLaMA3-2B [ 99 ] to large-scale ones such as GPT-4o [ 37 ] and Qwen-VL [ 82 ] , and extends to models designed specifically for video understanding including LongVLM [ 87 ] , LLaVA-NeXT-Interleave [ 47 ] , and InternVideo2.5 [ 85 ] . Efforts to enhance temporal reasoning through specialized temporal modeling [ 70 , 66 , 86 ] and fine-grained temporal localization [ 55 , 15 , 81 ] have not addressed the challenge of extracting meaning from purely temporal patterns without reliable spatial features. Our findings suggest that achieving human-like video understanding requires rethinking how neural architectures process temporal information. Rather than treating temporal integration as secondary to spatial feature extraction, future models may need dedicated mechanisms for temporal pattern recognition, drawing inspiration from cognitive neuroscience research on distributed neural timing mechanisms [ 64 , 58 ] and specialized brain regions for temporal processing [ 8 , 59 ] . The gap between human and machine performance on SpookyBench indicates that current architectures remain “time-blind” despite strong performance on standard benchmarks. We release SpookyBench to catalyze research into temporal reasoning in Video-VLMs, with applications ranging from medical diagnostics to autonomous systems that must interpret temporal cues in complex environments. 2 Related Work 2.1 Temporal Reasoning in Video-VLMs Video-VLMs have advanced through several architectural families, including LLaVA variants [ 54 , 101 , 47 , 45 , 56 , 53 ] , the Qwen series [ 82 , 5 ] , InternVL models [ 21 , 20 , 85 ] , dual encoders [ 57 ] , interleaved tokens [ 3 , 105 ] , compression techniques [ 72 ] , and multimodal fusion [ 100 , 90 , 83 ] . All of these exhibit limited temporal reasoning, with hallucinations [ 46 ] , grounding failures [ 81 ] , and reliance on linguistic shortcuts [ 42 ] persisting across action recognition [ 91 , 40 , 102 ] , question answering [ 98 , 60 , 4 ] , and captioning [ 95 , 41 , 13 ] . Targeted interventions such as timestamp-aware encoding [ 70 ] , segment-level reasoning [ 66 ] , direct token processing [ 55 ] , temporal separation tokens [ 15 ] , specialized temporal streams [ 81 , 86 ]

Read the complete text