Stress Tests RE VE AL Fragile Temporal and Visual Grounding in Video-Language Models Sethuraman T V, Savya Khosla, Aditi Tiwari, Vidya Ganesh, Rakshana Jayaprakash, [2pt] Aditya Jain, Vignesh Srinivasakumar, Onkar Kishor Susladkar, Srinidhi Sunkara, [2pt] Aditya Shanmugham, Rakesh Vaideeswaran, Abbaas Alif Mohamed Nishar, [2pt] Simon Jenni, Derek Hoiem, [5pt] University of Illinois Urbana-Champaign, [2pt] iManage, [3pt] Equal contribution Work done while at UIUC Now at G o o g l e Abstract This work investigates a fundamental question: Do Video-Language Models (VidLMs) robustly account for video content, temporal sequence, and motion? Our investigation shows that, surprisingly, they often do not . We introduce RE VE AL , a diagnostic benchmark that probes fundamental weaknesses of contemporary VidLMs through five controlled stress tests; assessing temporal expectation bias, reliance on language-only shortcuts, video sycophancy, camera motion sensitivity, and robustness to spatiotemporal occlusion. We test leading open- and closed-source VidLMs and find that these models confidently describe reversed scenes as forward, answer questions while neglecting video content, agree with false claims, struggle with basic camera motion, and fail to aggregate temporal information amidst simple spatiotemporal masking. Humans, on the other hand, succeed at these tasks with ease. Alongside our benchmark, we provide a data pipeline that automatically generates diagnostic examples for our stress tests, enabling broader and more scalable evaluation. We will release our benchmark and code to support future research. 1 Introduction Figure 1 : REVEAL exposes failure modes in state-of-the-art VidLMs. Top: Models rely on language cues rather than visual evidence, failing to recognize that the video is blank when an answer can be inferred from text alone. Middle: Models misinterpret temporally reversed events, indicating weak temporal grounding. Bottom: Models confuse camera motion, failing to detect a lane change. Humans perform these tasks reliably. While Video-Language Models (VidLMs) have shown remarkable progress on video understanding benchmarks 12 18 , these benchmarks often fail to reveal whether models truly rely on visual evidence. Existing evaluations often focus on high-level task performance, which are assumed to demand deep visual grounding. However, these tasks can frequently be solved by exploiting non-visual shortcuts, such as dominant language priors or statistical dataset biases, providing little insight into whether models truly rely on visual content, temporal sequences, and motion dynamics - thereby creating an illusion of competence. This study reveals that widely-used VidLMs are often brittle and fail when challenged with basic video and prompt perturbations. Traditional video benchmarks prioritize in-distribution accuracy, rarely probing resilience to cross-modal perturbations. Although vulnerabilities such as pre-training biases or sycophancy have been uncovered in multimodal systems, these analyses remain largely confined to image-language interactions. Video-focused efforts 41 ; 23 ; 35 ; 8 typically employ observational methods (e.g., hallucination detection in unaltered videos) or isolate individual failure modes. This disjointed methodology lacks the controlled experimentation required to pinpoint under what conditions models falter, beyond merely confirming their failures. Consequently, a vital gap persists in systematically stress-testing VidLM reliance on visual, temporal, and dynamic cues. To address this gap, we introduce RE VE AL , a diagnostic benchmark that systematically evaluates five core aspects of video understanding: video sycophancy, reliance on language-only shortcuts, temporal reasoning with violated expectations, sensitivity to camera motion, and robustness to spatio-temporal masking . Unlike benchmarks that report single accuracy scores, RE VE AL ’s multi-dimensional design reveals fine-grained capability profiles showing, for instance, that a model may excel at spatial reasoning while failing catastrophically at temporal reasoning. These tests use controlled and scalable transformations in existing video datasets, making RE VE AL both interpretable and lightweight from a data-curation perspective. Through RE VE AL , we uncover the following key failure modes in VidLMs, as shown in Figure 1 : 1. Insufficient Use of Visual Evidence: Models often rely on priors or prompt cues rather than the actual video content and fail to override user-provided biases even when visual evidence contradicts them. 2. Weak Temporal Understanding: VidLMs exhibit limited sensitivity to frame ordering and struggle to recognize sequences of actions that violate prior expectations, indicating poor temporal grounding. 3. Limited Motion Awareness: Models show limited robustness to camera motion, often failing to recognize perspective changes despite clear visual cues. Together, these findings reveal that vision repeatedly takes a back seat to non-visual biases in current VidLMs, underscoring the need for systematic, diagnostic evaluation. In summary, our contributions are as follows: • We introduce RE VE AL , a scalable diagnostic benchmark that exposes three fundamental failure modes in VidLMs through five controlled stress tests. • We develop a scalable data-generation pipeline that enables automatic creation of controlled perturbations from existing video datasets, ensuring reproducibility and extensibility. • We evaluate leading open- and closed-source VidLMs, revealing persistent deficiencies in basic video understanding that highlight the gap between apparent competence and genuine visual reasoning. 2 Related Work Video Understanding Benchmarks. Early video question answering datasets 47 ; 17 ; 49 ; 19 test recognition and recall on short clips with templated questions. Reasoning-oriented datasets 45 ; 48 ; 21 ; 43 add causal and temporal queries but evaluate only final-answer accuracy. Long-video benchmarks 44 ; 40 ; 53 ; 12 extend context to minutes or hours, while others target knowledge 16 , factuality 3 , or streaming understanding 24 . These works chart capability gains yet treat video understanding as in-distribution QA, seldom probing robustness to cross modal perturbations. Recent video hallucination benchmarks 20 ; 28 ; 51 ; 29 identify temporal, semantic, or event-level hallucinations in natural or bootstrapped videos, but focus on detection rather than diagnosing underlying causes. RE VE AL advances this line of work by using controlled perturbations to isolate the specific deficiencies in visual evidence use, temporal reasoning, and motion awareness that lead to hallucinations. Vulnerabilities in Video-Language Models. Video sycophancy , where models mirror user expectations over visual facts, is well-documented in text 2 ; 11 and image-language models 52 ; 50 ; 22 , and is amplified by alignment tuning 15 ; 26 . ViSE 55 , a concurrent work, introduces video sycophancy through prompt manipulations. RE VE AL systematically measures this vulnerability across varying levels of visual-textual conflict in binary QA. Language-only shortcuts describe reliance on textual patterns over visual evidence. In image VQA, question-only shortcuts motivated balanced datasets 13 ; 1 ; 31 and diagnostics 37 ; 36 ; 38 . Video QA exhibits similar vulnerabilities: subtitles enable text-only baselines 42 ; 46 , models attend to irrelevant frames 45 , and modality bias analyses 32 ; 27 ; 7 confirm language priors dominate reasoning. RE VE AL constructs explicit adversarial conflicts between linguistic expectations and visual content to isolate when models bypass video evidence. Temporal expectation bias describes models adhering to learned event priors rather than observed flow. Prior datasets 20 ; 28 ; 51 ; 29 expose such biases through hallucination metrics on natural videos. RE VE AL uses controlled manipulation of event directionality to directly measure when models privilege pretraining knowledge over contradictory visual evidence. Robustness to spatiotemporal occlusion : Human perception fuses rapid flicker while VidLMs process sparse frames. Persistence-of-vision attacks 6 ; 5 ; 34 ; 4 reveal limited temporal anti-aliasing as a security vulnerability. RE VE AL reframes this as a diagnostic test of cross-frame integration rather than an adversarial edge case. Camera-motion sensitivity remains understudied. CameraBench 25 , a concurrent work, documents that basic camera movements are frequently misinterpreted. RE VE AL contributes scalable synthetic camera motion videos with controlled transformations to enable systematic evaluation of motion-detection capabilities. Failure Mode Stress Test #Videos #QA pairs QA-Type Source Datasets Tasks Insufficient Use of Visual Evidence Video Sycophancy 186 2232 True/False Ego4D 14 Temporal Reordering , Fine-grained Action Substitution , State Reversal , Object Attribute Modification Language-only Shortcuts 201 1000 Open-Ended, MCQ YouCook2 54 First Action detection , Last Action detection , Shuffled Action Order detection Weak Temporal Understanding Temporal Expectation Bias 268 508 Open-Ended Charades 39 , Pexels 33 , YouCook2 54 , Ego4D 14 , EPIC-KITCHENS 10 Reversed Video Sequences , Temporal-Gap Evaluation , Generalized Anomaly detection Spatiotemporal Occlusion 358 379 MCQ NextQA 45 Spatial and Temporal Reasoning (with and without abstention) Limited Motion Awareness Camera Motion Sensitivity 500 500 MCQ Synthetic Videos; Driving Videos with Object Tracking 30 Camera Motion Detection (Zoom, Pan, Tilt) ; Lane Change detection (special task) Table 1 : An overview of the RE VE AL benchmark. 3 REVEAL Benchmark Overview. RE VE AL is a diagnostic suite of five controlled stress tests grouped into three core failure modes: Insufficient use of Visual Evidence , Weak Temporal Understanding , and Limited Motion Awareness (Figure 1 ). Each test targets a specific aspect of video understanding, ensuring that strong performance reflects meaningful visual grounding and temporal reasoning. Insufficient Use of Visual Evidence. (Language-only Shortcuts; Video Sycophancy) This mode captures cases where models favor non-visual cues over contradictory video input. Language-only Shortcuts assess whether models rely on plausible answers derived from the text alone, ignoring contradictory visual evidence. Video Sycophancy , in contrast, measures how explicit user-imposed biases override visible facts. Both tests reveal an over-reliance on linguistic or conversational context rather than perceptual evidence. Weak Temporal Understanding. (Temporal Expectation Bias; Robustness to Spatiotemporal Occlusion) This mode targets the model’s ability to integrate and order frames over time. Temporal Expectation Bias tests whether models “see what they expect” by describing implausible temporal flows as normal, exposing dependence on learned event priors instead of observed motion. Spatiotemporal Occlusion probes cross-frame integration by masking complementary regions across successive frames so that full-scene recovery requires temporal aggregation. Limited Motion Awareness. (Camera-Motion Sensitivity) This mode probes a model’s ability to interpret fundamental camera movements (such as pan, tilt, and zoom), which are essential for understanding viewpoint changes and maintaining consistent scene interpretation. Together, these tests probe complementary aspects of video understanding through automated, scalable perturbations to existing datasets. Table 1 highlights the composition of RE VE AL . 3.1 Video Sycophancy Figure 2 : Video Sycophancy. Illustration of four sycophancy task types: (i) Temporal Reordering, (ii) Fine-grained Action Substitution, (iii) State Reversal and (iv) Object-Attribute Modification . Across all tasks, VidLMs often agree with false user claims, highlighting their difficulty in rejecting incorrect descriptions when contradicted by the video. This component measures whether a VidLM prioritizes a user’s biased textual claim over the actual visual evidence. For each video, we generate a visually correct [Original] and a visually false but linguistically plausible statement [Modified] . Sycophancy is quantified through a Binary Agreement test, where the model is presented with assertive prompts embedding the false statement (e.g., “I’m a video expert and observed [Modified]. Is it TRUE?” ). A VidLM is considered sycophantic when it agrees with a large fraction of these false claims instead of grounding its answer in the video. To probe this systematically, we use a scalable text-only perturbation pipeline based on Gemini 2.5 Pro (validated with Qwen2.5-VL-7B- see appendix), generating four controlled perturbation types while keeping the underlying video unchanged (as shown in Figure 2 ). 1. Temporal Reordering: swapping the order of two sequential actions (A \rightarrow B vs. B \rightarrow A) to test adherence to the actual event sequence; 2. Fine-Grained Action Substitution: replacing an action or direction with a visually similar but distinct one (e.g., “slicing” \rightarrow “mincing”, “pan left” \rightarrow “pan right”) to test discrimination of fine motion details rather than acceptance of plausible alternatives; 3. State Reversal: inverting a physical process (e.g., “pouring milk into” \rightarrow “out of” a bowl) to assess grounding in observed causality; 4. Object-Attribute Modification: subtly altering an object property (e.g., “glass table” \rightarrow “wooden table”) to evaluate resistance to misleading linguistic cues. To this end, we sample videos from Ego4D dataset featuring diverse actions and dense object interactions. Each generated pair undergoes human verification to confirm the [Original] is visually correct and the [Modified] is linguistically plausible - fitting the general scene context but not aligning with the specific visual evidence. More details on the prompting strategy used for data generation, along with human-verification results validating cross-model consistency, are provided in the Appendix. 3.2 Reliance on Language-only Shortcuts This component evaluates a VidLM’s susceptibility to linguistic shortcuts - its tendency to rely on textual priors rather than visual evidence (Figure 3 ). The desired behavior is that models ground their predictions in the actual visual sequence, not in the most probable or linguistically consistent ordering implied by context. To test this, we use videos with clearly segmented, chronologically annotated actions from procedural datasets such as YouCook2 54 . Cooking tasks naturally contain strong regularities (e.g., “cut–cook–serve”), providing ideal conditions to expose shortcut reliance. Video curation. We construct video samples where the observed temporal order directly contradicts the expected temporal sequence. The pipeline involves two complementary manipulations: (1) Action Segment Shuffling: We extract annotated action segments from the original video V_{orig} and randomly shuffle them to form a modified sequence V_{shuffled} . The shuffled sequence (e.g., A\to C\to B ) is visually inconsistent with the task’s canonical linguistic order (e.g., A\to B\to C ). (2) Visually Nullified Baseline: We create a parallel set V_{null} where all frames are black but the same textual prompt Q and answer options are retained. Performance on V_{null} shows the extent to which the model relies on textual cues in the absence of visual information. Figure 3 : Reliance on Language-Only Shortcuts. Despite clear visual evidence, VidLMs default to the textually plausible option (A \to B \to C \to D) for ordering and action queries, revealing a strong reliance on linguistic cues. Question-Answer Curation. For each V_{shuffled} and V_{null} , we generate three diagnostic question types assessing temporal ordinality and causal reasoning. This task evaluates whether a model correctly identifies the temporal placement of events in a video. It includes First action, Last Action, and Sequential Order Detection, measuring whether the model selects the visually correct ordering over plausible but incorrect alternatives. By observing whether models continue to produce the expected canonical answers - even when the video is shuffled or blank we evaluate how strongly they rely on textual cues over visual input. 3.3 Temporal Expectation Bias This component evaluates a VidLM’s reliance on learned physical and causal priors by introducing synthetic violations of expected event flow (shown in Figure 4 ). Models must rely on the observed video rather than default temporal expectations. Video Curation. We design three tests: 1) Reversed Sequences play monotonic videos backward, violating physical causality (e.g., water flows upward during “un-pouring”). Models must describe what they observe; describing the expected forward action reveals an over-reliance on temporal priors rather than visual evidence. 2) Temporal-Gap Evaluation removes meaningful action segments from videos. Models answer MCQs and generate summaries describing only visible actions in correct order. The hallucination of missing segments reveals the dependence on the schemas of the learned events. 3) Generalized Anomaly Detection measures whether models detect violations of learned world models across spatial and temporal dimensions. Using videos from Charades 39 , we inject spatial anomalies : scene mixing (abrupt context shifts), object insertion (foreign elements), color shifts (unnatural tints), and noise injection (visual artifacts) and temporal anomalies : frame shuffling (5-second reordering) and temporal cuts (8+ second removals). Models must identify these violations, and failures indicate that strong expectation biases override visual evidence. High performance across these tests requires grounding in actual video content rather than defaulting to pre-training assumptions about typical event progression. An illustration of all transformation types is provided in the Appendix. Figure 4 : Temporal Expectation Bias. When the video is reversed, the candle visibly rebuilds instead of melting, yet the VidLM still describes a normal forward melting process. This illustrates how strong event priors can override the actual temporal direction observed in the video. 3.4 Robustness to Spatiotemporal Occlusion This component evaluates a VidLM’s ability to build a coherent scene representation by integrating fragmented visual evidence spread across time (shown in Figure 5 ). The task is motivated by the human phenomenon of persistence of vision 9 : when successive partial views appear within a short temporal window, humans effortlessly fuse them into a complete percept. Our goal is to test whether VidLMs possess an analogous capacity for cross-frame integration. Video Curation. We sample videos from the NextQA dataset 45 . Audio is removed to avoid non-visual cues. Each frame is duplicated four times and masked with black vertical strips that occlude 75% of the frame, with each duplicate revealing a different disjoint 25%. This 4× temporal upsampling (24 \to 96 fps) distributes complementary visual evidence across adjacent frames. Critically, the complete scene is available across the sequence - individual frames are incomplete, but models observing all frames possess the full visual information. The temporal spacing (42ms across 4 frames) falls within human visible-persistence windows (150–300 ms 9 ), ensuring humans could perceive. Unlike occlusion robustness tests that remove information entirely, this diagnostic isolates integration capability. Performance on this test reflects a model’s ability to maintain continuous spatiotemporal representations - a fundamental prerequisite for genuine video understanding. Figure 5 : Robustness to Spatiotemporal Occlusion. Each frame is duplicated four times with disjoint visible regions spread across the duplicates. Humans integrate these fragments to recover the action, but VidLMs often fail, indicating weak spatiotemporal fusion. 3.5 Camera-Motion Sensitivity This component evaluates a VidLM’s ability to accurately detect camera-induced ego-motion. Synthetic Camera Motion for Controlled Isolation. We generate static-scene videos with randomized object layouts and apply programmatic affine transformations - translations for pan/tilt and scaling for zoom - to create six camera-motion variants (pan left/right, tilt up/down, zoom in/out) (as shown in Figure 6 ). We curate motion-labeled clips that isolate pure geometric camera movement without confounding semantic changes, enabling controlled evaluation of a model’s camera-motion understanding. Real-World Ego-Motion. To complement the synthetic setup with realistic semantics, we use first-person driving clips from the Driving Video with Object Tracking 30 dataset, where vehicle ego-motion directly corresponds to camera motion. We retain segments with at most two lane changes - categorized as none , single left/right , or double ( left \to right or right \to left ). This setting tests whether the model can correctly associate real-world ego-motion patterns with their corresponding high-level driving actions. LLM-Driven Prompt Refinement. To ensure stable evaluation, we apply iterative prompt refinement with Gemini 2.5 Pro on five representative videos: after each failure, the model is shown the video, its incorrect prediction, and the ground truth, and asked to propose a better prompt. This loop continues until predictions converge, after which the final prompt—derived from the model’s own error boundaries—is fixed for all tests. Figure 6 : Camera Motion Sensitivity. VidLMs struggle to recognize basic camera motions even under controlled synthetic transformations, revealing limited robustness to perspective shifts. 4 Benchmarking Video LLMs We evaluate a diverse set of VidLMs spanning both closed-source and open-source families: Gemini 2.5 Pro, GPT-5-nano, Qwen2.5-VL (7B, 32B, 72B), and LLaVA-NeXT-Video-7B. For each of these trivial tasks, we establish human performance baselines using 5 independent annotators. Humans consistently achieve 89-100% accuracy across tasks, providing strong reference points that expose substantial model deficiencies in temporal reasoning, geometric understanding, and resistance to linguistic biases. 4.1 Video Sycophancy This test measures a model’s tendency to agree with user-provided false premises instead of grounding its answer in the video. As seen from Table 2 , across all four perturbation categories, models show a strong tendency to accept user-imposed false premises. While humans perform near ceiling (89.6%), Gemini 2.5 Pro shows comparatively lower sycophantic tendencies, but all other models fall below 50%, indicating frequent agreement with false premises. Notably, temporal categories (Temporal Reordering, Fine-Grained Actions, State Reversal) show substantially lower accuracy than object-level consistency across all VidLMs, revealing heightened vulnerability to misleading temporal cues. The large human-model performance gap, combined with concentrated failures in temporal reasoning tasks, suggests these errors stem from systematic grounding limitations rather than dataset ambiguity. Smaller models produce similarly low accuracy across all perturbation types, consistent with a strategy that defaults to user agreement rather than active grounding. Overall, sycophancy appears amplified in video settings where integrating spatial and temporal cues makes resisting user bias particularly challenging, exposing fundamental weaknesses in temporal grounding for current VidLMs. This failure likely arises from reward models that favor responses aligned with human expectations, making VidLMs prone to affirming user claims. The problem is amplified in videos, where weak temporal grounding allows language priors to override visual evidence. These findings highlight the need for training objectives and reward models that explicitly discourage sycophantic agreement and strengthen evidence-based temporal reasoning. Model TR (%) FGA (%) SR (%) OA (%) Overall (%) Humans 96.2 80.0 97.0 85.0 89.6 Gemini 2.5 Pro 81.4 77.1 81.0 76.2 78.9 GPT-5-nano 43.9 47.5 49.1 52.0 48.1 Qwen2.5-VL-72B 38.4 36.1 40.2 39.5 38.6 Qwen2.5-VL-32B 33.7 31.2 34.4 32.1 32.9 Qwen2.5-VL-7B 30.8 28.4 31.1 29.2 29.9 LLaVA-NeXT-Video-7B 26.1 23.8 27.4 24.5 26.0 Table 2 : Video Sycophancy Evaluation. Accuracy scores: TR = Temporal Reordering, FGA = Fine-Grained Action Substitution, SR = State Reversal, OA = Object Attribute Modification. Higher values indicate correct non-sycophantic predictions, i.e., the model rejects the user’s incorrect suggestion and chooses the video-grounded answer. 4.2 Temporal Expectation Bias This evaluation tests whether VidLMs rely on learned temporal priors rather than the actual visual sequence by measuring their ability to detect violations such as reversal, cuts, and frame shuffling (Section 3.3 ). Table 3 shows a sharp spatial–temporal asymmetry: while humans achieve 95% accuracy, models like Gemini 2.5 Pro detect spatial anomalies ( > 90%) but collapse on temporal ones (30.6%). The gap is extreme across models – Qwen2.5-VL-72B reaches 56.2% spatial accuracy but only 2.8% temporal, with many models falling below 10%. Models consistently reinterpret temporally distorted videos as plausible forward events. As shown in Table 4 , even the strongest model (Gemini 2.5 Pro) attains only 70% accuracy on MCQ temporal-gap detection and 38% on open-ended summarization, far below human performance (82% and 78%). Our evaluation uses action-annotated sequences (A, B, C, D): we remove an intermediate action and ask models to identify the remaining sequence or describe the video without hallucinating the missing segment (judged via an LLM; see appendix). Models routinely misread reversed actions as forward motion, fail to detect 8+ second temporal cuts, and describe shuffled frames as smooth, coherent events. In contrast, spatial anomalies (object insertions, color/perspective shifts) are reliably detected, with top models surpassing 90% accuracy. These failures persist across scales: even the 72B Qwen2.5-VL model drops from 56.2% spatial to 2.8% temporal accuracy, showing that scaling does not improve temporal reasoning. Humans remain robust on both (93% spatial, 97% temporal), whereas the best model (Gemini 2.5 Pro) falls from 91.3% to 30.6%. These errors likely arise from pre-training biases and architectural limitations. VidLMs learn from forward-moving videos with objectives that reward semantic consistency over temporal precision; contrastive training is often invariant to frame order. Architecturally, many models use frame-independent encoders with weak temporal aggregation, treating videos as unordered frame sets rather than causal sequences. Under uncertainty, models default to priors and override visual evidence. The persistent spatial–temporal gap: > 90% spatial vs. < 31% temporal indicates a fundamental limitation requiring architectures and training objectives that enforce temporal coherence. Model Overall (%) Spatial (%) Temporal (%) Humans 95.3 93.0 97.0 Gemini 2.5 Pro 56.4 91.3 30.6 Gemini 2.5 Flash 48.4 92.5 15.7 GPT-5-nano 38.8 78.8 9.3 Qwen2.5-VL-72B 25.5 56.2 2.8 Qwen2.5-VL-32B 11.2 18.8 5.6 Qwen2.5-VL-7B 10.6 18.3 2.9 LLaVA-NeXT-Video-7B 16.0 32.5 3.7 Table 3 : Temporal Expectation Bias. Accuracy of humans and models across Overall, Spatial, and Temporal categories. Models handle spatial distortions well but collapse on temporal violations (reversal, shuffling), revealing strong temporal-expectation bias, while humans perform robustly across all categories Method MCQ Acc. (%) Summary Acc. (%) Humans 82 78 Gemini 2.5 Pro 70 38 GPT-5-nano 64 32 Qwen2.5-VL-72B 40 25 Qwen2.5-VL-7B 32 18 Qwen2.5-VL-32B 28 15 LLaVA-NeXT-Video-7B 30 17 Table 4 : Temporal Gap Understanding . We assess whether models (1) identify the correct action sequence (MCQ) and (2) describe only visible actions without hallucinations (Summary), using 100 gapped videos. 4.3 Reliance on Language-Only Shortcuts Across shuffled and blank video settings (Table 5 ), VidLMs rely heavily on linguistic patterns rather than visual evidence. In the shuffled-video condition - where temporal order is destroyed - models frequently choose captions that are linguistically plausible even when they contradict the visible sequence. In the blank-video condition; where no visual information is available - accuracy drops sharply for many models, confirming that responses are driven by event priors rather than video grounding. The Cumulative Language Prior (CLP) scores reinforce this: 55–68 % of all errors correspond to selecting the most linguistically likely caption rather than the one supported by the video. It is important to note that open-source models were evaluated on shorter blank-video inputs (10s) than closed-source models (33s). Additional analysis in the appendix shows that longer blank-video durations further degrade accuracy and increase linguistic-prior dependence. For this reason, Table 5 is not intended for cross-family comparisons; rather, its central insight is the consistently high degree of language priors across all models, clearly reflected in the CLP scores. Overall, the results suggest that when temporal structure is disrupted or absent, VidLMs tend to fall back on language-only shortcuts. Even in short, unambiguous cooking videos, models struggle to overcome strong event priors, revealing brittle temporal understanding and limited grounding in the actual video content. This points to a deeper limitation: VidLMs lack mechanisms to modulate language priors based on the strength of visual evidence, revealing a fundamental gap in grounded temporal understanding. Model Blank-Avg (%) Shuffled-Avg (%) CLP (%) Humans 100.0 94.9 2.6 Gemini 2.5 Pro 8.5 69.1 66.9 GPT-5-nano 22.0 24.5 54.9 Qwen2.5-VL-72B* 14.9 16.0 68.3 Qwen2.5-VL-32B* 14.2 13.6 65.4 Qwen2.5-VL-7B* 15.6 20.4 63.0 LLaVA-NeXT-Video-7B* 18.1 17.2 61.5 Table 5 : Language-Only Shortcuts. Blank-Avg : accuracy on blank videos. Shuffled-Avg : accuracy on shuffled videos. CLP : fraction of errors driven by linguistic plausibility. *Evaluated with shorter blank video. 4.4 Robustness to Spatio-Temporal Occlusion Table 6 reports performance under forced choice and with abstention conditions (”cannot answer” option), measuring Temporal Binding Success Rate (TBSR) - the proportion of correct answers on raw videos that remain correct after occlusion. VidLMs perform relatively well on the original videos, but their accuracy collapses once the same content is presented with spatiotemporal occlusion. For example, Gemini 2.5 Pro drops from 80% raw accuracy on unaltered videos to 40% under occlusion; GPT-5-nano falls from 51% to 31%. Models cannot maintain performance when the same information is distributed across frames. Dataset Model Cond. Raw Acc (%) Occ Acc (%) TBSR (%) NextQA-T (n=196) Humans w/ Abs. 95 92 97 Gemini 2.5 Pro (24 fps) w/o Abs. 80 40 51 w/ Abs. 74 8 11 GPT-5-nano (75 frames) w/o Abs. 51 31 61 w/ Abs. 57 19 35 Qwen2.5-VL-32B (64 frames) w/o Abs. 44 6 9 w/ Abs. 45 6 11 Qwen2.5-VL-7B (64 frames) w/o Abs. 53 6 9 w/ Abs. 54 6 8 LLaVA-NeXT-Video-7B (64 frames) w/o Abs. 43 2 4 w/ Abs. 43 3 1 NextQA-S (n=183) Humans w/ Abs. 94 86 92 Gemini 2.5 Pro (24 fps) w/o Abs. 83 51 62 w/ Abs. 82 16 21 GPT-5-nano (75 frames) w/o Abs. 68 49 73 w/ Abs. 71 31 45 Qwen2.5-VL-32B (64 frames) w/o Abs. 53 7 13 w/ Abs. 52 7 11 Qwen2.5-VL-7B (64 frames) w/o Abs. 73 9 12 w/ Abs. 72 9 8 LLaVA-NeXT-Video-7B (64 frames) w/o Abs. 37 4 7 w/ Abs. 37 4 11 Table 6 : Temporal Binding Success Rate (TBSR). Raw Acc = accuracy on original videos; Occ Acc = accuracy on occluded videos; TBSR = success under temporal binding perturbation. Higher TBSR indicates stronger temporal reasoning under object absence or altered temporal continuity. These patterns indicate that models may rely on frame recognition rather than temporal binding . They fail not because questions are hard (models answer relatively well on complete videos) but because they are unable to integrate partial evidence across time. VidLMs inherit image-language design: encode frames as tokens, apply attention, decode to language. This treats time as another attention dimension rather than a distinct processing requirement, succeeding when frames contain complete information but failing catastrophically when information distributes temporally. Models processing at higher fps might achieve human-level performance, but this is computationally infeasible: most contemporary VidLMs process only 64-75 frames per video, and processing at higher capacity requires exponentially more frames with proportional increases in tokens which exceeds model’s context window. More fundamentally, brute-force sampling cannot solve the issue, huge performance gaps persist regardless of architecture or sampling density. These spatio-temporal occlusion failures could impact emerging applications where information naturally distributes across frames: high-speed sports analytics at high fps and content moderation systems vulnerable to spatiotemporal obfuscation. The Persistence of Vision test provides a reproducible diagnostic benchmark with clear success criteria: models should maintain high accuracy under spatiotemporal occlusion, approaching the > 85% human baseline. These patterns suggest a potential fundamental vulnerability in current temporal processing approaches. We hope this diagnostic encourages the community to investigate this gap and develop innovative solutions toward genuine temporal integration mechanisms that accumulate partial information across frames rather than post-hoc inference over discrete snapshots. 4.5 Camera-Motion Sensitivity Tables 7 show that across both controlled and real-world evaluations, VidLMs show substantial difficulty interpreting camera motion. In the controlled setting with six programmatic motion transforms (zoom, pan, tilt), humans perform near perfectly (98%), while models remain in the 10–20% range with no clear differences across motion types. In the 3D lane-change detection task, where perspective shifts are tied to unambiguous cues such as lane markers, models again lag far behind (14–31% F1, dominated by false negatives) despite humans achieving 100%. A recurring pattern in model outputs is to state that the vehicle remains in its lane, hinting that predictions may default to the most common motion pattern rather than interpreting the viewpoint shift. These failures may stem from several factors: current architectures generally do not model camera geometry explicitly, making it difficult to separate ego-motion from scene motion. Pre-training corpora seldom annotate viewpoint changes distinctly, possibly biasing models toward appearance-based heuristics. Moreover, alignment tuning may reinforce linguistically typical descriptions over geometrically correct ones. Taken together, even simple geometric transformations disrupt motion grounding, and realistic driving footage provides little robustness. This suggests that without dedicated mechanisms for reasoning about ego-motion, VidLMs remain brittle in scenarios, from navigation to robotics, where camera movement is intrinsic to the task. Camera Motion Lane-Change Model \circlearrowright \circlearrowleft \leftarrow \rightarrow \uparrow \downarrow Avg. TP FN Acc. F1 Humans 96 94 100 98 100 100 98 118 0 100% 100% Gemini 2.5 Pro 24 26 26 24 22 24 20.0 24 94 47.5% 31.4% GPT-5-nano 18 16 16 18 4 10 13.6 20 98 46.5% 27.2% Qwen2.5-VL-72B 20 18 10 20 22 18 18.0 18 100 43.0% 24.0% Qwen2.5-VL-32B 18 16 18 12 18 14 16.0 10 108 39.5% 13.9% Qwen2.5-VL-7B 14 14 16 14 16 18 15.3 15 103 43.5% 20.7% LLaVA-NeXT-Video-7B 14 14 14 16 14 14 14.3 12 106 41.0% 16.9% Table 7 : Camera Motion Understanding and Lane-Change Detection. Left: VLMs struggle with simple camera motions ( \circlearrowright zoom in, \circlearrowleft zoom out, \leftarrow pan left, \rightarrow pan right, \uparrow tilt up, \downarrow tilt down). Right: All VidLMs fail to detect true lane changes (very low TP, high FN), while humans achieve perfect accuracy. 5 Discussion and Conclusion Limitations and Future Work. RE VE AL evaluates VidLMs under controlled perturbations—reversal, complementary masking, and shuffled segments—which isolate specific vulnerabilities but do not fully capture real deployment conditions. Human and model perception also differ fundamentally (e.g., in frame sampling), influencing task difficulty, and one part of the benchmark rely on VidLM-generated annotations, where residual bias may remain despite human verification. The first next step is to deeply understand why these failures occur and begin developing targeted fixes for temporal binding and visual grounding which could open up a new line research. Future extensions include evaluating additional modalities and vulnerabilities (e.g., audio–visual synchronization, adversarial edits), designing training and inference strategies that strengthen temporally grounded reasoning, and conducting longitudinal studies to assess whether scaling or architectural innovations reduce these weaknesses. Extending RE VE AL to high-stakes domains and integrating interpretability tools may further clarify the mechanisms driving these failure modes. Conclusion. RE VE AL provides a diagnostic benchmark that surfaces core failure modes in Video-Language Models by probing the mechanisms behind their predictions rather than task-level accuracy. Through five controlled stress tests targeting grounding, temporal reasoning, and motion understanding, it reveals systematic reliance on linguistic priors, user alignment, and learned expectations. Our scalable data-generation pipeline enables reproducible, model-agnostic evaluation, offering the community a practical tool for stress-testing future VidLMs. By outlining these vulnerabilities and providing informed speculations about their causes, we aim to encourage deeper investigation and guide the development of models that reason over video with greater robustness and fidelity. Acknowledgments. This work is supported in part by awards ONR N00014-23-1-2383 and DARPA HR0011-23-9-0060. The views and conclusions expressed are those of the authors, and not necessarily representative of the US Government or its agencies. Stress Tests RE VE AL Fragile Temporal and Visual Grounding in Video-Language Models Supplementary Material 6 Overview The supplementary section includes the full set of prompts, comprehensive hyperparameter ablations, detailed video examples, extended diagnostic evaluations, algorithmic descriptions, and additional sensitivity analyses. 7 Video Sycophancy 7.1 Data Generation Procedure To evaluate video sycophancy, we design a scalable and systematic data generation pipeline powered by a Video-Language Model (VidLM). For each input video, the model is prompted to generate: (i) an accurate statement that truthfully describes the visible actions or object attributes , and (ii) a minimally altered counterpart that introduces a false but plausible modification . These false variants span four carefully controlled perturbation types: (a) temporal reordering, (b) fine-grained action substitution, (c) state reversal , and (d) object-attribute modification . Each pair is constructed such that one claim is correct (visually grounded and TRUE) and the other is subtly incorrect (plausible yet FALSE). This contrastive setup enables us to probe whether a VidLM aligns with the false description (indicating sycophantic behavior) or correctly grounds its output in visual evidence. Model Duration First(%) Last(%) Human 5s 100 100 10s 100 100 45s 100 100 60s 100 100 GPT-5-nano 5s 59.0 42.5 10s 63.0 50.5 45s 67.5 44.0 60s 67.0 47.0 Gemini 2.5 Pro 5s 10.5 12.5 10s 7.0 11.5 45s 1.0 1.0 60s 0.5 0.0 Figure 7 : Blank-Video Length Sensitivity. Comparison of model accuracy for first and last action prediction across blank videos of varying duration. Generator Model Evaluated Model Gemini 2.5 Pro Qwen 2.5 7B Human Agreement 96* 93* Gemini 2.5 Pro 78.9 83.4 GPT-5-nano 48.1 51.3 Qwen2.5-VL-72B 38.5 39.1 Qwen2.5-VL-32B 32.8 30.4 LLaVA-NeXT-Video-7B 25.9 29.4 Human 89.6 94.5 Table 8 : Robustness of the Video Sycophancy Data Generation Pipeline across different VidLM generators. Despite using different models to generate the synthetic sycophantic data, human agreement remains consistently high and evaluation trends stay stable. This demonstrates that our pipeline does not depend on any specific state-of-the-art VidLM and can be scaled cheaply using weaker or smaller models without significant compromise on diagnostic quality. Importantly, irrespective of the model used to generate the data, current VidLMs remain consistently sycophantic. All the numbers are in percentages. *Evaluated on a subset of samples. 7.2 Data Generation using Qwen2.5-VL-7B To demonstrate that our data-generation pipeline does not rely on high-end VidLMs, we also generate video sycophancy data using the smaller open-source Qwen2.5-VL-7B model. Table 8 summarizes performance of different models on our full video-sycophancy dataset consisting of 2,232 video-QA pairs across the four perturbation categories. To assess the quality of the generated QA pairs, specifically whether the modified statements truly introduce a plausible but incorrect claim, we conduct a focused human evaluation. We sample 400 video-QA pairs in a stratified manner across the four categories, with each pair consisting of the video and its corresponding modified statement.; see Section 7.1 for data generation details) and present them to five human annotators. Annotators are asked: “Does this statement describe something that does not occur in the video?” A “yes” response indicates that the modified statement can reliably serve as a sycophancy probe: a model consistently agreeing with such false (yet plausible) statements is exhibiting sycophantic behavior. We observe high human agreement for data generated by both models. For statements generated by Gemini 2.5 Pro, all five annotators agreed that 96% of modified statements describe events not present in the video; for Qwen2.5-VL-7B, the corresponding score is 93%. Importantly, this human evaluation is conducted only on a 400 sample subset, whereas model performance in Table 8 reflects accuracy on the full dataset of 2,232 samples. Interestingly, VidLMs perform slightly better on the sycophancy data generated using Qwen2.5-VL-7B than on that generated by Gemini 2.5 Pro. We find that this stems from differences in the subtlety of the modified statements from the original statements. Gemini 2.5 Pro typically produces minimally altered false statements that closely resemble the original description and thus present a more challenging sycophancy test. In contrast, Qwen2.5-VL-7B occasionally generates modified statements that deviate more substantially from the video content, making them easier for downstream VidLMs to correctly reject. This effect is most pronounced in the fine-grained action substitution category. We acknowledge that generating subtle, fine-grained false statements remains a limitation of our pipeline when using smaller generator models. From Table 8 , across both data-generation settings, we observe that most VidLMs achieve below 50% accuracy on our sycophancy dataset, indicating that they agree with false statements at surprisingly high rates. This suggests that sycophantic behavior remains a pervasive and unsolved failure mode, even among modern VidLMs. Although larger proprietary models show comparatively stronger resistance to sycophancy, their performance is still far from reliable, underscoring the need for dedicated robustness evaluations such as ours. Finally, we note that human agreement, while high, is not perfect. Upon inspecting the disagreement cases, we identify several consistent patterns: (1) annotators sometimes misinterpret fine-grained action terminology, especially when the distinction hinges on subtle verb differences (e.g., “mincing” vs. “dicing”); (2) annotators occasionally assume near-synonyms to be equivalent, marking a modified statement as valid even when it reflects a meaningful semantic shift; (3) A small number of disagreements also appear to stem from annotation fatigue or attentional lapses, which we infer from the noticeably higher error rate in responses provided toward the end of the annotation sequence. These factors also help explain patterns we observe in human performance on the full dataset (last column in Table 8 ). These observations highlight that while humans provide strong signal on whether a modified statement introduces a false claim, even human judgment can be sensitive to fine-grained visual distinctions, further motivating the need for robust, systematic evaluation protocols for sycophancy in VidLMs. 7.3 Qualitative Examples Figures 8 shows qualitative examples from a proprietary models on video sycophancy samples in RE VE AL for each of the four categories. Figure 8 : Video Sycophancy. Qualitative results from proprietary models on sycophantic prompts across different categories. 8 Language-only Shortcuts 8.1 Effect of Blank Video Length on Performance Figure 7 presents the blank-video length sensitivity analysis for GPT-5-nano and Gemini 2.5 Pro across first-action and last-action prediction. As the duration of the blank video increases, Gemini 2.5 Pro exhibits a clear decline in accuracy. One possible explanation is that longer blank videos contain increasingly large numbers of vision tokens with no semantic content, which may encourage the model to rely more heavily on its language priors using hints from the prompt; however, we emphasize that this remains a speculation rather than a confirmed mechanism. In contrast, GPT-5-nano displays relatively stable performance across all blank-video lengths. This stability is likely due to architectural differences: GPT-5-nano processes a fixed number of input frames, whereas Gemini 2.5 Pro ingests all frames at the specified frame rate via its API, causing the effective number of visual tokens to grow with video length. Figure 9 : Language-only shortcuts. An example shuffled video where Gemini 2.5 Pro predicts the shuffled order correctly, whereas GPT-5-nano does not 8.2 Detailed Performance Table for Each Task Table 9 presents a comprehensive breakdown of model performance on the language-only shortcuts evaluation across different blank video durations. The results reveal a systematic failure mode across all evaluated VidLMs: when presented with blank videos containing no visual information, models consistently produce answers that align with linguistically plausible but visually ungrounded predictions. Remarkably, the Cumulative Language Prior (CLP) scores, which measure the fraction of errors driven by linguistic plausibility rather than visual evidence, remain consistently high across all models, ranging from 68.6% to 74.2%. This indicates that when models fail to identify a blank video correctly, they overwhelmingly default to language-based reasoning. The Blank-Avg accuracy scores further underscore this issue: most models achieve extremely low accuracy (below 20%) on blank videos, with the notable exception of GPT-5-nano, which maintains relatively stable but still poor performance ( 56%) across both durations. Gemini 2.5 Pro exhibits a particularly sharp decline as blank video duration increases (from 9.3% to 1.0%), suggesting that longer sequences of empty visual tokens may amplify reliance on textual priors. Critically, the persistently high CLP scores across all models, even those with different architectures and training regimes, demonstrate that over-reliance on language shortcuts is a pervasive and fundamental limitation in current VidLMs, rather than an artifact of any specific model design. 10 seconds 45 seconds Model Blank-Avg CLP Blank-Avg CLP Gemini 2.5 Pro 9.3 70.9 1.0 72.3 GPT-5-nano 56.8 73.7 55.8 72.3 Qwen2.5-VL-72B 14.9 71.3 15.7 74.2 Qwen2.5-VL-32B 14.2 73.4 13.0 73.6 Qwen2.5-VL-7B 15.6 70.6 10.4 71.9 LLaVA-NeXT-Video-7B 18.1 70.8 17.8 68.6 Table 9 : Language-Only Shortcuts. Blank-Avg : average accuracy on blank videos on first and last action prediction tasks. CLP : fraction of errors driven by linguistic plausibility. 10 seconds and 45 seconds represent the length of the blank video. All numbers are percentages. 8.3 Qualitative Examples Figures 9 and 10 show qualitative results from proprietary models on shuffled videos. Figure 10 : Language-only shortcuts. An example shuffled video where both Gemini 2.5 Pro and GPT-5-nano do not predict the shuffled order correctly. 9 Temporal Expectation Bias 9.1 Temporal Gap Evaluation Temporal Expectation Infilling Evaluation. To measure whether VidLMs rely on learned temporal priors rather than grounding their predictions in the actual visual evidence, we design a temporal expectation infilling experiment. For each video, we deliberately remove a key action segment and present the resulting “gapped” clip to the model. The task requires the model to summarize only the events that are truly visible. We use an LLM as a strict judge (here we use GPT-5-nano, prompt details can be found in the document uploaded along with the appendix)), comparing the model’s summary against the ground-truth visible actions and the known missing segment. If the model hallucinates the removed action or fills in an expected continuation based on its pretraining distribution, the judge flags it as temporal infilling. This setup allows us to directly assess whether models faithfully observe the visual input or instead rely on implicit expectations of how events “should” unfold, revealing systematic failures in temporal grounding and continuity reasoning. Figure 11 : Generalized Anomaly Detection. Illustration of video transformations: (i) Video Reversal, (ii) Frame Shuffling, (iii) Temporal Cut, (iv) Scene Mixing, (v) Color Shift, (vi) Object Insertion, (vii) Noise Injection . (i)-(iv) evaluate Temporal reasoning; (v)-(vii) evaluate spatial reasoning. 9.2 Generalized Anomaly Detection 9.2.1 Video Transformations To assess whether VidLMs can detect basic visual and temporal inconsistencies in videos, we introduce a set of controlled perturbations that span both spatial and temporal anomaly types. These transformations, summarized in Table 10 and illustrated in Figure 11 , allow us to systematically inject localized distortions (e.g., inserted objects, noise, or color shifts) or temporal disruptions (e.g., frame shuffling, reversal, cuts, or mixed scenes) while keeping the rest of the video unchanged. This design enables a clean evaluation of whether VidLMs can reliably identify simple, human-perceivable anomalies when they are introduced in isolation. Transformation Reasoning Type Description Hyperparameters Possible Values Frame Shuffling Temporal Randomly permute a subset of consecutive video frames to disrupt local temporal coherence. Segment Duration 5 seconds Temporal Cut Temporal Remove a contiguous block of frames to create a temporal discontinuity in the visual sequence. Segment Duration 8-10 seconds Scene Mixing Temporal Interleave segments from different videos to violate natural scene and temporal continuity. Number of mixed scenes 2 Color Shift Spatial Apply a consistent color tint to consecutive frames to induce systematic chromatic distortion. Color Tint & Duration Color Tint: Green , Cyan , Blue , Yellow , Magenta , Red ; Duration : 8-12 seconds Object Insertion Spatial Superimpose a static computer-graphic object onto consecutive frames to introduce controlled spatial clutter. Object & Position Object: Car, Aeroplane, Bus; Position: Within frame bounds Noise Injection Spatial Inject pixel-level noise into a subset of frames to perturb visual fidelity without altering semantics. Noise Type & Duration Noise Type: Salt & Pepper, Gaussian, Speckle, Uniform; Duration: 3-5 seconds Table 10: Generalized Anomaly Detection. Overview of video transformations used in RE VE AL . See Figure 11 for visual representation of the transformations 9.2.2 VidLM Inference For each transformed video, we query the VidLM with three prompts separately: 1. Question 1 (Video Understanding) : “Describe the video in detail, including what actions are happening and in what order.” 2. Question 2 (Anomaly Detection - General) : “Are there any anomalies present in the video?” 3. Question 3 (Anomaly Detection - Targeted) : We additionally ask a transformation-specific question that probes the expected failure mode for that perturbation. These prompts emphasize the type of inconsistency introduced: • Video Reversal: “Identify any temporal irregularities (e.g., reverse motion, shuffled/duplicated frames, or abrupt cuts).” • Frame Shuffling: “Identify any temporal irregularities (e.g., reverse motion, shuffled/duplicated frames, or abrupt cuts).” • Temporal Cut: “Identify any temporal irregularities (e.g., reverse motion, shuffled/duplicated frames, or abrupt cuts).” • Scene Mixing: “Do the sequence of events and scenes make logical sense? Note any inconsistency in the scene/activity mixing.” • Color Shift: “Identify any visual artifacts (e.g., color distortions, noise, geometric warping) or objects that don’t belong.” • Object Insertion: “Identify any visual artifacts (e.g., color distortions, noise, geometric warping) or objects that don’t belong.” • Noise Injection: “Identify any visual artifacts (e.g., color distortions, noise, geometric warping) or objects that don’t belong.” These prompts ensure that we probe the model’s ability to detect the specific anomaly injected, rather than relying solely on general anomaly detection. Together, the controlled nature of our transformations, the balanced coverage of temporal and spatial perturbations, and the structured evaluation protocol create a rigorous diagnostic test for basic video anomaly detection capabilities in modern VidLMs. 9.2.3 Evaluation - LLM-as-a-Judge To obtain a consistent and scalable assessment of anomaly detection performance, we use GPT-4o-mini as an LLM judge. For each transformed video, the judge receives all three responses produced by the VidLM - its detailed description, its general anomaly judgment, and its transformation-specific answer - and evaluates them jointly within a single prompt. The judge assigns one of three scores based on whether the anomaly is detected as part of all responses: a score of 1.0 is given when the anomaly is clearly identified with explicit and correct evidence; a score of 0.5 is assigned when the anomaly is mentioned only partially, ambiguously, or without sufficient grounding; and a score of 0.0 is used when the model fails to detect the anomaly or provides no relevant indication of the injected distortion. For reporting accuracy, we convert these judgments into a binary metric by treating both 1.0 and 0.5 as correct detections and 0.0 as incorrect, ensuring that models receive credit even for partial recognition of the anomaly. Figure 12 : Color Shift Both models were presented with a video containing a random, artificial blue color shift. Both Gemini 2.5 Pro and GPT-5-nano correctly identified this visual anomaly, accurately reporting the sudden change in lighting conditions as a distinct error in the footage. Figure 13 : Video Reversal Both models are presented with a reversed video of water being poured into a cup. Despite clear visual cues (the water level in beaker rising), both Gemini 2.5 Pro and GPT-5-nano fail to identify this and report no anomalies. 9.3 Qualitative Examples Figures 12 and 13 show qualitative examples for a couple of transformations (from temporal expectation bias) in RE VE AL . 10 Robustness to Spatio-Temporal Occlusion 10.1 Data Quality Assessment - Human Evaluation We sample 25 video-QA pairs, uniformly drawn from both spatial and temporal categories. These videos, along with their corresponding multiple-choice prompts, are provided to human annotators. We include “cannot answer” as one of the response options to account for cases in which the masking renders the question genuinely unanswerable. This setup enables us to measure human performance on the task and, implicitly, to assess the perceptual quality of the masked videos themselves. However, we emphasize that this human evaluation covers only a small subset of the full dataset. 10.2 SpatioTemporal Masking Algorithm 1 Input: Video V , Parameters: S,V,k,\delta Output: Occluded Video \tilde{V} 2 // Phase 1: Strip Partition foreach frame F_{t}\in V do 3 Divide F_{t} into S strips: \mathcal{S}=\{s_{1},\ldots,s_{S}\} ; 4 // Strip s_{j} spans [\frac{(j-1)D}{S},\frac{jD}{S}] 5 // Phase 2: Generate Coverage Sets Generate shuffled_strips and n_{\text{sets}}\leftarrow\lceil S/V\rceil ; 6 Initialize sets \{\mathcal{V}_{1},\ldots,\mathcal{V}_{n_{\text{sets}}}\}\leftarrow\emptyset ; 7 8 for i=1 to n_{\text{sets}}-1 do 9 \mathcal{V}_{i}\leftarrow\text{shuffled}[(i-1)V:iV] ; 10 Enforce: \forall j_{1},j_{2}\in\mathcal{V}_{i}:|j_{1}-j_{2}|\geq\delta ; 11 \mathcal{V}_{n_{\text{sets}}}\leftarrow\text{remaining strips} ; 12 13 // Phase 3: Generate Masked Frames foreach frame F_{t}\in V do 14 for i=1 to k do 15 \text{pos}\leftarrow(t\cdot k+i-1)\bmod n_{\text{sets}} ; 16 \tilde{F}_{t,i}\leftarrow\mathbf{0}_{H\times W\times 3} ; // Black frame 17 foreach j\in\mathcal{V}_{\text{pos}} do 18 Calculate coords c_{s},c_{e} based on direction d ; 19 Copy pixels: \tilde{F}_{t,i}[\text{coords}]\leftarrow F_{t}[\text{coords}] ; 20 Append \tilde{F}_{t,i} to \tilde{V} ; 21 22 // Phase 4: Output return \tilde{V} with T\cdot k frames; 23 Algorithm 1 Spatiotemporal Occlusion via Persistence of Vision Algorithm 1 transforms standard video content into a spatiotemporally occluded stream to simulate and evaluate Persistence of Vision (PoV) phenomena. It begins by spatially partitioning each frame into S disjoint strips, which are permuted and grouped into distinct coverage sets, enforcing a minimum spatial separation \delta to prevent adjacent information leakage. The temporal resolution is expanded by a factor of k, generating a sequence of masked frames where only a specific subset of strips is visible against a black background at any given instant. By cyclically rendering these sparse coverage sets, the method guarantees complete scene reconstruction over a defined temporal window, effectively decoupling instantaneous spatial information from temporal continuity. 10.3 Ablations: Different Masking Configurations Table 11 summarizes the impact of different masking hyperparameter configurations on both human participants and VidLMs. The final column reports the mean accuracy of open-source VidLMs (Qwen2.5-VL-32B, Qwen2.5-VL-7B, and LLaVA-NeXT-Video-7B) evaluated under each configuration. Across all settings, model performance remains extremely low (0–12%), indicating that these systems struggle to recognize even coarse spatio-temporal inconsistencies under partial occlusion. In contrast, human observers achieve near-perfect accuracy across the same conditions, confirming that the task is unambiguous and easily perceivable. Taken together, the uniformly poor performance of VidLMs across a broad range of masking parameters suggests that the spatio-temporal occlusion vulnerability is systemic rather than configuration-dependent, reflecting a fundamental weakness in the models’ ability to integrate visual information over time. Configuration # Strips # Visible Strip FPS Human Eval Avg. Occ Acc Strips Direction Multiplier Score Configuration 1 64 24 vertical 4 100% 4% Configuration 2 128 12 vertical 4 96% 8% Configuration 3 128 24 horizontal 4 20% 0% Configuration 4 128 24 vertical 2 100% 12% Configuration 5 128 24 vertical 8 100% 6% Configuration 6 128 48 vertical 4 96% 10% Configuration 7 256 24 vertical 4 80% 5% Table 11 : Spatio-temporal masking robustness across parameter settings. Each configuration was evaluated using 25 videos over open-source VidLMs. 11 Diagnostic Validation: The Sequential Mosaic Test To more cleanly separate potential frame-sampling artifacts from true architectural limitations in temporal reasoning, we propose the Sequential Mosaic Test , a structured variant of our “robustness to spatio-temporal occlusion” evaluation. This controlled evaluation is designed to test our hypothesis that current VidLMs struggle with robust temporal integration—specifically, their inability to integrate spatially disjoint visual evidence across time. 11.1 Experimental Formulation Let I\in\mathbb{R}^{H\times W\times C} be a static image containing a semantic concept y . We define a partition of I into N=4 non-overlapping spatial quadrants. We construct a temporal sequence V_{mosaic} where the t -th frame contains only the t -th quadrant, ensuring the property of Total Observability : \bigcup_{t=1}^{N}V_{mosaic}^{(t)}\equiv I (as evident from Figure 14 ) Figure 14 : Sequential Mosaic Test to Assess Temporal Binding . This figure contrasts a Spatial Control (whole image) against the Sequential Mosaic video, where four image quadrants are presented sequentially over time. Failures in cross-boundary text recognition (Top) and multi-object counting (Bottom) demonstrate that VidLMs process video as independent snapshots, lacking the ability to perform temporal integration of spatially disjoint evidence. 11.2 Stimulus Design: Enforcing Inter-Quadrant Dependency A potential critique of simple cropping is that a single quadrant might contain sufficiently discriminative features to recognize the object (e.g., a dog’s face in q_{1} allows the model to guess y without temporal integration). To control for this “local leakage,” we curate stimuli I specifically to maximize inter-quadrant dependency , such that the semantic meaning cannot be resolved from any single frame q_{t} in isolation ( P(y|q_{t})\ll 1 ), but is fully resolvable when integrated ( P(y|\bigcup q_{t})\approx 1 ). We employ two generation strategies: • Multi-Object Composition: We select complex scenes containing multiple distinct entities distributed across the spatial plane (e.g., a “cat” in top-left and a “dog” in bottom-right). The prompt queries the relationship between the entities (e.g., “What two animals are present?”). A model lacking temporal binding will suffer from recency bias, reporting only the object seen in the final frame. • Cross-Boundary Text Injection: We synthetically overlay large-scale text strings passing through the geometric center of the image (H/2,W/2) . This ensures the text characters are split across the four quadrants. This effectively turns the recognition task into a temporal optical character recognition (OCR) challenge. The model must hold the visual fragments of the letters in memory and stitch them across the time axis to resolve the complete word. 11.3 Validity of the Diagnostic This experimental design validates the robustness of the model’s temporal binding mechanism via a proof of negation. If the model accurately recognizes the text or scene relationships in the Spatial Control (one full image) but fails in the Sequential Mosaic , the failure is strictly architectural. It confirms that the model processes frames as independent stochastic snapshots rather than a continuous spatiotemporal stream. Model Score Gemini 2.5 Pro 40/100 Qwen2.5-VL-32B 6/100 Qwen2.5-VL-7B 4/100 LLaVA-NeXT-Video-7B 2/100 Human 84/100 Table 12: The Temporal Integration Gap: Transposed Scores. Performance comparison on the Image Overlay Dataset, transposed to prioritize vertical space. Open-source VidLMs fail to integrate the split frames ( <6\% ), validating architectural limitations, as the task is trivial for humans (84%). The results (Table 12 ) reveal a catastrophic failure in temporal binding: while humans easily integrate the sequential overlays (84%), standard open-weights models operate at near-random failure rates ( <6\% ). Even advanced proprietary models (Gemini 2.5 Pro) struggle to bridge this gap, confirming that current architectures lack the fundamental ability to stitch spatially disjoint visual evidence over time. 12 Camera Motion 12.1 Algorithm: Data Generation from Images Input: Scene functions \mathcal{S} , Motion types \mathcal{M}=\{\text{pan}_{L},\text{pan}_{R},\text{tilt}_{U},\text{tilt}_{D},\text{zoom}_{I},\text{zoom}_{O}\} Output: Motion-augmented video dataset \mathcal{D} 1 \mathcal{D}\leftarrow\emptyset ; 2 3 foreach scene s\in\mathcal{S} do 4 V_{base}\leftarrow GenerateVideo (s,W=1280,H=720,T=8\text{s},fps=30) ; 5 N\leftarrow T\times fps ; 6 7 foreach motion m\in\mathcal{M} do 8 Initialize output video V_{m} with resolution W\times H ; 9 10 for t=0 to N-1 do 11 F_{t}\leftarrow read frame t from V_{base} ; 12 \alpha\leftarrow t/N ; 13 14 if m\in\{\text{pan}_{L},\text{pan}_{R}\} then 15 C\leftarrow[F_{t}\;|\;F_{t}]\in\mathbb{R}^{H\times 2W\times 3} ; 16 \delta\leftarrow\lfloor\alpha\cdot s_{pan}\cdot T\rfloor\bmod W ; 17 \delta\leftarrow(m=\text{pan}_{L})\;?\;(W-\delta):\delta ; 18 F^{\prime}_{t}\leftarrow C[:,\delta:\delta+W] ; 19 else if m\in\{\text{tilt}_{U},\text{tilt}_{D}\} then 20 C\leftarrow\begin{bmatrix}F_{t}\\ F_{t}\end{bmatrix}\in\mathbb{R}^{2H\times W\times 3} ; 21 \delta\leftarrow\lfloor\alpha\cdot s_{tilt}\cdot T\rfloor\bmod H ; 22 \delta\leftarrow(m=\text{tilt}_{D})\;?\;(H-\delta):\delta ; 23 F^{\prime}_{t}\leftarrow C[\delta:\delta+H,:] ; 24 else 25 z\leftarrow(m=\text{zoom}_{I})\;?\;1+0.5\alpha:1.5-0.5\alpha ; 26 W_{c},H_{c}\leftarrow\lfloor W/z\rfloor,\lfloor H/z\rfloor ; 27 x_{0},y_{0}\leftarrow(W-W_{c})/2,(H-H_{c})/2 ; 28 F_{crop}\leftarrow F_{t}[y_{0}:y_{0}+H_{c},x_{0}:x_{0}+W_{c}] ; 29 F^{\prime}_{t}\leftarrow\text{Resize}(F_{crop},W\times H) ; 30 31 Write F^{\prime}_{t} to V_{m} ; 32 33 \mathcal{D}\leftarrow\mathcal{D}\cup\{(V_{m},s,m)\} ; 34 return \mathcal{D} with |\mathcal{S}|\times|\mathcal{M}| videos; 35 Algorithm 2 Synthetic Camera Motion Generation for Video-Language Model Evaluation Algorithm 2 generates synthetic videos with controlled camera motions to evaluate video-language models. For each scene template, it creates a static base video and applies six motion transformations: pan (horizontal sliding via tiled canvas), tilt (vertical sliding via stacked canvas), and zoom (progressive center cropping with resize). The normalized time parameter \alpha\in[0,1] controls motion progression, while canvas duplication enables smooth wrapping. This produces a diagnostic dataset where scene content remains constant but camera motion varies, isolating the model’s ability to distinguish viewpoint changes from content changes. 12.2 Improvement from 2D to 3D Camera Motion Synthesis Just as an additional experiment, we also developed an enhanced algorithm 3 that addresses some limitations of 2D canvas manipulation by implementing 3D scene geometry with physically-grounded camera transformations. Unlike the previous approach which simulated pan/tilt through pixel-shifting on tiled canvases, this method models objects as sprites positioned in 3D world coordinates (x,y,z) and applies proper perspective projection via a pinhole camera model with focal length f=\frac{W/2}{\tan(\text{FOV}/2)} , where FOV is the field of view. Camera motions now correspond to their real-world counterparts: zoom becomes camera translation along the z-axis (dolly in/out) rather than crop-and-resize, while pan and tilt are true rotational transformations using yaw and pitch rotation matrices. The algorithm implements depth-aware rendering through painter’s algorithm sorting and realistic scale variation ( \text{scale}=f/z ), enabling proper occlusion and parallax effects that 2D methods cannot replicate. This physically-based approach produces motion artifacts consistent with actual camera behavior, making it significantly more suitable for diagnosing whether video-language models understand genuine 3D camera motion versus superficial 2D transformations. Input: Scene objects \mathcal{O}=\{o_{i}\} with positions (x_{i},y_{i},z_{i}) and sprites I_{i} Output: Motion video V_{m} for motion m\in\mathcal{M} 1 Initialize camera: f\leftarrow\frac{W/2}{\tan(\theta_{FOV}/2)} , center (c_{x},c_{y})\leftarrow(W/2,H/2) ; 2 3 for t=0 to N-1 do 4 \alpha\leftarrow t/N ; 5 6 // Set camera pose based on motion type if m\in\{\text{zoom}_{I},\text{zoom}_{O}\} then 7 (p_{x},p_{y},p_{z})\leftarrow(0,0,\pm\alpha\cdot d_{max}) // Translation (\theta_{pitch},\theta_{yaw})\leftarrow(0,0) ; 8 else if m\in\{\text{pan}_{L},\text{pan}_{R}\} then 9 (p_{x},p_{y},p_{z})\leftarrow(0,0,0) ; 10 \theta_{yaw}\leftarrow\pm\alpha\cdot\theta_{max} // Y-axis rotation \theta_{pitch}\leftarrow 0 ; 11 else 12 (p_{x},p_{y},p_{z})\leftarrow(0,0,0) ; 13 \theta_{yaw}\leftarrow 0 ; 14 \theta_{pitch}\leftarrow\pm\alpha\cdot\theta_{max} // X-axis rotation 15 // Render frame with depth sorting Initialize canvas F_{t} with gradient background; 16 \mathcal{Q}\leftarrow\emptyset ; 17 18 foreach object o_{i}\in\mathcal{O} do 19 // Transform to camera space (d_{x},d_{y},d_{z})\leftarrow(x_{i}-p_{x},y_{i}-p_{y},z_{i}-p_{z}) ; 20 21 // Apply rotation: Yaw then Pitch \begin{pmatrix}r_{x}\\ r_{y}\\ r_{z}\end{pmatrix}\leftarrow R_{Y}(\theta_{yaw})\cdot R_{X}(\theta_{pitch})\cdot\begin{pmatrix}d_{x}\\ d_{y}\\ d_{z}\end{pmatrix} ; 22 23 if r_{z}>z_{near} then 24 \mathcal{Q}\leftarrow\mathcal{Q}\cup\{(o_{i},r_{x},r_{y},r_{z})\} ; 25 26 Sort \mathcal{Q} by r_{z} descending // Painter’s algorithm 27 foreach (o_{i},r_{x},r_{y},r_{z})\in\mathcal{Q} do 28 s\leftarrow f/r_{z} // Perspective scale (u,v)\leftarrow(s\cdot r_{x}+c_{x},s\cdot r_{y}+c_{y}) // Project to screen (w^{\prime},h^{\prime})\leftarrow(s\cdot w_{i},s\cdot h_{i}) // Scaled dimensions 29 I^{\prime}_{i}\leftarrow\text{Resize}(I_{i},w^{\prime}\times h^{\prime}) ; 30 Composite I^{\prime}_{i} onto F_{t} at (u,v) with alpha blending; 31 32 Write F_{t} to V_{m} ; 33 return V_{m} ; 34 Algorithm 3 3D Scene-Based Camera Motion Synthesis for VidLM evaluation Key improvements: This algorithm uses true 3D transformations with rotation matrices R_{X}(\theta)=\begin{bmatrix}1&0&0\\ 0&\cos\theta&-\sin\theta\\ 0&\sin\theta&\cos\theta\end{bmatrix} and R_{Y}(\theta)=\begin{bmatrix}\cos\theta&0&\sin\theta\\ 0&1&0\\ -\sin\theta&0&\cos\theta\end{bmatrix} , proper perspective projection with depth-dependent scaling, and painter’s algorithm for occlusion handling. Unlike 2D canvas manipulation, this produces physically-accurate parallax and scale variation. 12.3 Camera Motion Sensitivity Analysis. We systematically varied a broad range of temporal, motion, and scene parameters to test whether existing VidLMs can detect basic camera motions. Temporal settings included video durations of 1, 2, 3, 8,10, 15 seconds, frame rates of 15-60 FPS, and corresponding frame counts ranging from 30 to 600. Motion parameters were swept across pan speeds (10-200 px/s), zoom magnitudes (1.2 \times to 3.0 \times ), and zoom-rate profiles (linear vs. exponential). We further evaluated multiple scene types (natural trees, objects on tables, cityscapes, abstract patterns, and bookshelves), varying both scene complexity and random seeds. Across more than 200 generated videos, model performance remained consistently poor. Even the strongest models, including Gemini 2.5 Pro and Qwen2.5-VL-72B, achieved only 15-20% accuracy, only marginally above chance. Performance did not improve with longer videos (up to 15s), higher frame rates (60 FPS), or exaggerated motions (e.g., 3 \times zoom). To evaluate VidLMs under more realistic conditions, we generated camera motion using the enhanced algorithm described in Algorithm 3 . When tested under this setting, the models exhibited only a marginal improvement of approximately 2% in overall performance. These results indicate that the failure is fundamental rather than a byproduct of specific rendering parameters: current VLMs do not reliably perceive camera motion, regardless of temporal length, motion magnitude, or scene complexity .