RankVideo : Reasoning Reranking for Text-to-Video Retrieval Tyler Skow, Alexander Martin, Benjamin Van Durme, Rama Chellappa, Reno Kriz Abstract Reranking is a critical component of modern retrieval systems, which typically pair an efficient first-stage retriever with a more expressive model to refine results. While large reasoning models have driven rapid progress in text-centric reranking, reasoning-based reranking for video retrieval remains underexplored. To address this gap, we introduce RankVideo , a reasoning-based reranker for video retrieval that explicitly reasons over query-video pairs using video content to assess relevance. RankVideo is trained using a two-stage curriculum consisting of perception-grounded supervised fine-tuning followed by reranking training that combines pointwise, pairwise, and teacher confidence distillation objectives, and is supported by a data synthesis pipeline for constructing reasoning-intensive query-video pairs. Experiments on the large-scale MultiVENT 2.0 benchmark demonstrate that RankVideo consistently improves retrieval performance within a two-stage framework, yielding an average improvement of 31% on nDCG@10 and outperforming text-only and vision-language reranking alternatives, while more efficient. 1 1 1 https://github.com/tskow99/RANKVIDEO-Reasoning-Reranker 1 Introduction Platforms across education, entertainment, and social media now host billions of videos, creating a growing demand for effective and scalable retrieval methods. Text-to-video retrieval (video retrieval) addresses this need by ranking large video collections in response to natural language queries. However, the task remains challenging due to the need for strong multimodal representations 37 , cross-modal alignment between text and audiovisual content 35 ; 24 , and the ability to scale to large real-world collections containing hundreds of thousands of videos 19 . A common paradigm in information retrieval (IR) for scalable systems is to pair an efficient bi-encoder 17 ; 43 with a more expressive reranker that refines first-stage results 36 ; 32 . This two-stage pipeline reduces the number of query-document comparisons, making the slower reranking tractable at query time. While recent work in video retrieval has produced strong first-stage models, 35 ; 37 ; 11 ; 24 ; 48 , Figure 1: RankVideo judges the relevance between a query-video pair, dynamically reasoning or answering depending on the difficulty of the query-video pair. reranking remains largely unexplored. Leveraging textual reasoning models from IR is limited by the usefulness of extracted text (e.g., captions, transcribed speech, embedded text); this often omits critical visual or audio information, is not always available, and can be computationally expensive to generate. In contrast, video-native reranking, which uses audiovisual inputs directly rather than relying on extracted text, provides a more robust alternative to text-only rerankers. Inspired by the recent success of Large Reasoning Models (LRMs) in multimodal understanding 22 ; 2 ; 12 and reasoning reranking 44 ; 23 ; 50 , we introduce RankVideo , a video-native reasoning reranker for text-to-video retrieval. Given a query–video pair, RankVideo predicts relevance by comparing the log-probabilities of discrete answer tokens, producing a scalar relevance score without requiring reasoning traces ( Figure 1 ). To train an effective reranker, we first propose a data synthesis method for creating reasoning-intensive queries by leveraging the visual, audio, on-screen embedded text, and metadata features of videos. Using this data, we adopt a two-stage training process. In the first stage, perception-grounded supervised fine-tuning (SFT), the model learns to generate captions grounded in video content. In the second stage, the perception-grounded model is then fine-tuned as a reranker using a unified objective that combines pointwise classification, pairwise ranking, and teacher distillation. The pointwise term trains the model to classify query-video pairs as relevant or not, while the pairwise term encourages correct ranking of one positive video against two hard negatives per query. Soft-relevance probabilities are distilled from a large reasoning teacher, providing calibrated supervision that captures confidence beyond binary labels. With this training process, RankVideo achieves substantial retrieval gains across diverse first-stage retrievers, averaging 31% improvement on nDCG@10, while remaining significantly faster than existing reasoning-based reranking baselines. We also observe that the model adaptively allocates reasoning effort, engaging in deeper reasoning only when necessary, which further improves efficiency without sacrificing performance. Our contributions can be summarized as follows: 1. We introduce RankVideo , a video-native reasoning reranker for text-to-video retrieval trained with a two-stage curriculum directly on audiovisual inputs. 2. We develop a data synthesis pipeline to generate reasoning-intensive query-video pairs. 3. We conduct extensive experiments demonstrating the effectiveness, generalizability, and efficiency of RankVideo compared to strong text-only and vision-language reranking baselines. 2 Related Large Reasoning Models Large reasoning models (LRMs) extend pretrained language and multimodal models with the ability to perform multi-step reasoning, often by generating intermediate rationales or explicitly allocating additional computation at inference time 2 ; 46 ; 6 ; 8 . Prior work has shown that structured reasoning with chain-of-thought or reinforcement learning can substantially improve performance on complex understanding and decision-making tasks 41 ; 12 ; 55 . However, the amount of compute spent at test time is a common concern in the literature 1 ; 7 ; 14 ; 52 . Reranking Neural information retrieval (IR) commonly adopts a two-stage approach to retrieval, seperating a fast, high-recall first-stage retriever from a more expressive reranker that operates on a samll subset of the first-stage results 16 ; 25 ; 26 ; 31 . Canonically, reranking is performed using a cross-encoder that jointly encodes the query and each candidate document to produce a relevance score 30 ; 36 ; 29 ; 32 . Recently, large reasoning models have been adapted as rerankers, demonstrating substantial performance gains by explicitly scaling test-time compute for improved relevance judgments 23 ; 38 ; 44 ; 50 ; 56 . While effective, existing approaches are largely developed for text-based retrieval. Our method follows this trajectory but extends reasoning-based reranking to a video-native setting, where relevance judgments require jointly reasoning over visual, audio, textual, and temporal signals. Text-to-Video Retrieval Video retrieval is a core research area in video-language understanding 53 ; 51 ; 42 ; 3 ; 39 . However, it has traditionally been done with captioning datasets converted to retrieval tasks at small scales 5 ; 15 ; 18 ; 47 ; 40 . MultiVENT 2.0 19 introduced a large-scale, more reasoning intensive dataset for video retrieval, which better reflected real world retrieval needs. 19 found that state-of-the-art methods don’t scale to 100k+ videos or fail with real-world queries. Instead, for optimal performance and mirroring text IR, video retrieval should be split into two stages, where a first-stage retriever produces a ranked list on the entire index 9 ; 24 ; 35 ; 37 ; 54 and then a subset of that list is reranked by a more expensive cross-encoder. This work looks to introduce the second stage of this process. Concurrent work also introduces a reranker for video content 20 . While this model performs state-of-the-art on contrived retrieval tasks (e.g., MSR-VTT), we find it significantly decreases first stage performance in real-world retrieval settings. 3 Data Synthesis Video retrieval lacks training data for extensive finetuning and challenging retrieval needs that better match real world video retrieval ( 19 , beyond descriptive captions turned into queries,) . To create high-level, reasoning intensive queries, we first generate and extract text representations of the video content. We caption the videos with Qwen3-Omni-30B-A3B-Instruct 46 , transcribe the audio with Whisper-Large-v2 34 , and extract OCR with a state-of-the-art multilingual OCR system 10 . Using these texts as a proxy for the video content, we then provide a text reasoning model ( 49 , Qwen3-32B ,) 5 variations of the data: caption only, audio only, OCR only, metadata only, and all information. We filter these queries to a high quality reasoning intensive subset and ensure the queries are not relevant to other videos (e.g., broad queries). To do this filtering, we first discard queries whose true positive video did not exist within the first 1000 candidates returned from OmniEmbed 24 . Next, we removed queries whose top first hard negative had a first stage score more then 2x the size of the true positive. Finally, we discarded query video pairs wrongly classified by ReasonRank -32B, supplying video captions as evidence for judgment. Our filtered dataset contains 35684 records, with 9267 unique positive query-video pairs and 26258 negative query-video pairs. On average each query has 3.85 candidates: 7995 queries have 3 negative samples, 1014 queries have 2 negatives samples, 245 queries have 1 negative sample and 13 queries have 0 negative samples. 4 RankVideo Two-Stage Training Figure 2: RankVideo is trained with a two-stage process. Stage 1 uses a perception-grounded supervised finetuning, where the model learns to generate captions grounded in video content. In Stage 2, for each text query, we sample a query grouped batch containing one positive (relevant) video and one or more negatives (not relevant), and score each candidate using the difference between the logits for yes and no. The model is optimized with a combined objective: (1) teacher-probability distillation toward p_{yes} , (2) a pointwise loss for stable binary calibration, and (3) pairwise ranking loss that pushes the positive to the top within the query batch. With our training data, a mixture of the human-written and synthetic queries, we are able to reasonably train a video-native reranker. Our training approach is composed of two stages: (1) perception grounded cold-start supervised SFT and (2) teacher guided ranking optimization. 4.1 Stage 1: Perception Cold Start SFT An effective video native reranker must be able to reliably extract grounded evidence from raw video, including objects, actions and event context, and then align that evidence with a query. Motivated by recent findings that separating perception from downstream reasoning can improve multimodal training stability and performance 6 , we introduce a perception grounded ‘cold start’ stage before applying any ranking-specific supervision. In this stage, we supervise the model to generate teacher provided captions for videos. Captioning provides a direct and dense learning signal for video understanding. Rather than only learning a binary relevance decision, the model is trained to produce an explicit textual description of salient entities present in the clip. Our intuition is that this encourages the model to attend to discriminative visual content. Training Data Construction We use teacher captions generated offline for the training videos. To bound compute and avoid repeatedly captioning near-duplicate candidates, we restrict this stage to one video per query so that the captioning objective covers a broad set of unique events rather than many candidates for the same query. Objective Let v denote a video and c^{(T)}=(c^{(T)}_{1},\dots,c^{(T)}_{L}) its teacher caption token sequence. We fine-tune the model parameters \theta to maximize the likelihood of the teacher caption conditioned on the video. \mathcal{L}_{\mathrm{cap}}\;=\;-\sum_{t=1}^{L}\log p_{\theta}\!\left(c^{(T)}_{t}\mid c^{(T)}_{\alpha_{2} . All samples that meet the \alpha_{2} threshold are dropped to reduce false negative contamination. See Appendix A for details on threshold selection. • Hard negatives, consisting of the remaining candidates. We retain ambiguous negatives because they closely resemble the positive under the first stage retriever and thus dominate reranking errors, (see Appendix D ). Training In Stage 2 we train the model to judge query video relevance using a three part objective: pointwise accuracy, pairwise ranking ability, and teacher distilled confidence. Given a query q and a candidate video v , the model is prompted to answer yes or no using a structured output format yes/no . Rather than relying on generated text, we compute a scalar relevance score from the model’s logits at the decision point: s_{\theta}(q,v)\;=\;\ell_{\theta}(\texttt{yes}\mid q,v)\;-\;\ell_{\theta}(\texttt{no}\mid q,v), (2) where \ell_{\theta}(t\mid q,v) denotes the model logit for token t at the decision position after the tag. The logit delta score provides a stable, monotonic ranking signal, and enables fast scoring without decoding long rationales. Training proceeds on query grouped mini batches. For each query q , we sample a candidate set \mathcal{B}_{q}=\{(q,v_{i})\}_{i=1}^{K+1} containing one positive v^{+}=v^{*}(q) and K negatives \{v^{-}_{1},\dots,v^{-}_{K}\} from the same query. 2 2 2 In our experiments we use K=2 . Now let s_{i}=s_{\theta}(q,v_{i}) be the score for each candidate in \mathcal{B}_{q} . Pairwise ranking loss We define a softmax distribution over candidates within the query batch: p_{i}\;=\;\frac{\exp(s_{i}/\tau_{pair})}{\sum_{j}\exp(s_{j}/\tau_{pair})}, (3) and optimize a batch wise objective that pushes the positive to the top of its query group: L_{pair}=-\log p_{+} In Equation 3 , we use \tau_{pair} to prevent early saturation and vanishing gradients as score gaps grow. 3 3 3 We use \tau_{pair}=10 in our experiments. Teacher probability distillation Next, we distill a teacher-provided relevance probability using a temperature-scaled binary cross-entropy with logits (BCEL) on the same score: \mathcal{L}_{\mathrm{t}}\;=\;\mathrm{BCEL}\!\left(\frac{s_{\theta}(q,v)}{\tau_{\mathrm{teacher}}},\;p^{(T)}_{\mathrm{yes}}(q,v)\right), (4) This transfers calibrated confidence beyond the binary label and helps align scores across queries. Pointwise loss To stabilize training under class imbalance and provide pointwise supervision, we add a calibration loss with softened negative targets to account for noise in our training data. Let y\in\{0,1\} be the binary relevance label and define \tilde{y}=\begin{cases}1.0&\text{if }y=1,\\ 0.1&\text{if }y=0,\end{cases}\qquad\hskip-20.00003ptw=\begin{cases}1.0&\text{if }y=1,\\ 0.5&\text{if }y=0.\end{cases} (5) \mathcal{L}_{\mathrm{pt}}\;=\;\mathrm{BCEL}\!\left(\frac{s_{\theta}(q,v)}{\tau_{\mathrm{point}}},\;\tilde{y};\text{weight}=w\right), (6) Combining Pairwise ranking loss , Teacher probability distillation and Pointwise loss , the final Stage 2 loss is a weighted sum 4 4 4 We set \lambda_{teacher}=5 and \lambda_{pt}=0.5 : \mathcal{L}\;=\;\mathcal{L}_{\mathrm{pair}}\;+\;\lambda_{\mathrm{teacher}}\,\mathcal{L}_{\mathrm{t}}\;+\;\lambda_{pt}\mathcal{L}_{\mathrm{pt}}. (7) 5 Experiments Evaluation Setup We evaluate on the MultiVENT 2.0 19 test set, consisting of 109,800 videos. For evaluation, we retrieve the top 1000 candidates per query from the first stage retriever. We then score all candidates with the reranker report recall at 10 (R@10), 20 (R@20), 50 (R@50), and 100 (R@100), and normalized discounted cumulative gain (nDCG@N) for the same cutoffs as recall. For all metrics, a higher number indicates better performance. Method R@10 nDCG@10 R@20 nDCG@20 R@50 nDCG@50 R@100 nDCG@100 OE 0.523 0.495 0.598 0.524 0.690 0.556 0.749 0.572 RR 0.570 0.543 0.694 0.585 0.764 0.608 0.800 0.619 8.99 9.70 16.05 11.64 10.72 9.35 6.81 8.22 QVL-I 0.508 0.478 0.60 0.513 0.692 0.544 0.748 0.56 -2.87 -3.43 0.33 -2.10 0.29 -2.16 -0.13 -2.10 QVL-T 0.515 0.483 0.610 0.519 0.719 0.555 0.784 0.572 -1.53 -2.42 2.01 -0.95 4.20 -0.18 4.67 N/A QVL-R 0.537 0.465 0.663 0.517 0.733 0.542 0.749 0.546 2.68 -6.06 10.87 -1.34 6.23 -2.52 N/A -4.55 RV-1 0.582 0.559 0.661 0.589 0.727 0.612 0.752 0.619 11.28 12.93 10.54 12.40 5.36 10.07 0.40 8.21 RV-2 0.590 0.566 0.682 0.599 0.769 0.630 0.820 0.644 12.81 14.34 14.05 14.31 11.45 13.31 9.48 12.59 Table 1: Performance changes from OmniEmbed first-stage retriever on MultiVENT 2.0 . Each method reports raw scores (top) and deltas as a percentage relative to OmniEmbed (bottom). Green denotes an increase in performance, while Red denotes a decrease in performance. OE: OmniEmbed , RR: ReasonRank , QVL-I: Qwen3-VL-8B-Instruct , QVL-T: Qwen3-VL-8B-Thinking , QVL-R Qwen3-VL-Reranker-8B , RV-1/2: RankVideo Stage 1/2. Results Setup Our main results are reranked from OmniEmbed 24 , a dense retrieval model built on Qwen2.5 Omni 45 , as the first-stage model. We also explore using four other first-stage systems: (1) MMMORRF 37 a multimodal rank fusion approach; (2) CLIP 33 with 16 key frames selected by PySceneDetect 4 ; (3) LanguageBind 57 using connected language-vision encoders; (4) and Video-ColBERT 35 a multi-vector late interaction model. For all first-stage retrievers, we rerank top 100 candidates using a first-stage depth of 1000. Along with RankVideo , we evaluate four other baseline rerankers: (1) ReasonRank 23 a text-based reasoning reranker, which reranks the captions and audio transcripts of the video produced by Qwen3-Omni-30B-A3B-Instruct 2 and Whisper-Large-v2 34 ; (2) Qwen3-VL-8B-Instruct ( 2 , QVL-I,) , a frontier video understanding model; (3) Qwen3-VL-8B-Thinking ( 2 , QVL-T,) , the reasoning variant of QVL-I; and (4) Qwen3-VL-Reranker-8B ( 20 , QVL-R,) , the reranking variant of Qwen3-VL . Implementation Details Our base model is intialized from Qwen3-VL-8B-Instruct ( 2 , QVL-I,) . For stage one of training, we use one sample per query in order to generate a single unique caption per video, resulting in a training size set size of 9267. We fix the frame rate at 2 frames per second (FPS), with a maximum 32 frames per video, and train with a learning rate of 1e-5 and batch size of 16. In stage two of training, we form mini-batches of one positive and two negative queries. This yields a dataset of 7995 distinct queries with 23985 total videos. The final data mixture used to train our model contains 1361 human written queries and 7906 synthetic queries. 5.1 RankVideo and Baselines In Table 1 , we show the results of the reranking methods on MultiVENT 2.0 with OmniEmbed (OE) as the first-stage retriever. We see that RankVideo achieves state-of-the-art retrieval results across all metrics, increasing the change in metric performance significantly more than any other reranking baseline. Both Stage-1 and Stage-2 of RankVideo are able to significantly increase reranking performance, with the first and second best results across each metric. We find that the only other method to increase performance from the first stage results is ReasonRank , with strong gains over the first-stage results. The two other video-native baselines are not able to increase performance, struggling to judge the relevance of query-video pairs. QVL-I is unable to improve upon the first stage results. QVL-T is able to improve the recall from a cutoff of 20 and above, but unable to improve nDCG. This result means that QVL-T is able to remove easy negatives (lower in the ranked list), but struggles to rank relevant videos highly, especially against harder negatives in the first-stage results. These results demonstrate three core findings. (1) Vision language or reasoning models (QVL-I/T) are not able to effectively judge the relevance of videos in zero-shot. This finding largely aligns with hallucination calibration literature 21 ; 13 , as we find QVL-I/T struggle with low precision as a result of a high false positive rate (QVL-I precision 0.055, QVL-T precision 0.037). (2) The gain in performance of RankVideo over ReasonRank demonstrates the importance of performing video-native reranking, incorporating heterogeneous multimodal signals into relevance judgments instead of extracting captions and transcripts, which are not information exhaustive. (3) Reranking models trained for contrived video-retrieval tasks 5 5 5 Video retrieval tasks that are captioning datasets converted to retrieval datasets do not generalize well to real-world video retrieval ( 19 , mirroring the first stage findings of ) . When comparing to QVL-R, we see the most significant decrease in performance from the first stage results, even compared to models not calibrated for reranking. Figure 3: Training Stage-2 increases score separation in the reranking regime. Empirical CDF of the reranker score s_{\theta}(q,v)=\ell_{\theta}(\textbf{yes}|q,v)-\ell_{\theta}(\textbf{no}|q,v) for relevant and non relevant query video pairs. Stage 2 shifts relevant pairs towards larger positive margins and suppresses non relevant candidates towards more negative margins, reducing overlaps in the score distributions within reranking candidate pools. 5.2 Score Distribution Shift RankVideo ranks candidates using logit delta score which provides a monotonic scalar ranking signal without decoding long rationales. Because reranking occurs over the first state top-1000 candidate pool, the dominant error mode is high scoring false positives among hard negatives. Figure 3 shows that Stage 2 training reshapes the score distribution in a label consistent way: relevant pairs shift toward substantially larger positive margins, while non relevant candidates are pushed towards more negative margins, increasing separation and reduction overlap. This distribution shift is key for improving rank quality, especially in early ranks, since top-k metrics are most sensitive to the impact of high scoring hard negatives outranking the true item. This results suggest training does not just improve point wise correctness, but also calibrates the scores to match the reranking objective. Additionally, the increased separation suggests that the logit margin score can serve as a more reliable confidence signal. Method R@10 nDCG@10 R@20 nDCG@20 R@50 nDCG@50 R@100 nDCG@100 CLIP 0.333 0.306 0.419 0.339 0.522 0.373 0.603 0.394 + RV 0.477 0.478 0.540 0.503 0.584 0.520 0.603 0.525 43.24 56.21 28.88 48.38 11.88 39.41 N/A 33.25 MRF 0.611 0.585 0.697 0.620 0.779 0.649 0.828 0.663 + RV 0.634 0.639 0.725 0.639 0.800 0.665 0.828 0.673 3.76 8.45 4.02 3.06 2.70 2.47 N/A 1.51 LB 0.355 0.326 0.445 0.359 0.550 0.393 0.620 0.412 + RV 0.498 0.487 0.560 0.512 0.607 0.530 0.620 0.533 40.28 49.39 25.84 42.62 10.36 34.86 N/A 28.37 VCB 0.341 0.422 0.402 0.447 0.471 0.477 0.518 0.494 + RV 0.448 0.535 0.490 0.549 0.514 0.560 0.518 0.561 31.38 26.78 21.89 22.82 9.13 17.40 N/A 13.56 Table 2: Impact of RankVideo applied to different first-stage retrievers on MultiVENT 2.0 . Each method reports raw retrieval performance (top) and absolute deltas as a percentage relative to its corresponding baseline retriever (bottom). Green indicates an improvement in ranked list quality. CLIP: CLIP on 16 frames, MRF: MMMORRF, LB: LanguageBind, VCB: Video-ColBERT, RV: RankVideo . 5.3 Stability Across First-Stage Model In Table 2 we explore the stability of RankVideo across the other first-stage retrieval methods on MultiVENT 2.0 . Like OmniEmbed, we observe a significant gain in performance from first-stage to second-stage results for all first-stage methods. On a strong first stage retriever (MMMORRF), we continue to see solid gains between ranked lists. Most promisingly, we see the largest and most significant gains (>0.1) on the weakest first-stage retrievers. Figure 4: Median query latency for Qwen3VL Instruct/Thinking (QVL-I/T) and RankVideo stages 1 and 2. Latency is computed as the mean for 100 query-video pairs with a batch size of 1. All evaluations are run with batch size 1 as larger batches exceed GPU memory for VLMs. These results demonstrate that RankVideo is beneficial to any first-stage retriever. Additionally, RankVideo is not dependent on the quality of the initial candidate list, allowing for the use of faster, more efficient (although less accurate) first-stage models to handle large indices and rely on RankVideo to refine the ranked list. 5.4 Efficiency and Dynamic Reasoning In Figure 4 we compare the query latency of ReasonRank, QVL-T/I/R and RankVideo Stages 1 and 2 (RV-1/2) on a randomly selected subset of 100 query-video pairs from MultiVENT 2.0 , reporting the median query latency on these 100 instances. 6 6 6 We chose 100 instances to match the reranking cutoff. We find that RankVideo is much more efficient than the alternative baseline video-native reasoning model (QVL-T), with a difference of 2.67s between RankVideo Stage 2 and QVL-T. We attribute the large difference in query latency because QVL-T produces a reasoning trace for every query-video pair. In Figure 4 , we observe that RankVideo performs within 0.15s of ReasonRank (1.02s vs. 0.87s). We include a preprocessing latency penalty for ReasonRank to contextualize the offline compute costs for captioning. While our model does require captions for training, at inference, RankVideo ’s reliance soley on video frames avoids expensive preprocessing latency. This comes at the trade off of only a small gap in model latency vs. text based rerankers (ReasonRank). ReasonRank’s latency values relative to all other models should be interpreted as an amortized per candidate estimate, since ReasonRank is listwise and evaluates a sliding window of candidates simultaneously. Method \boldsymbol{\alpha} - nDCG nDCG StRecall InfoP CLIP 0.538 0.498 0.724 81.5 +RV 0.628 0.629 0.826 91.9 LB 0.476 0.457 0.634 84.0 +RV 0.566 0.563 0.754 94.5 VCB 0.431 0.383 0.634 83.8 +RV 0.553 0.522 0.734 87.1 OE 0.530 0.454 0.721 88.0 +RV 0.584 0.587 0.778 91.2 MRF 0.540 0.503 0.724 94.4 +RV 0.605 0.617 0.785 91.8 Table 3: Retrieval and generation results on WikiVideo . The retrieval results are calculated using claim-based relevance and a cutoff at 10. 5.5 Downstream Impact in RAG We explore the effect of RankVideo in a retrieval augmented generation setting. We perform RAG with the WikiVideo dataset 27 , evaluating retrieval with three claim-based relevance judgments \alpha -nDCG, nDCG, and StRecall, where the number of supported article claims are used for document relevance; evaluating retrieval with MiRAGE 28 , reporting Information Precision (InfoP) for article factuality; and generating articles using CAG with a Qwen-3-VL backbone 2 ; 27 . In Table 3 , we find that RankVideo substantially improves upon the claim coverage of the top 10 videos provided to the generation system, which leads to a large increase in article factuality. This demonstrates the effectiveness of RankVideo , not only as an effective reranker for traditional retrieval metrics, but also as a crucial step in a multimodal RAG pipeline, increasing diversity of information for generation. Query Before \downarrow After \downarrow \Delta\downarrow KAMAZ Autonomous Mining Dump Truck 100 1 -99 Korea Blockchain Week 2022 98 5 -93 2018 Mark Zuckerberg senate hearing 92 1 -91 Sixth SpaceX operational mission 93 2 -91 2019 Cairo International Film Festival 2 1 -1 2019 Moscow International Film Festival 1 1 0 Digital Services Act 1 1 0 2019 Typhoon Lekima 2 5 +3 Typhoon Maysak 2 6 +4 2018 opinion rigging scandal in South Korea 11 44 +33 Table 4: Most improved and degraded queries after reranking. Before: rank without reranking. After: rank with reranking. \Delta : change in rank (After - Before); lower is better. The maximum rank is capped at 100. 5.6 Qualitative Results Discussion We analyzed the failure modes of RankVideo by examining query performance grouped by metadata attributes. Overall, we observe relatively stable performance across, event types, modalities and some performance stratification across languages (see Appendix C ). To quantify whether coarse metadata can explain which queries the reranker performs poorly on, we fit a regression model to predict per query nDCG@10 using video language , query event type , video type , video modality , and query length . We randomly split queries into train (75%) and test (25%) and train a Random Forest Regressor. On held out queries, the model achieves R^{2}=0.093 , indicating that these coarse metadata features explain only a small fraction of variance in per query nDCG@10 . Feature importances suggest query length is the most informative single feature, consistent with the intuition that shorter/underspecified queries are more difficult. Separately, we test whether RankVideo ’s ranking scores exhibit query or video specific priors by decomposing variance in the reranker score s(q,v) . Let N_{q} denote the number of scored candidates for query q , and define the query mean score \mu_{q}=\frac{1}{N_{q}}\sum_{v}s(q,v) . Let N_{v} denote the number of queries for which video v is scored, and define the per video mean score \mu_{v}=\frac{1}{N_{v}}\sum_{q}s(q,v) . We evaluate three simple predictors of s(q,v) : (i) query only \hat{s}(q,v)=\mu_{q} , (ii) video only \hat{s}(q,v)=\mu_{v} , and (iii) additive \hat{s}(q,v)=\mu_{q}+\mu_{v}-\mu where \mu is the global mean. The resulting R^{2} values are 0.139 (query only) and 0.090 (video only), indicating that RankVideo ’s scoring is not dominated by a video level prior and instead depends substantially on query–video interaction. In contrast, baseline rerankers exhibit stronger video priors (e.g., QVL-R : R^{2}=0.755 ). In Table 5 we see the broader trend that the weaker performing re-ranker also appear to have strong modality bias. Qualitatively, we attribute the performance of RankVideo to its effectiveness with videos and queries that have visually anchorable events. Consider, for example, the queries that see that greatest improvements in Table 4 . Mining dump trucks and SpaceX operational mission are object specific and have discrete visuals (trucks and rockets). On the other side, we attribute the negative performance of the other vision baselines to non-visual or weakly visual topics, like "2018 opinion rigging scandal in South Korea." One confounding category of events the model appears to occasionally struggle with, despite intuition suggesting their ought to be an abundance of distinct visually grounded signals, is natural disasters. A plausible explanation is, storm footage, for example, may share more generic visuals (rushing water) rather than unique cues needed for quality reranking, leading to negative results. Method Query Only Video Only Q+V QVL-I 0.192 0.119 0.262 RR 0.033 0.105 0.130 QVL-T 0.295 0.265 0.431 QVL-R 0.108 0.755 0.739 RV 0.139 0.090 0.200 Table 5: Query/video prior strength via variance decomposition. Stronger biases underlined. 6 Conclusion In this work, we introduce RankVideo , a novel approach for video-native reasoning reranking. RankVideo is trained with a two stage process. The first stage perception-grounded SFT teaches the model to generate captions grounded in video content, while the second stage finetunes the stage 1 model for effective reranking by combining pairwise, pointwise, and distillation objectives. On MultiVENT 2.0 , a large scale video retrieval task, RankVideo consistently enhances retrieval performance across various first-stage retrievers, achieving an average improvement of 31% on nDCG@10. Not only is RankVideo the most effective reranker, but it is also significantly faster than existing reasoning-based reranking baselines. Future work in video reranking should look towards new training objectives, like list-wise or grouped reranking which exhibit strong performance gains over their pointwise counterparts in text IR 23 ; 38 ; 50 . Additionally, exploring further optimizations for dynamic reasoning, the disconnect between accuracy and retrieval performance, and improving performance on reasoning-intensive or ambiguous video types suggest ample direction for future work. In applications, reranking and modified objectives could be explored for retrieval-augmented generation to provide better optimized results to article generation models. Limitations Computational Costs In this work, we did not explore list-wise reranking, even though it’s benefit over pointwise is well supported in the literature, because of the computational costs of multivideo inference. To train our pairwise objective (at most 3 videos per query), required greatly reducing the batch size and max frames to fit on 8 80GB A100s for training. Efforts towards making multi-video inference more computationally feasible will help reduce this burden. Appendix A RankVideo Details We provide additional details about our training configurations in Table 6 and provide the system and user prompts for RankVideo ( Figure 11 , Figure 12 ), QVL-I/T ( Figure 13 , same as RankVideo ), and ReasonRank ( Figure 14 ). For hard negative mining, we used thresholds \alpha_{1}=-6 and \alpha_{2}=-8 after reviewing the distribution of logit scores of all query, video pairs. Setting Value Learning Rate Schedule cosine Learning Rate 1e-5 Warmup Proportion (Linear) 0.03 Optimizer AdamW Frame rate (FPS) 2 Stage 1 Data 9267/9267 Stage 1 Batch Size 16 Stage 1 Epochs 1 Stage 1 Max Frames 32 Stage 2 Data 7995/23985 Stage 2 batch Size 3 Stage 2 Epochs 2 Stage 2 Max Frames 24 Table 6: Training settings for RankVideo . Stage 1/2 Data is written as queries/videos. Appendix B Training Loss Ablation Method R@10 nDCG@10 R@20 nDCG@20 R@50 nDCG@50 R@100 nDCG@100 OE 0.554 0.541 0.614 0.566 0.711 0.599 0.755 0.614 P, PT, T 0.650 0.640 0.694 0.657 0.731 0.672 0.755 0.678 P, PT 0.650 0.630 0.687 0.644 0.733 0.663 0.755 0.668 P 0.630 0.620 0.668 0.634 0.705 0.649 0.755 0.661 Table 7: Performance changes from OmniEmbed first-stage retriever on MultiVENT 2.0 . OE: OmniEmbed ; P, PT, T: Our full loss objective; P, PT: Our loss objective without teacher distillation; P: Pairwise loss. We include the ablation results of our three part loss objective in Table 7 to understand which signals are responsible for reranking gains. This ablation includes our pairwsie ranking loss (P), pointiwse calibration loss (PT), and teacher probability distillation (T). We report results on a 50 query subset of the MultiVENT 2.0 test set. We observe substantial gains from the inclusion of the pointwise objective. The inclusion of the teacher probability distillation term resulted in an improved nDCG. Appendix C RankVideo Performance Across Query Types To better understand whether our gains are concentrated in a subset of the evaluation distribution, we break down RV retrieval performance across several metadata slices. Specifically, we report per query nDCG@10 aggregated by video language ( Figure 6 ), query event type ( Figure 5 ), video type ( Figure 7 ), and video modality ( Figure 8 ). Figure 5: RankVideo nDCG@10 by query event type. Multi-word categories are abbreviated as acronyms in the plot (e.g., PD=Political Development, LD=Launch/Discovery, SE=Social Events). Only attributes with \geq 30 test queries are included. Figure 6: RankVideo nDCG@10 by video language (en=English, es=Spanish, ar=Arabic, zh=Chinese, ru=Russian, ko=Korean). Only attributes with \geq 30 test queries are included. Figure 7: RankVideo nDCG@10 by video type. Professional: e.g., news broadcasts with reports; Edited: e.g., videos with multiple spliced clips and visual effects; Diet Raw (DR): single-stream videos with minimal text/speech overlays; Raw: continuous, unedited streams. Only attributes with \geq 30 test queries are included. Figure 8: RankVideo nDCG@10 by query/video modality. OCR=queries written from text visible in video frames; Text=queries written using only the YouTube description; Base=Wikipedia-title-style queries; Speech=queries written from spoken content; Specific=queries targeting fine-grained aspects of events. Only attributes with \geq 30 test queries are included. Appendix D Disconnect Between Binary Classification and Reranking Method Accuracy R@10 nDCG@10 OE - 0.52 0.50 Neg-1 0.74 0.48 0.52 Neg-2 0.76 0.52 0.54 Neg-3 0.79 0.54 0.54 Table 8: Evidence of the disconnect between accuracy and reranking performance. Neg: negative result while training RankVideo . One failure mode we observed during our training is a disconnect between binary classification 7 7 7 In this case, yes/no judgments for the relevance of a query-video pair. and reranking quality. During development, multiple models achieved strong accuracy and recall ( Table 8 ) when evaluated as relevance classifiers. However, these metrics are not indicators of strong second-stage results because they largely reflect performance on easy negatives. Classifiers blindly trained to increase accuracy end up being insensitive to the error regime that dominates second-stage retrieval. When reranking, the model is only given the top-k candidates returned by a first-stage retriever, where negatives are hard—semantically or visually plausible near-misses—and therefore disproportionately likely to become high-scoring false positives. While accuracy and reranking performance are slightly positively correlated, we observe that higher accuracy does not always mean strong reranking results. To address this, we designed our training supervision to better match the reranking objective by mining negatives from each query’s candidate pool and filtering them using teacher confidence. Specifically, we removed likely false negatives by dropping suspected positives. We thresholded suspected positives as non-qrels candidates that the teacher labeled as relevant with high margin. Among the remaining candidates, we retained trusted negatives that the teacher rejected with a large negative margin, and ambiguous negatives that remain difficult but are flagged with lower confidence. We required at least one trusted negative per query. This curation shifts learning away from separating irrelevant videos and toward suppression of retrieval hard negatives while avoiding noisy supervision from unlabeled positives. Appendix E Reasoning Examples In Figure 9 and Figure 10 we provide examples of reasoning traces from the baseline and RankVideo rerankers. Figure 9 shows an example where RankVideo did not need to reason to produce a correct answer with a query-video pair, while Figure 10 shows an example where RankVideo needed to reason. In the example with a reasoning trace, the large difference in reasoning length between Qwen3-VL-8B-Thinking and RankVideo is evident. Appendix F Use of AI Assistants AI assistants were used to improve the fluency of the writing and in some of the code development. Query: Emergency response Notre-Dame fire ReasonRank ✗ Pred: no |  Gold: yes The video shows interior damage to Notre-Dame after the fire. While related to the event, it does not depict firefighters or emergency responders. Since the query is about emergency response actions, the video is judged not relevant. no Qwen3-VL-8B-Thinking ✗ Pred: no |  Gold: yes The video depicts post-fire damage inside Notre-Dame Cathedral. No emergency response activities (e.g., firefighters or rescue operations) are visible. Therefore, the video is not relevant to the emergency response query. no Captioning model The footage shows catastrophic damage to Notre-Dame Cathedral following a deadly fire. Debris and structural collapse are visible inside the building. The cause of the fire remains under investigation. RankVideo ✓ Pred: yes |  Gold: yes yes Figure 9: Reasoning traces for the query “Emergency response Notre-Dame fire” . ReasonRank and Qwen3-VL-8B-Thinking models are substantially more verbose then our model. Our final model correctly predicts relevance. Video ID: 45391 Query: Super Bowl 2023 Philadelphia Eagles ReasonRank Pred: no |  Gold: no Okay, let me try to figure this out. The query is about the Super Bowl 2023 Philadelphia Eagles. So, the user wants to know if the video is relevant to that. Looking at the video content, it starts with a close-up of Richard Sherman from the 49ers. Then there’s info about his salary from 2020-2022. Next, there’s game footage of him tackling someone in a yellow uniform (maybe another team?). Then he’s celebrating on the field. The video ends with a blue screen and social media stuff. Now, the query mentions Philadelphia Eagles and Super Bowl 2023. The video mostly talks about Richard Sherman’s salary and some game play. There’s no mention of the Eagles or the 2023 Super Bowl. The focus seems to be on Sherman’s career stats and a play. Since the query is specifically about the Eagles in the 2023 Super Bowl, and the video doesn’t cover that, I think it’s not relevant. Unless maybe the Eagles are involved in the footage, but the caption says Sherman tackles someone in yellow, so probably not Eagles. So the answer should be no. no Qwen3-VL-8B-Thinking Pred: no |  Gold: no Got it, let’s check the query: Super Bowl 2023 Philadelphia Eagles. Now, look at the video content. The video shows a player in a San Francisco 49ers uniform (jersey number 20, helmets with 49ers logo), social media prompts, salary info for 2020-2022, game action with 49ers, and other teams like Eagles? Wait, let’s see. The video has players from the 49ers, and there’s a part where the team is in red and gold, which is 49ers. The query is about Super Bowl 2023 Philadelphia Eagles, but the video seems to focus on the 49ers, not the Eagles. Also, Super Bowl 2023 isn’t actually a thing yet (2023 would be Super Bowl LVII, but the Eagles’ Super Bowl is like 2023 vs the Chiefs maybe? Wait, but the video doesn’t mention Philadelphia Eagles or Super Bowl 2023. Let’s check all segments. The video starts with a 49ers player, then salary info, then a game action (49ers), then another team (Patriots vs Seahawks?), then a thank you screen. There’s no mention of Philadelphia Eagles or Super Bowl 2023. The query is about Eagles in 2023 Super Bowl, but the video is about 49ers, so it’s not relevant. no Captioning model A San Francisco 49ers football player in his white jersey with the number 20 is shown on the field. The video then displays a financial breakdown of his contract for the years 2020, 2021, and 2022, detailing his base salary, guaranteed amounts, bonuses, and total salary-cap hits. Action shots from games are shown, including a player celebrating with arms outstretched. The video concludes with an outro screen thanking viewers and asking them to subscribe. RankVideo ✓ Pred: no |  Gold: no The video is not relevant to the query. The query asks about Super Bowl 2023 and the Philadelphia Eagles, but the video content shows a San Francisco 49ers player (number 20) in a Super Bowl LVII jersey, along with salary cap information for the years 2020-2022, and other football-related images. There is no mention or depiction of the Philadelphia Eagles or Super Bowl 2023 specifically. no Figure 10: Reasoning traces for the query “Super Bowl 2023 Philadelphia Eagles” . In this instance, all models correctly classify the video as not relevant but we see ReasonRank and Qwen3-VL-8B-Thinking use substantially more tokens then RankVideo . Video ID: 45391 [System Prompt] You are a helpful assistant specialized in video and text understanding. Given a video, your task is to produce an accurate caption. Respond within [User Prompt] Caption this video. Respond within . Figure 11: RankVideo prompt for stage 1 [System Prompt] You are a helpful assistant specialized in video and text understanding. Given a text query and a video, your task is to determine if the video is relevant to the query. Respond with yes if the video is relevant, or no if it is not. [User Prompt] Query: {query} Is the video relevant to the query? Respond with yes or no. Figure 12: RankVideo prompt for stage 2 [System Prompt] You are a helpful assistant specialized in video and text understanding. Given a text query and a video, your task is to determine if the video is relevant to the query. Respond with yes if the video is relevant, or no if it is not. [User Prompt] Query: {query} Is the video relevant to the query? Respond with yes or no. Figure 13: Prompt for the QwenVL Models [System Prompt] You are RankLLM, an intelligent assistant that can rank passages based on their relevance to the query. Given a query and a passage list, you first thinks about the reasoning process in the mind and then provides the answer (i.e., the reranked passage list). The reasoning process and answer are enclosed within and tags, respectively, i.e., reasoning process here answer here . Figure 14: Prompt for the ReasonRank