A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender sys
Mert Can Cakmak, Nitin Agarwal · 2025
Read A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender sys by Mert Can Cakmak, Nitin Agarwal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02543.txt.
Introduction
Mert Can Cakmak and Nitin Agarwal present a 2025 computational social science paper on auditing bias in YouTube Shorts recommendations. Short video feeds shape what billions see, yet the ranking systems stay opaque, and small shifts can skew exposure on sensitive topics.
Instead of full videos or metadata, the authors extract perceptually salient keyframes, the still images that carry a clip's visual gist. Captions are generated for those frames, and both are embedded into one content space. Mapping recommendation chains then reveals drift and clustering patterns. Politically sensitive topics behaved differently from general categories, with consistent shifts suggesting filtering effects. The authors present keyframes as a cheap, readable lens for auditing short-form recommendation systems.
Catalog record
| CREATOR | Mert Can Cakmak, Nitin Agarwal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Information systems Recommender systems Mert Can Cakmak, Nitin Agarwal Abstract . YouTube Shorts and other short-form video platforms now influence how billions engage with content, yet their recommendation systems remain largely opaque. Small shifts in promoted content can significantly impact user exposure, especially for politically sensitive topics. In this work, we propose a keyframe-based method to audit bias and drift in short-form video recommendations. Rather than analyzing full videos or relying on metadata, we extract perceptually salient keyframes, generate captions, and embed both into a shared content space. Using visual mapping across recommendation chains, we observe consistent shifts and clustering patterns that indicate topic drift and potential filtering. Comparing politically sensitive topics with general YouTube categories, we find notable differences in recommendation behavior. Our findings show that keyframes provide an efficient and interpretable lens for understanding bias in short-form video algorithms. 1. Introduction Short-form video platforms like YouTube Shorts have rapidly reshaped online content consumption, offering users a continuous, swipe-driven stream of personalized video recommendations. Since its global launch in 2021, YouTube Shorts has grown to over two billion monthly users, making it one of the most influential content delivery systems worldwide. As algorithms decide what users see next, even small recommendation biases can significantly affect exposure to underrepresented or sensitive topics. Prior studies have highlighted ideological skew, homogenization, and popularity bias in recommender systems, often through metadata or user interaction data. However, these approaches overlook the actual visual content users consume, and analyzing entire videos at scale is computationally intensive. This challenge is especially pronounced in short-form ecosystems where content changes rapidly. To address this, we introduce a keyframe-based method for analyzing recommendation behavior in YouTube Shorts. Using the PRISM framework, we extract perceptually important keyframes from both seed and recommended videos. We then use CLIP to embed these frames, along with captions generated using the Llama-3.2-11B-Vision-Instruct model, and project them into a shared space using UMAP. This allows us to observe how recommendations visually and thematically shift from the original content. We collect data across two domains: videos related to the politically sensitive “South China Sea” (SCS) and a control group sampled from general YouTube video categories. The SCS topic is frequently subject to geopolitical framing and content moderation, making it a meaningful case for studying potential recommendation bias. In contrast, general content provides a baseline for understanding typical recommendation behavior. This contrast allows us to assess whether algorithmic dynamics differ when political sensitivity is involved. Our study is guided by the following research questions: • RQ1: Do YouTube Shorts recommendations visually or thematically shift away from their seed content? • RQ2: Can keyframes and their captions reveal patterns in recommended content that suggest potential bias or filtering? • RQ3: Does the degree of content shift differ between politically sensitive topics and general categories? Our findings show that keyframes provide a focused and interpretable lens for studying short-form recommendations. We observe measurable divergence and clustering, with more pronounced shifts in sensitive-topic cases. This work contributes a scalable, content-driven method for auditing algorithmic behavior in modern video platforms. 2. Literature Review YouTube’s recommendation system, powered by deep neural networks, optimizes for predicted watch time but has raised concerns about various biases and drifts Covington et al. 2016 . These include selection, popularity, and position bias, as well as evolving issues like recommendation drift, algorithmic drift, and content drift Groh et al. 2012 ; Chen et al. 2023 ; Garapati & Chakraborty 2025 . Empirical audits reveal ideological and emotional skews, right-leaning accounts receive more radical content, while emotionally negative or morally charged videos are filtered out deeper in the chain Haroon et al. 2022 ; Ledwich & Zaitsev 2019 ; Habib & Nithyanand 2025 . For sensitive topics such as China–Uyghur, recommendations increasingly favor positive sentiment content, suppressing moral critique Cakmak et al. 2024 ; Cakmak et al. 2023 . Network analyses show that recommendation structures reinforce thematic silos, limiting content diversity Kirdemir & Agarwal 2021 ; Kirdemir et al. 2021 ; Kirdemir et al. 2021a . Short-form platforms like YouTube Shorts and TikTok intensify these issues. Studies find rapid content drift from controversial seeds to apolitical or visually neutral recommendations Cakmak & Agarwal 2025 . TikTok audits reveal partisan bias favoring Republican content Ibrahim et al. 2025 , while Kuaishou exhibits duration bias favoring longer videos Zheng et al. 2022 ; Zhan et al. 2022 ; Lin et al. 2023 . Algorithmic feedback loops further reinforce mainstream narratives, reducing minority exposure Mansoury et al. 2020 . Mitigation strategies include re-ranking, causal modeling, and adversarial learning to address ideological, temporal, and duration-based biases in recommendation pipelines Haroon et al. 2022 ; Zhan et al. 2022 ; Zheng et al. 2022 . However, most existing efforts rely on full video analysis or engagement traces, with limited focus on visual-semantic dynamics in short-form formats like YouTube Shorts. We present the first keyframe-based approach of bias and drift in YouTube Shorts. Unlike prior work that processes full videos or relies solely on metadata, we extract perceptually salient keyframes as concise visual summaries. These serve as effective proxies for comparing content across recommendation chains, revealing shifts in visual framing and content representation. This approach enables a faster and more focused analysis of recommendation behavior in short-form video systems. 3. Data Collection To examine potential algorithmic and recommendation biases in YouTube Shorts, we collected data across two domains: (1) the South China Sea territorial disputes, and (2) general content categories reflecting YouTube’s broader ecosystem. This dual focus enables comparison between politically sensitive and neutral domains, allowing us to assess whether the platform’s algorithm exhibits differential behavior in response to geopolitical versus everyday content. 3.1. Keyword Selection The South China Sea (SCS) dataset was curated through expert consultations with researchers from the Atlantic Council’s Digital Forensic Research Lab, the National University of Singapore, and De La Salle University. Keywords focus on regional disputes, military activity, and environmental issues Atlantic Council 2025 ; Centre for International Law, National University of Singapore 2025 . For the General Content dataset, we followed YouTube’s content taxonomy EntreResource 2023 , covering a wide spectrum from entertainment and sports to science and education. Table 1 summarizes the keyword sets. Table 1. Keywords used for data collection. Representative keywords are shown for the South China Sea; full categories are listed for General Content. Dataset Keywords South China Sea South China Sea dispute, China nine dash line, China artificial islands, US Navy South China Sea, China maritime claims, South China Sea latest news, China vs Philippines, South China Sea oil and gas, ASEAN South China Sea talks, China territorial dispute. General Content Film & Animation, Autos & Vehicles, Music, Pets & Animals, Sports, Travel & Events, Gaming, People & Blogs, Comedy, Entertainment, News & Politics, How-to & Style, Education, Science & Technology, Nonprofits & Activism. 3.2. YouTube Shorts Collection Since the YouTube Data API v3 does not support Shorts, we used APIFY’s YouTube Scraper Streamers 2024 to collect Shorts video IDs based on the keywords in Table 1 .