Revisiting Algorithmic Audits of TikTok: Poor Reproducibility and Short-term Validity of Findings Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGI
Matej Mosnar, Adam Skurla, Branislav Pecher, Matus Tibensky, Jan Jakubcik, Adrian Bindas, Peter Sakalik, Ivan Srba · 2025
Read Revisiting Algorithmic Audits of TikTok: Poor Reproducibility and Short-term Validity of Findings Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGI by Matej Mosnar, Adam Skurla, Branislav Pecher, Matus Tibensky, Jan Jakubcik, Adrian Bindas, Peter Sakalik, Ivan Srba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18140.txt.
Introduction
Matej Mosnar and co-authors re-run past audits of TikTok's feed in 2025 and find the results hard to repeat. Audits use fake accounts to test if feeds trap users or push them toward worse content. Watchdogs and lawmakers lean on such audits, so their findings need to last.
The team hit snags from shifts in the app and its content, plus flaws in the past studies themselves. Heavy fixes were needed to rerun the work at all. Worse, the one-shot findings often held only for a short time, they report. The lesson is not that audits are useless, but that they must be built to be rerun. A feed that changes weekly cannot be judged by a test that runs once.
Catalog record
| CREATOR | Matej Mosnar, Adam Skurla, Branislav Pecher, Matus Tibensky, Jan Jakubcik, Adrian Bindas, Peter Sakalik, Ivan Srba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Algorithmic Audits of TikTok: Poor Reproducibility and Short-term Validity of Findings Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730293 ISBN: 979-8-4007-1592-1/2025/07 Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy CCS: Information systems Personalization CCS: Information systems Content ranking CCS: Human-centered computing Human computer interaction (HCI) Matej Mosnar, Adam Skurla, Branislav Pecher, Matus Tibensky, Jan Jakubcik, Adrian Bindas, Peter Sakalik, Ivan Srba Abstract . Social media platforms are constantly shifting towards algorithmically curated content based on implicit or explicit user feedback. Regulators, as well as researchers, are calling for systematic social media algorithmic audits as this shift leads to enclosing users in filter bubbles and leading them to more problematic content. An important aspect of such audits is the reproducibility and generalisability of their findings, as it allows to draw verifiable conclusions and audit potential changes in algorithms over time. In this work, we study the reproducibility of the existing sockpuppeting audits of TikTok recommender systems, and the generalizability of their findings. In our efforts to reproduce the previous works, we find multiple challenges stemming from social media platform changes and content evolution, but also the research works themselves. These drawbacks limit the audit reproducibility and require an extensive effort altogether with inevitable adjustments to the auditing methodology. Our experiments also reveal that these one-shot audit findings often hold only in the short term, implying that the reproducibility and generalizability of the audits heavily depend on the methodological choices and the state of algorithms and content on the platform. This highlights the importance of reproducible audits that allow us to determine how the situation changes in time. 1. Introduction Social media platforms are pervasive in our lives, with their popularity and the amount of user-generated content on them rapidly increasing. In recent years, these platforms have shifted from simply providing chronologically ordered content based on users’ connection towards algorithmically curated content that does not require extensive and explicit user feedback ( Klug et al. 2021 ) . At the same time, the content on the platform has shifted to a short-video format (e.g., videos that are 30-60 seconds long). Together, these changes lead to a shift in how users consume and engage with social media content, e.g., increasing the time spent on the platform ( Zannettou et al. 2024 ) . However, such a shift leads to significant problems as well. First, the algorithmically curated recommendations are viewed as non-transparent to end users. In addition, they tend to enclose users in filter bubbles, showing only the content that achieves strong engagement, which leads to a proliferation of highly problematic content such as misinformation or extreme and polarising viewpoints ( Ribeiro et al. 2020 ; Tomlein et al. 2021 ) . To address these problems, researchers started performing social media algorithmic auditing studies ( Urman et al. 2024 ; Bandy 2021 ; Sandvig et al. 2014 ) to better understand the algorithms and ways to mitigate the potential negative consequences. Significant focus was dedicated to the understanding of the "traditional" social media and their content, such as Facebook, YouTube as well as different search engines ( Boeker and Urman 2022 ; Vombatkere et al. 2024 ; Mousavi et al. 2024 ; Spinelli and Crovella 2020 ; Hussein et al. 2020 ; Papadamou et al. 2022 ; Tomlein et al. 2021 ) . These studies have confirmed the negative impact of social media, where the algorithm unintentionally leads users to more and more problematic content, as it generates high levels of engagement. However, the auditing studies for the short-video format platforms (e.g., TikTok) are only starting to slowly appear. At the same time, to mitigate the negative consequences, new policies are enacted – most prominently the Digital Services Act (DSA) ( Commission 2023 ) . This legislation calls for systematic audits of social media platforms (according to Article 37) to assess the obligations that social media platforms need to follow – some of them are specifically dedicated to algorithms that curate content on these platforms. Algorithmic audits represent one of the means by which algorithmic-related obligations of social media platforms can be assessed. An important aspect of such algorithmic audits is their reproducibility and generalisability of their results, even as the content on the platform may evolve and change. Without reproducibility, the findings of the algorithmic audits are not actionable as they do not allow the policymakers and regulators to demand and consequently verify changes on the platforms. The main goal of this study is to determine the reproducibility of the social media algorithmic audits and the generalisability of their results when performed after some period of time and in extended settings (i.e., in additional countries). For this purpose, we reproduce and extend the previous works that audit the popular, short-video format social media platform TikTok. We chose TikTok due to its popularity (mainly among younger audiences) and the strength and complexity of its recommender (i.e., it can quickly determine user interests and steer users towards niche content). Specifically, we build on the works of Boeker and Urman 2022 , Vombatkere et al. 2024 and partially also Mousavi et al. 2024 (in terms of evaluation metrics). The primary aim of these reference studies was to investigate how personalisation factors , such as user location or various user interactions with content, influence the level of personalisation or explanations of why the videos were recommended. All of them audit the recommender system behind the For You section of the TikTok platform. Since the previous studies were performed before the enactment of the DSA, the policy may already have an impact on the platforms. At the same time, the recommender systems, as well as the content itself, evolve constantly. As all of these factors may affect the nature of recommended items, it is important to continuously reproduce the findings (or use a longitudinal algorithmic audit study). By revisiting the experiments from the reference studies, we address the following research questions: • (RQ1) What is the level of reproducibility of algorithmic audits taking into consideration the constantly evolving nature of social media platforms? • (RQ2) What is the effect of different personalisation factors , such as location, watch duration, liking and following, on the For You video recommendation on TikTok? How has the importance of different personalisation factors evolved since the reference studies ? To answer these research questions, while trying to reproduce the reference studies as faithfully as possible, we also did several extensions and some inevitable deviations, specifically: 1) extending the setting to different countries, as the behaviour of the recommender system may differ across them ( Urman et al. 2024 ) ; 2) adjusted and fixed specific methodological decisions (e.g., watch time being lower for the videos of interest, or dealing with shadow-bans) that may have unintentionally affected the findings ( Chandio et al. 2024 ) ; 3) updated the algorithmic audit implementation based on the platform changes (e.g., dealing with ads and