Doing Audits Right? The Role of Sampling and Legal Content Analysis in Systemic Risk Assessments and Independent Audits in the Digital Services Act
Marie-Therese Sekwenz, Rita Gsenger, Scott Dahlgren, Ben Wagner · 2025
Read Doing Audits Right? The Role of Sampling and Legal Content Analysis in Systemic Risk Assessments and Independent Audits in the Digital Services Act by Marie-Therese Sekwenz, Rita Gsenger, Scott Dahlgren, Ben Wagner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03601.txt.
Introduction
Marie-Therese Sekwenz and co-authors examine how platform audits should be done under the European Digital Services Act in a 2025 paper. The law forces big platforms to assess systemic risks, from illegal content to threats to rights and democratic processes, with outside audits on top. A delegated rule sets expectations for both sides, yet method choices stay open.
The authors weigh sampling and legal content analysis against other qualitative and quantitative tools. They argue a mixed method, risk specific approach works best, and they review how sample choice is treated in law and across fields. Early platform risk reports are also read for their testing and sampling habits.
Catalog record
| CREATOR | Marie-Therese Sekwenz, Rita Gsenger, Scott Dahlgren, Ben Wagner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Doing Audits Right? The Role of Sampling and Legal Content Analysis in Systemic Risk Assessments and Independent Audits in the Digital Services Act Marie-Therese Sekwenz, Rita Gsenger, Scott Dahlgren, Ben Wagner Abstract . A central part of the European Union’s Digital Services Act (DSA) is the requirement for internal and external audits of online platforms. A key component of these audits is the analysis of platform systemic risk assessments, which cover wide-ranging and multifaceted topics such as the dissemination of illegal content, negative effects on human rights, the electoral process, and gender-based violence. The Delegated Regulation on how to conduct such audits under the DSA further specifies what is expected from platforms and auditors to ensure compliance with the law. This article discusses the pros and cons of different quantitative and qualitative methods that could be used to audit the varying systemic risks of online platforms and search engines under Europe’s DSA regime. Following an evaluation of key methods, we propose an empirical mixed-method approach based on risk-specific sampling techniques combined with (legal) content analysis as a reliable auditing method for the DSA. We argue that audits should be based on the evaluation of content samples representing spot checks for systemic risks. To develop this mixed-method approach, we first analyse the European provisions relevant to auditable content samples and their selection. Then, we explore sampling techniques that can be meaningfully applied in internal and external DSA audits and assess methodological techniques for testing systemic risk categories. Furthermore, we examine how the representativeness of a sample can be understood in different disciplines. Finally, we analyse the first systemic risk assessment reports of platforms with a view to understanding how the tests and methodologies (such as sampling techniques) used in these assessments evaluate systemic risks. By proposing a novel empirical mixed-method approach for conducting DSA audits, this article addresses the complex challenges of evidence-based risk assessment in content moderation. Furthermore, we contribute to the rapidly growing field of DSA compliance assessments, emphasizing the specificities of audits and systemic risks. In doing so, we recognize that there is no “one-size-fits-all” solution for DSA audits, but rather a use case-specific and platform-tailored approach is needed. CCS: Social and professional topics Computing / technology policy 1. Introduction The Digital Services Act (DSA) ( European Parliament and Council of the European Union, 2022b ) represents a cornerstone of Europe’s efforts to regulate online expression and intermediary services and aims to harmonize rules for user-generated and synthetic content on online platforms. The DSA aims to increase transparency and protect users from illegal content online. It aspires to achieve the so-called “Brussels effect”, potentially influencing global regulatory standards ( Nunziato, 2023 ) . It entered into force on February 17, 2024. The DSA applies to intermediary services operating within the European Union (EU) according to Art. 2(1). These services offer network infrastructure, such as hosting platforms (for example, cloud services) or online platforms (such as app stores or online marketplaces). Importantly, these services also include Very Large Online Platforms (VLOPs) and Very Large Search Engines (VLOSEs) ( European Parliament and Council of the European Union, 2022b ) , which face the most stringent requirements. They are defined as platforms with over 45 million average monthly users (or 10 % of the EU population) in the European Union (Art. 33 DSA). The European Commission published (and continuously updates) a list of designated VLOPs and VLOSes, including platforms such as Amazon, LinkedIn, Wikimedia, and Facebook ( Commission, 2024 ) . Therefore, the Regulation adopts a layered approach to obligations based on platform size and function, considering an increased risk for society due to content that is accessible to a large number of users. The DSA includes various new obligations for users, businesses, and platforms. Key provisions include the adaptation of Terms and Conditions (TaC) (Art. 14), transparency obligations (Arts. 15, 24, 42), the design of notice and take-down mechanisms (Art. 16), platforms’ content moderation decision reporting (Art. 17), Trusted Flagger involvement (Art. 22), recommendation system disclosures (Art. 27), researcher access for independent evaluations (Art. 40), ex ante risk assessments (Arts. 34, 35), or third-party audits (Art. 37). These external audits are done by auditing firms like KPMG, EY, or Deloitte ( Hohlfeld, 2024 ) . Recent enforcement actions underscore the DSA’s impact. For example, on December 18, 2023, the European Commission initiated formal proceedings against X (formerly Twitter) for noncompliance, citing risks related to illegal content dissemination (Art. 34(1)(a)) and threats to civic discourse and electoral processes (Art. 34(1)(c)) ( European Commission, 2023a ) . The Commission also opened proceedings against TikTok on election risks ( European Commission, 2024c ) and against Temu for the sale of illegal products, their addictive platform design, failed disclosure of recommendation system, and failed compliance of researcher data access ( European Commission, 2024b ) . 2. Background Various research endeavours have already looked into audits in the online space in general and the auditing requirements of the DSA specifically. Research has focused on auditing algorithms ( Imana et al., 2023 ; Terzis et al., 2024 ) and their challenges. More recently, AI audits have emerged as a discussion point, as increasingly, approaches are proposed to audit AI systems ( Lam et al., 2024 ; Falco et al., 2021 ) to increase fairness and accountability ( Calvi and Kotzinos, 2023 ) . Furthermore, some research elaborates on increasing auditability, for instance, in extended reality (XR) applications ( Norval et al., 2023 ; Gray et al., 2024 ) . These studies have highlighted issues with accessing systems, highlighting that black-box access is not enough ( Casper et al., 2024 ) . AI can be used to perform audits ( Li and Goel, 2024 ) ; however, these audits can be biased ( Groves et al., 2024 ) . European Regulations are not the first to establish auditing as a mechanism of transparency. Moreover, the DSA, the Digital Markets Acts ( European Parliament and Council of the European Union, 2022a ) , and the AI Act ( European Parliament and Council of the European Union, 2024 ) include auditing provisions. Effective auditing under the DSA requires a robust framework that ensures auditor independence and competence. Kaushal et al. propose guidelines for audit quality assurance, including performance standards and quality control procedures, which are essential for credible assessments ( Kaushal et al., 2024 ) . They provide an empirical analysis of the DSA’s transparency database, highlighting the legal obligations and the role of automated systems in ensuring compliance. Terzis et al. highlight that although the auditing mechanisms are not new, they have a larger scope than other legally required audits. However, they only apply to a smaller number of audited entities (namely, to VLOPs and VLOSEs) ( Terzis et al., 2024 ) . Moreover, to align sampling with specific risks, such as the dissemination of illegal content (Art. 34(1)(a) DSA), auditors must ensure that samples represent the population accurately, particularly in ex post content moderation scenarios. Studies such as Mubarak et al.’s work on disinformation emphasize the value of comparing deleted and undeleted content samples ( Mubarak et al., 2023 ) . Various challenges in auditing remain, and as such, “auditing the auditors”, finding best practices, and standards