The Malicious Technical Ecosystem: Exposing Limitations in Technical Governance of AI-Generated Non-Consensual Intimate Images of Adults Conference: CHI’25 Sociotechnical AI Governance Workshop; April 2025; Yokohama, Japan

Michelle L. Ding, Harini Suresh · 2025

Read The Malicious Technical Ecosystem: Exposing Limitations in Technical Governance of AI-Generated Non-Consensual Intimate Images of Adults Conference: CHI’25 Sociotechnical AI Governance Workshop; April 2025; Yokohama, Japan by Michelle L. Ding, Harini Suresh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17663.txt.

Introduction

Michelle Ding and Harini Suresh wrote a 2025 paper on the ecosystem that makes fake intimate images of adults without consent. They take a survivor centered view of how tech governance fails to stop this abuse. Face swap models and near two hundred nudifying tools let lay users make such fakes in minutes.

The authors map that malicious tech ecosystem piece by piece. They then hold current governance practice, as summed in a national standards report, up against the map. The match shows clear holes for adult cases: open tools spread freely, and rules built for other harms miss this one. False beliefs inside the standard frame help explain the gaps. The paper was made for a workshop on AI governance.

Catalog record

CREATORMichelle L. Ding, Harini Suresh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Malicious Technical Ecosystem: Exposing Limitations in Technical Governance of AI-Generated Non-Consensual Intimate Images of Adults Conference: CHI’25 Sociotechnical AI Governance Workshop; April 2025; Yokohama, Japan Michelle L. Ding, Harini Suresh Abstract . In this paper, we adopt a survivor-centered approach to locate and dissect the role of sociotechnical AI governance in preventing AI-Generated Non-Consensual Intimate Images (AIG-NCII) of adults, colloquially known as “deep fake pornography.” We identify a “malicious technical ecosystem” or “MTE,” comprising of open-source face-swapping models and nearly 200 “nudifying” software programs that allow non-technical users to create AIG-NCII within minutes. Then, using the National Institute of Standards and Technology (NIST) AI 100-4 report as a reflection of current synthetic content governance methods, we show how the current landscape of practices fails to effectively regulate the MTE for adult AIG-NCII, as well as flawed assumptions explaining these gaps. ACM Reference Format: Michelle L. Ding and Harini Suresh. 2025. The malicious technical ecosystem: Exposing limitations in technical governance of AI-generated non-consensual intimate images of adults. In CHI’25 Sociotechnical AI Governance Workshop (STAIG ’25) , April 2025, Yokohama, Japan. 1. Introduction AI-Generated Non-Consensual Intimate Images (AIG-NCII), colloquially known as “deep fake pornography,” are a form of image-based sexual abuse that disproportionately harm and silence women, girls, LGBTQ+ people, and racial minorities ( Flynn et al. 2022 ; Center for Democracy and Technology 2025 ; Santiago Lakatos 2023 ; McGlynn et al. 2017 ) . Image-based sexual abuse covers a spectrum of behaviors including the non-consensual creation of intimate images, non-consensual distribution of intimate images, and threatening to distribute those images ( McGlynn et al. 2017 ; Qin et al. 2024 ) . Recognizing the extensive prior work on preventing AI-generated child sexual abuse materials (AIG-CSAM) and non-consensual distribution of intimate images, this paper aims to contribute to the unique challenge of preventing the creation of AIG-NCII of adults ( Thiel 2024 ; Qin et al. 2024 ; Akerley 2021 ; Becca Branum 2024 ; Thorn, All Tech Is Human 2024 ) . These include the documented 35,545 images depicting 26 members of congress and thousands more depicting celebrities, public figures, and ordinary women and teens ( Thiel 2024 ; Thorn & WeProtect Global Alliance 2024 ; Thorn, All Tech Is Human 2024 ; American Sunlight Project 2024 ; Henry Ajder et al. 2019 ; My Image My Choice 2023 ) . While recent advancements in corporate image-generation models like Stability AI’s Stable Diffusion have resulted in alarmingly photorealistic AIG-NCII, there has existed since 2017 a prolific ecosystem of open-source face-swapping models such as DeepFaceLab, DeepNude, and FaceSwap that support nearly 200 “nudifying” software programs allowing non-technical users to create what they call “deep fake pornography” within minutes ( Thiel 2024 ; Perov et al. 2020 ; My Image My Choice 2023 ; torzdf 2017 ; iperov 2018 ; yuanxiao 2020 ; Mehta et al. 2023 ; Timmerman et al. 2023 ) . This decentralized technical ecosystem of open-source models and tools (which we characterize as the “malicious technical ecosystem” or “MTE”) enables the creation of AIG-NCII of adults that are often noticeably “fake” yet still a form of abuse that results in mental and physical harm, reputational damage, financial costs, and disruption of social relationships ( Santiago Lakatos 2023 ; Sophie Cockerham 2022 ; McGlynn et al. 2021 ; Umbach et al. 2025 ; Akerley 2021 ; Flynn et al. 2022 ) . Image-based sexual abuse, like other forms of technology-facilitated gender-based violence, also causes a gendered chilling effect where victim-survivors retreat from online spaces ( Sambasivan et al. 2019 ; McGlynn et al. 2017 ; Kristine Baekgaard 2024 ) . These effects already appear in the case of AIG-NCII as seen from interviews and testimonials ( Flynn et al. 2022 ; My Image My Choice 2024 ) . Despite extensive documentation of the harms and scale of these models and tools, they remain largely ungoverned by current technical prevention methods. In this paper, we locate the MTE as a unique challenge for the sociotechnical governance of adult AIG-NCII. First, we identify and contextualize the MTE within the broader synthetic content pipeline of creation, distribution, and consumption. Next, by drawing on the National Institute of Standards and Technology (NIST) AI 100-4 report on governing synthetic content, we show how current governance mechanisms fail to effectively regulate the MTE: 1) broad synthetic content governance measures focus on transparency, which is insufficient to address the harms of watermarked, human-detectable adult AIG-NCII, 2) AIG-NCII specific governance measures conflate CSAM and adult NCII, proposing methods more effective in regulating CSAM than adult NCII, and 3) adult-specific AIG-NCII governance measures acknowledge the existence of the MTE but propose methods that will primarily regulate large corporate models. Overall, we stress the need for governance frameworks that cite AIG-NCII of adults as a high-risk harm to assess their effectiveness in regulating the MTE by considering the three listed limitations. Figure 1. Supply chain for AI Generated Non-Consensual Intimate Images that expands on the NIST AI 100-4 synthetic content pipeline of creation, distribution, and consumption. Yellow circles represent the malicious technical ecosystem (MTE). Training dataset, user input, and AIG-NCII are highlighted to represent artifacts within the pipeline and not any specific example. Supply chain for AI Generated Non-Consensual Intimate Images (AIG-NCII) that expands on the NIST AI 100-4 synthetic content pipeline (creation, distribution, and consumption) showing the key technical components involved in each stage from dataset, model, user input, user tool, to output in creation and then various websites, platforms, and forums for distribution and consumption. Yellow highlighted circles represent the core of the malicious technical ecosystem (MTE). Note: while ``deep fake websites'' in distribution/consumption are closely associated with the MTE as the primarily destination of AIG-NCII, we do not include them within the MTE because their governance requires downstream response, not technical prevention. 2. Response vs. Prevention: Locating the Role of Sociotechnical AI Governance in Mitigating AIG-NCII Before diving into the challenges of technical prevention, it is important to contextualize the role of sociotechnical AI governance within the larger multi-stakeholder space of AIG-NCII mitigation efforts. The issue of AIG-NCII, like other forms of image-based sexual abuse and technology-facilitated gender-based violence, requires both prevention and response efforts. There are already many powerful responses to AIG-NCII, including: two federal legislation efforts that both passed the Senate (the DEFIANCE Act and the Take It Down Act) which require expedited platform take-down procedures and create civil liability for malicious actors; advocacy petitions for the take-down of "deepfake pornography" apps and sites; and voluntary platform trust and safety commitments ( Sen. Cruz, Ted [R-TX] 2023 ; Sen. Durbin, Richard J. [D-IL] 2023 ; My Image My Choice [n. d.] ; Partnership on AI 2023 ; Center for Democracy and Technology 2025 ) . While these response measures are critical to addressing the downstream distribution and consumption of AIG-NCII, they place the burden of removal on survivors, many of whom may be unwilling to report abuse, hesitant to engage with the criminal justice system, and/or lack the emotional, financial, and physical capacity to engage in lengthy take-down procedures or legal action ( Flynn et al. 2022 ; Lorenz et al. 2019 ) . Public attitudes toward “deepfake pornography” often involve victim-blaming and

Read the complete text