First, do no harm: Breaking suicidogenic echo chambers in media recommendation

Alberto Díaz-Álvarez, Raúl Lara-Cabrera, Fernando Ortega-Requena, Víctor Ramos-Osuna · 2026

Read First, do no harm: Breaking suicidogenic echo chambers in media recommendation by Alberto Díaz-Álvarez, Raúl Lara-Cabrera, Fernando Ortega-Requena, Víctor Ramos-Osuna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25258.txt.

Introduction

Alberto Díaz-Álvarez and colleagues add a safety layer to media recommenders that otherwise trap vulnerable users in loops of harmful clips. Their RankAid method demotes risky items and promotes calming material, with the strength of the shift tied to the user's current distress level. In simulations built from a million-rating movie dataset annotated for clinical risk, the system blocks harmful suggestions at crisis peaks. It also steers feeds toward de-escalation. The safety gain costs only a modest dip in standard accuracy scores.

Catalog record

CREATORAlberto Díaz-Álvarez, Raúl Lara-Cabrera, Fernando Ortega-Requena, Víctor Ramos-Osuna
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24403
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

First, do no harm: Breaking suicidogenic echo chambers in media recommendation Alberto Díaz-Álvarez, Raúl Lara-Cabrera, Fernando Ortega-Requena, Víctor Ramos-Osuna Abstract Recommender systems generally optimises user engagement, but this approach is dangerous in mental health contexts. When vulnerable users show signs of suicidal ideation, standard algorithms often trap them in echo chambers of harmful content, worsening their psychological state. In response, we introduce RankAid , a re-ranking method that prioritises clinical safety alongside predictive relevance. It works as an add-on layer to existing models: it penalises risky items and boosts therapeutic content depending on the user’s current level of vulnerability. We evaluated this approach using the MovieLens 1M dataset, where items were semantically annotated for clinical risk and therapeutic value using large language models . Our simulations show that our algorithm successfully blocks the recommendation of harmful content during crisis peaks, actively reshaping the feed to support emotional de-escalation. Furthermore, this safety intervention only causes a controlled, acceptable drop in standard accuracy metrics like NDCG . By using asymmetric hyperparameters, RankAid also gives system administrators the flexibility to tune the severity of the intervention based on specific clinical guidelines. 1 Introduction Recommender systems (RSs) are an essential component in digital platforms, typically designed to maximise user engagement and retention. Their effectiveness is traditionally evaluated by ranking quality metrics. In this work, we adopt the Normalized Discounted Cumulative Gain (NDCG) [ 12 ] as our primary accuracy metric. This metric is widely used as it rewards systems that place the most relevant items at the top while penalising those further down the list. However, because these models prioritise click probability, they assume all high-engagement content is suitable. In mental health and suicide prevention, this optimisation is dangerous. Previous research shows that vulnerable users experiencing suicidal ideation often fall into “digital rumination” [ 16 ] . In such cases, traditional RS risk trapping individuals in echo chambers of nihilistic or self-harming content, potentially worsening their mental state [ 1 ] . While some studies use heuristics or language models to mitigate harmful content [ 9 , 15 ] , there is currently no mathematical function that combines clinical vulnerability with psychological risk to explicitly penalise exposure. We propose a “preventive ethical adjustment” through RankAid . This algorithm implements a re-ranking scoring function that modifies predictions in crisis contexts. By weighing the user’s latent vulnerability against the item’s psychological risk and therapeutic value, RankAid mathematically penalises harmful recommendations for high-risk profiles (regardless of engagement potential) and promotes therapeutic content to help de-escalate the crisis (or at least, the rumination). 2 Related work Recent advancements in large language models (LLMs) have expanded the intersection of artificial intelligence (AI) and mental health; yet, recommendation algorithms still fall short in crisis contexts. 2.1 Suicide risk detection Current literature primarily addresses suicide prevention through natural language processing (NLP) techniques. Many studies use deep learning (DL) models, such as transformer architectures, to identify suicidal ideation and depressive rumination. Specifically, analysing non-clinical texts (such as social media posts or search histories) is essential to assess psychological states in digital environments, even without access to medical records [ 6 , 4 ] . This line of research has evolved into triage systems that estimate intervention urgency based on these signals. Notable in this area is the adaptation of metrics like hierarchical Time-Biased Gain (hTBG) , designed to quickly identify individuals at suicide risk when time is limited [ 19 ] . However, this approach focuses exclusively on alerting clinical evaluators; it lacks mechanisms to assess the psychological risk of individual content items or to prevent users from being exposed to harmful material. On the other hand, conversational agents (i.e., chatbots designed for crisis intervention) have shown positive results in providing early emotional support [ 7 ] . The drawback is that both these chatbots and detection systems operate independently of the user’s regular content consumption on the platform. Consequently, they fail to address a fundamental problem: the RS itself might suggest harmful content that worsens the crisis. 2.2 Echo chambers The formation of echo chambers [ 14 , 2 ] has recently been mathematically modeled. For example, [ 5 ] shows that when user preferences evolve, static penalties on problematic content can fail or even increase long-term harm. To mitigate this, the “trustworthy recommender systems ” paradigm [ 10 ] requires systems to be fair, explainable, and safe. Within this safety dimension, researchers have proposed filters for negative experiences [ 9 ] or the use of LLMs to re-rank feeds and hide self-harm material [ 15 ] . Our work also utilises the semantic capabilities of LLMs . However, unlike models that delegate re-ranking directly to a LLM (which is computationally expensive and/or opaque), we use the LLM beforehand to build a clinical knowledge matrix. While prior studies use static penalties, our approach treats LLM assessments as variables in a mathematical function. This determines rewards or penalties based on the user’s vulnerability, ensuring protection aligned with digital wellbeing principles. 2.3 Ethical algorithm modification A common strategy to avoid echo chambers is to force the algorithm to (also) optimise content diversity. Some systems attempt to break these bubbles by combining diversity and accuracy metrics [ 22 ] . However, in the mental health domain, purely mathematical diversity metrics are insufficient. Because they optimise list heterogeneity without explicitly modelling clinical risk, a system could generate a topically diverse list that still includes explicit, implicit, or ambivalent self-harming content [ 21 , 18 ] . This risk is exacerbated by position bias: high-risk items appearing at the top of the list receive disproportionate user attention [ 13 ] . Even if these diverse items are harmless, introducing variety without ensuring relevance or clinical suitability is often perceived as mere noise [ 23 ] . Given the risks of unfiltered optimisation, recommendation objectives must be adapted for clinical contexts. Although modifying formulas for clinical reasons is uncommon, the literature on fairness in RSs considers the recommendation space a scarce resource that must be distributed fairly and under certain constraints [ 20 ] . Following this principle, several authors have adapted classic metrics by introducing penalties to ensure the visibility of minority groups [ 8 , 3 ] . Essentially, the algorithm’s objective is redefined: sacrificing some predictive accuracy is accepted in exchange for meeting (in principle) ethical criteria. The tech industry has also begun to acknowledge these harms (caused by their own algorithms) and has implemented mechanisms to limit exposure to negative user experiences ( Negative User Experiences (NUXs) ) [ 9 ] . The problem is that social networks typically apply these limits as reactive a posteriori filters (heuristic down-ranking ), rather than structurally integrating them into the mathematical formulas that train and guide the model from the ground up. 3 Methodology Instead of designing a new predictive model from scratch, our approach uses a two-stage pipeline. First, a standard model generates relevance predictions. Second, our algorithm intervenes to recalculate each item’s value based on clinical parameters before display. An advantage of this strategy is that it operates independently of the underlying recommendation model, simply adding an extra layer to re-rank the proposed items. 3.1 System architecture The workflow implemented in our experiment consists

Read the complete text