Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants

Pavan Reddy, Nithin Reddy · 2025

Read Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants by Pavan Reddy, Nithin Reddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07022.txt.

Introduction

Pavan Reddy and Nithin Reddy propose modular safety middleware for AI assistants used near health topics, in a 2025 paper. Their starting case is Tessa, a chatbot run by an eating disorders association. In 2023 Tessa was taken down after it gave harmful weight-loss advice to users who were already vulnerable.

The authors read that failure as missing safety engineering, and argue it could have been prevented. Their design pairs fixed word-level gates with a policy check by a large language model inside the same call. The system fails closed and passes hard cases upward for review. On tests built by the authors, the design caught every unsafe prompt at baseline cost and delay, beating standard multi-stage setups. They also link Tessa's failure patterns to known governance frameworks, tying day-to-day safeguards to controls that can be audited.

Catalog record

CREATORPavan Reddy, Nithin Reddy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants Pavan Reddy, Nithin Reddy Abstract In 2023, the National Eating Disorders Association’s (NEDA) chatbot Tessa was suspended after providing harmful weight-loss advice to vulnerable users—an avoidable failure that underscores the risks of unsafe AI in healthcare contexts. This paper examines Tessa as a case study in absent safety engineering and demonstrates how a lightweight, modular safeguard could have prevented the incident. We propose a hybrid safety middleware that combines deterministic lexical gates with an in-line large language model (LLM) policy filter, enforcing fail-closed verdicts and escalation pathways within a single model call. Using synthetic evaluations, we show that this design achieves perfect interception of unsafe prompts at baseline cost and latency, outperforming traditional multi-stage pipelines. Beyond technical remedies, we map Tessa’s failure patterns to established frameworks (OWASP LLM Top10; NIST SP 800-53), connecting practical safeguards to actionable governance controls. The results highlight that robust, auditable safety in health-adjacent AI does not require heavyweight infrastructure: explicit, testable checks at the last mile are sufficient to prevent “another Tessa,” while governance and escalation ensure sustainability in real-world deployment. Introduction AI assistants are increasingly deployed in health and counseling contexts, where even minor errors can result in significant harm. In 2023, the National Eating Disorders Association (NEDA) transitioned its scripted prevention chatbot, “Tessa,” from a supplementary tool to the primary point of contact for individuals seeking help ( Hoover 2023 ) . Although the initial deployment proceeded without issue, public reports later indicated that Tessa was promoting calorie deficits, weigh-ins, and other weight-loss behaviors incompatible with evidence-based eating disorder care. As a result, NEDA suspended the chatbot ( Aratani 2023 ; Hoover 2023 ) . This incident highlights a key principle: in high-risk domains, content moderation must not be treated as an afterthought, and safety mechanisms should be explicitly engineered as testable, first-class components of the system, rather than relying solely on prompt design. This paper analyzes the Tessa failure as a concrete case of unsafe assistance in a clinical-adjacent context and proposes a minimal, deployable remedy: a lightweight safety middleware that deterministically enforces health-specific redlines while keeping latency and cost compatible with real-world operations. The design combines (i) a fast lexical gate that blocks obviously risky intents (e.g., calorie targets, weigh-ins, dieting frames) and (ii) an in-line large language model (LLM) policy filter that judges and, when allowed, generates responses in a single call. The module emits a strict JSON verdict ( is_safe , violation categories), operates fail-closed, and performs a final numeric/lexical scan of the buffered answer before rendering. Crisis signals trigger escalation and safe-mode responses by construction. Our contributions are threefold. First, we map the observed breakdowns in Tessa to generalizable safety failures including, but not limited to policy like implementation drift, missing input triage, insecure output handling, etc. and connect them to OWASP LLM Top10 and NIST800-53 controls to make the lessons actionable. Second, we present a modular safety layer that integrates a denylist/regex fast path with an LLM policy filter that produces both an answer (when permissible) and a self-audit verdict, avoiding the latency and cost of two-model pipelines. Third, we prototype this design and demonstrate, on representative interactions, that it intercepts disallowed content while preserving responsiveness, illustrating how modest engineering can lead to safer versions of “Tessa” Date Event Description February 2022 Tessa debuts on NEDA’s website as a scripted, prevention-focused chatbot operating alongside the human helpline. ( Nguyen 2023a ) March 17, 2023 Helpline staff win their union election; NLRB certification follows later in the month. ( Nguyen 2023b ) March 27, 2023 NLRB certifies the Helpline Associates Union. ( Aratani 2023 ) March 31, 2023 NEDA tells staff the helpline will be wound down and services will transition to Tessa around June 1. ( Wells 2023 ) May 26, 2023 Helpline Associates Union publicly warns that replacing trained humans with a chatbot endangers help-seekers. ( Nguyen 2023b ) May 29, 2023 Activist Sharon Maxwell posts evidence that Tessa recommends caloric deficits, weekly weigh-ins, and other weight-loss guidance. ( Nguyen 2023a ) May 30, 2023 NEDA suspends Tessa pending investigation, citing harmful and non-programmed responses. ( Wells 2023 ) June 1, 2023 National coverage details the helpline’s closure and Tessa’s suspension, with criticism of reliance on a chatbot. ( Nguyen 2023b ) Table 1: Timeline of key public events surrounding NEDA’s “Tessa” chatbot. Overview and Timeline of Events The National Eating Disorders Association (NEDA) introduced the chatbot “Tessa” as a scripted, rule-based, prevention-oriented resource intended to expand access to support for individuals experiencing body image concerns and eating disorders ( Hoover 2023 ; Wells 2023 ) . This deployment occurred in parallel with the continued operation of a long-running, human-staffed helpline ( Wells 2023 ; Hoover 2023 ) . NEDA initially described Tessa as an evidence-based tool that delivered structured content derived from cognitive-behavioral and related therapeutic frameworks ( Hoover 2023 ; Wells 2023 ) . Early use did not generate public controversy. In late March 2023, internal labor developments brought the chatbot to broader attention ( Nguyen 2023b ; Wells 2023 ) . Following the certification of the helpline staff’s union on March 27, NEDA leadership notified employees that the live helpline would be discontinued, with services transitioning to Tessa ( Wells 2023 ; Nguyen 2023b ) . Layoffs were announced at the end of March ( Nguyen 2023b ; Wells 2023 ) . Although NEDA denied that the decision was motivated by union activity—citing strategic and operational considerations—the timing prompted criticism ( Nguyen 2023b ; Aratani 2023 ) . Observers argued that replacing human-led support with automation in a high-risk mental health setting could compromise the quality of care ( Nguyen 2023b ; Hoover 2023 ) . Public concern intensified in late May 2023, when clinicians and advocacy groups documented instances of Tessa providing responses that contradicted established eating disorder treatment principles ( Aratani 2023 ; Hoover 2023 ) . Reports indicated that the chatbot offered weight-loss advice and recommended behaviors such as calorie counting and routine weigh-ins—interventions widely considered clinically inappropriate and potentially harmful for this population ( Aratani 2023 ; Hoover 2023 ) . These findings reinforced earlier ethical concerns raised by former helpline staff and community members, who emphasized that automated systems lack the empathy, contextual awareness, and dynamic risk assessment required to support individuals in psychological distress ( Nguyen 2023b ; Wells 2023 ) . In response to the growing evidence of harm, NEDA suspended Tessa on May 30, 2023, pending further investigation ( Aratani 2023 ; Wells 2023 ) . The organization acknowledged that the chatbot’s outputs had diverged from the intended, policy-consistent messaging ( Aratani 2023 ) . NEDA stated that Tessa would remain offline until adequate safeguards could be implemented ( Wells 2023 ) . However, the simultaneous closure of the helpline created a service gap, with users redirected to external crisis support resources. This case has since been cited as a cautionary example in the application of AI to mental health care, highlighting the importance of accountability, oversight, and the continued relevance of human judgment in high-risk therapeutic contexts ( Harvard T.H. Chan School of Public Health 2023 ) . Background and Related Work Modern AI

Read the complete text