Understanding and Mitigating Risks of Generative AI in Financial Services
Sebastian Gehrmann, Claire Huang, Xian Teng, Sergei Yurovski, Iyanuoluwa Shode, Chirag S. Patel, Arjun Bhorkar, Naveen Thomas, John Doucette, David Rosenberg, Mark Dredze, David Rabinowitz · 2025
Read Understanding and Mitigating Risks of Generative AI in Financial Services by Sebastian Gehrmann, Claire Huang, Xian Teng, Sergei Yurovski, Iyanuoluwa Shode, Chirag S. Patel, Arjun Bhorkar, Naveen Thomas, John Doucette, David Rosenberg, Mark Dredze, David Rabinowitz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20086.txt.
Introduction
Gehrmann and a large team map safety risks of generative AI in finance and test current guards, in a paper from twenty twenty-five. Banks face strict laws on what advice and content they can give. Most AI safety work looks at broad harms like bias or toxic speech. That misses risks tied to finance rules and duties.
The team builds a risk list shaped for finance products and laws. They weigh what a breach would mean for clients, firms, and markets. Then they test open-source guard tools on data from red-team probes. The guards missed most risks on the finance list. The gap shows that general guards are not enough. Firms need checks built for their own rules and products.
Catalog record
| CREATOR | Sebastian Gehrmann, Claire Huang, Xian Teng, Sergei Yurovski, Iyanuoluwa Shode, Chirag S. Patel, Arjun Bhorkar, Naveen Thomas, John Doucette, David Rosenberg, Mark Dredze, David Rabinowitz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20086
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding and Mitigating Risks of Generative AI in Financial Services Sebastian Gehrmann, Claire Huang, Xian Teng, Sergei Yurovski, Iyanuoluwa Shode, Chirag S. Patel, Arjun Bhorkar, Naveen Thomas, John Doucette, David Rosenberg, Mark Dredze, David Rabinowitz Abstract To responsibly develop Generative AI (GenAI) products, it is critical to define the scope of acceptable inputs and outputs. What constitutes a “safe” response is an actively debated question. Academic work puts an outsized focus on evaluating models by themselves for general purpose aspects such as toxicity, bias, and fairness, especially in conversational applications being used by a broad audience. In contrast, less focus is put on considering sociotechnical systems in specialized domains. Yet, those specialized systems can be subject to extensive and well-understood legal and regulatory scrutiny. These product-specific considerations need to be set in industry-specific laws, regulations, and corporate governance requirements. In this paper, we aim to highlight AI content safety considerations specific to the financial services domain and outline an associated AI content risk taxonomy. We compare this taxonomy to existing work in this space and discuss implications of risk category violations on various stakeholders. We evaluate how existing open-source technical guardrail solutions cover this taxonomy by assessing them on data collected via red-teaming activities. Our results demonstrate that these guardrails fail to detect most of the content risks we discuss. 1 Introduction Text-based generative AI (GenAI) technologies have a near-limitless capacity to consume user input and produce sensible-sounding responses in a diverse range of use cases. Advances in large language models (LLMs) have catalyzed the development and deployment of conversational systems for many domains. This expansive design space comes with the challenge that the system could create unsafe output in response to user inputs. “AI Safety” research has proposed definitions, standards, and best practices for determining when the input to or output from GenAI systems is unsafe, undesirable, or poses potential harm to the user, system provider, or others. Operationalizing undesirable inputs and outputs can be grounded in ethics, laws, rules and regulations, cultural norms, and the nature of the application. Mitigation approaches for such risky content can be implemented at multiple levels, for example by changing the underlying model [ 7 ] or through a separate filter layer [ 40 , 96 , e.g. ,] . Governance frameworks and related policies and procedures may further define how to handle flagged content safety violations [ 56 ] . AI content safety begins with taxonomies to define unsafe behaviors. Most prior work considers the safety of isolated models, which are only one component of a complex sociotechnical system [ 57 ] . Moreover, while definitions of “safe” and “unsafe” vary by application and applicable rules, Rauh et al. 2024 point out that academic work to date focuses on a narrow set of general risk categories. However, AI risk management must consider a holistic approach to AI system safety to ensure the responsible development of AI systems [ 55 ] . The Organization for Economic Co-operation and Development (OECD) defines risks as a function of both the probability of an event occurring and the severity of the consequences that would result from that event [ 54 ] . By ignoring the broader sociotechnical system in which AI is deployed and thus falling into the Framing Trap [ 67 ] , work may disproportionately focus on less likely or less relevant sources of harm and miss critical domain-specific risks. Avoiding this trap is crucial, as deployment of models in complex systems, especially in knowledge-intensive domains, is one of the most prominent uses of GenAI models. Our primary hypothesis is that general purpose safety taxonomies and guardrail systems are insufficient to meet the needs of real-world GenAI systems. When considering new technologies for a complex system and domain, we must evaluate the specific potential harms to assess risk within the domain. Failure to do so creates a potential “Safety Gap.” We test this hypothesis by evaluating how general-purpose AI guardrail systems perform when applied to the financial services domain. We develop a new domain-specific taxonomy for this domain, and conduct an empirical study of existing LLM-based guardrail systems with this taxonomy. Our taxonomy reflects the broader environment in which financial services systems operate. The NIST AI Risk Management Framework emphasizes that AI systems often operate in complex settings and are influenced by societal dynamics and human behavior. Risks can emerge from “the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.” [ 55 ] The Responsible Innovation Framework [ 38 ] which is broadly adopted in the UK [ 58 ] , emphasizes risk assessment and management as a requirement for responsible integration of technology into sociotechnical systems. While individual actors or system components may be individually responsible [ 73 ] , complex coupled systems may create “organized irresponsibility” [ 88 ] . Identifying and mitigating this organized responsibility risk requires a holistic study of the complete system and not just individual components. Following the recommendations of these frameworks, technologists must work with subject matter experts to understand, anticipate and prioritize risks, identify and characterize precipitating hazards and harms, and develop related governance structures. Risk quantification requires identifying potential harms ( e.g. , customer harm, regulatory enforcement, civil litigation) that result from hazards ( e.g. , breach of confidentiality, reliance on misinformation) posed by the technology [ 54 ] . To explore and validate our hypothesis, we present a focused empirical case study applying a domain-specific GenAI risk taxonomy to the investment management and capital markets financial services domain, hereafter referred to as financial services domain. Financial services are a major focus for the development of GenAI systems [ 93 , 47 , 94 ] for which there has been limited AI Safety discussions [ 52 ] . The financial service sector is highly regulated across broad subject matter areas to maintain safety, stability, and faith in the system; ensure access to the system; protect investors/depositors; foster fairness, order, and efficiency; and facilitate investment and growth [ 9 ] . For this reason, laws, rules, and regulations are often designed to be technology agnostic in anticipation of the next innovation [ 61 ] . Specific hazards that may be present within a GenAI application must be contextualized in this domain to understand the potential harms and risks they pose to individual users and the broader system. Our study demonstrates that a safety gap can emerge from a failure to take a holistic view of GenAI systems. We offer three contributions in support of our hypothesis that only a holistic approach can prevent an AI safety gap. First, we conduct a conceptual analysis of the literature by reviewing AI safety taxonomies and risk mitigation strategies, and how these can be adapted to knowledge-rich domains ( Section 3 ). Our analysis illuminates opportunities in the current literature for how to develop domain-specific taxonomies. Second, we present a case study for the development of a holistic GenAI risk taxonomy for the financial services domain ( Section 4 ). 1 1 1 Other financial services stakeholders like banks or insurance companies may have overlapping