Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search

Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, Xinlei He · 2025

Read Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search by Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, Xinlei He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04951.txt.

Introduction

Zeren Luo and co-workers gauged the safety of AI-powered web search engines in a 2025 study. These engines blend model knowledge with live web lookup to answer precisely. The authors watched them quoting malicious content and citing hostile websites, spreading harmful or unverified information with a confident tone. Defining a threat model and risk taxonomy, they systematically probed seven making systems with varied query types. Drawing malicious indicators from PhishTank, ThreatBook, and LevelBlue threat feeds.

Even benign keyword queries often drew responses containing malicious links. Asking about a URL straight raised the risk further, while natural language phrasing reduced it slightly. Encouragingly, the AI engines still beat traditional search on both utility and safety overall. Case studies on document spoofing and phishing show how easily the systems can be deceived.

Catalog record

CREATORZeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, Xinlei He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04951

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, Xinlei He Abstract Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering precise and efficient responses by integrating external databases with pre-existing knowledge. However, we observe that these AIPSEs raise risks such as quoting malicious content or citing malicious websites, leading to harmful or unverified information dissemination. In this study, we conduct the first safety risk quantification on seven production AIPSEs by systematically defining the threat model, risk type, and evaluating responses to various query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our findings reveal that AIPSEs frequently generate harmful content that contains malicious URLs even with benign queries (e.g., with benign keywords). We also observe that directly querying a URL will increase the number of main risk-inclusive responses, while querying with natural language will slightly mitigate such risk. Compared to traditional search engines, AIPSEs outperform in both utility and safety. We further perform two case studies on online document spoofing and phishing to show the ease of deceiving AIPSEs in the real-world setting. To mitigate these risks, we develop an agent-based defense with a GPT-4.1-based content refinement tool and a URL detector. Our evaluation shows that our defense can effectively reduce the risk, with only a minor cost of reducing available information by approximately 10.7%. Our research highlights the urgent need for robust safety measures in AIPSEs. 1 Introduction Recently, Large Language Models (LLMs) have demonstrated great potential in various applications [ 69 , 54 , 65 , 30 ] . Notably, conversational chatbots like ChatGPT [ 1 ] have achieved remarkable success. However, these models face inherent limitations. Since GPT models learn from pre-existing data (e.g., GPT-4’s knowledge is up-to-date only until October 2023 [ 12 ] ), they are unable to accurately address queries about information beyond this cutoff. Furthermore, LLMs are prone to generating hallucinations [ 37 ] . To address these issues, AI-Powered Search Engines (AIPSEs) have emerged as an application of Retrieval-Augmented Generation (RAG) [ 41 , 31 ] . Concretely, AIPSEs combine a knowledge database, a retriever, and an LLM to provide up-to-date information by integrating external data with the LLM’s existing knowledge. Unlike Traditional Search Engines (TSEs) that focus on keyword matching and semantic search [ 61 , 48 ] , AIPSEs leverage LLMs to interpret user intent, retrieving and summarizing relevant external data to generate precise and concise answers, enhancing efficiency over TSEs. Furthermore, through our analysis of 2,875 search result URLs from both AIPSEs and TSEs, we observe that AIPSEs consistently outperform TSEs in terms of utility and safety at the current stage ( Section 6 ). Despite being powerful, AIPSEs may face several risks. For example, the retriever may access unfiltered malicious websites, and the LLM might refer to their data without any safety check, which results in harmful or unverified responses. For instance, when users search for software to install, AIPSEs may directly provide the malware website instead of the official one in some cases. Such risks have already caused monetary loss in the real world: In November 2024, a developer lost about $2,500 after following code generated by ChatGPT Search. 1 1 1 https://twitter-thread.com/t/1859656430888026524 . The code directed him to a fake Solana API website, where he was tricked into sending his private key. This led to his assets being quickly stolen within 30 minutes. This incident highlights the urgent need for thorough safety checks in production AIPSEs. Figure 1 demonstrates the process of our paper. In this paper, we perform the first safety risk quantification against 7 production AIPSEs. Concretely, we first systematically define the threat model, construct different types of queries (including keyword list query, URL query, and natural language query), and categorize the URLs in responses of AIPSEs into different risk types (main, warning, source, and none) based on their harmfulness ( Section 4.1 ). To perform the evaluation, we collect candidate URLs as well as their keyword lists (generated by GPT-4o) from three popular cyber threat detection platforms: PhishTank [ 14 ] , ThreatBook [ 19 ] , and LevelBlue [ 8 ] . Our evaluation shows that all AIPSEs suffer from generating harmful content based on malicious URLs. For example, when querying seven AIPSEs, we observe that 47% of responses are risky ( Figure 4 ). Moreover, 34% of responses directly cite the harmful content in the answers ( Figure 4 ). We also find that, compared to keyword list queries, URL queries tend to increase the number of main risk-inclusive responses ( Figure 6 ), whereas natural language queries can slightly mitigate these risks ( Figure 4 ). We further illustrate the feasibility of deceiving AIPSEs through two case studies focused on online document spoofing and phishing websites. These studies demonstrate how adversaries can effortlessly mislead AIPSEs into quoting malicious code or recognizing phishing websites as legitimate official sites ( Section 7 ). To mitigate these risks, we propose two defense strategies ( Section 8 ): prompt-based and agent-based. The prompt-based approach utilizes a proxy LLM, where responses generated by AIPSE are fed into gpt-4.1-2025-04-14 (GPT-4.1) along with a defense prompt to filter out harmful content. Agent-based defense contains a content refinement tool and a URL detector tool, where different detectors can be integrated into this tool. The agent will iteratively invoke the refinement tool and the detector tool until sufficient information is collected and will generate an attached response that contains as much information as possible while notifying the safety vulnerabilities in the original response. Evaluations show that the agent-based defense can outperform prompt-based defense and reduce the number of main risk-inclusive responses to a significantly larger extent. The evaluation shows that our proposed HtmlLLM-Detector can address 78.3% of main risk-inclusive responses and achieves a high F1 Score of 0.822 ( Table 2 ). By conducting this work, we aim to provide valuable insight for future research to improve the safety of AIPSEs. Figure 1: Overall Process of Our Work: We collect 100 websites and their corresponding keyword lists as the evaluation dataset (see Section 4.2 for more details). Then, we evaluate seven representative AIPSEs on this dataset to reveal the safety risk of them ( Section 4 and Section 5 ). We also conduct two case studies about malicious online documents and phishing websites to demonstrate the feasibility of deceiving production AIPSEs ( Section 7 ). Lastly, we propose a simple yet effective agent-based defense strategy at the user end to help filter unsafe responses ( Section 8 ). In conclusion, we make the following contributions: 1. We present the first quantitative assessment of safety risks associated with the inclusion of malicious content or URLs in AIPSE responses. Our evaluation reveals that all currently operational AIPSEs are vulnerable to such risks ( Section 5 ). 2. We conduct a comparative analysis of utility and safety performance between AIPSEs and TSEs. The results show that AIPSEs outperform TSEs in both dimensions ( Section 6 ). Additionally, two real-world case studies demonstrate how easily AIPSEs can be manipulated into endorsing harmful content in real-world

Read the complete text