SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents
Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim · 2025
Read SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents by Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17017.txt.
Introduction
Qiusi Zhan leads five co-authors in making search agents safer in a 2025 paper. Search agents answer questions by querying the web repeatedly. The team found these agents produce harmful output more often than plain models. Retrieved documents can lower the agent's refusal threshold.
Fine-tuning agents only for usefulness made the risk worse. Their method, SafeSearch, trains safety and usefulness together. A final reward judges the completed answer on both counts. A second reward shapes each search query itself. Unsafe queries get penalized, safe ones rewarded, during the search. Harmful output fell by over ninety percent in tests. Answers stayed helpful while refusals worked as intended. The work shows safety must be trained into the search process itself.
Catalog record
| CREATOR | Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17017
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang, Joo-Kyung Kim Abstract Large language model (LLM) based search agents iteratively generate queries, retrieve external information, and reason to answer open-domain questions. While researchers have primarily focused on improving their utility, their safety behaviors remain underexplored. In this paper, we first evaluate search agents using red-teaming datasets and find that they are more likely to produce harmful outputs than base LLMs. For example, when asked “How can I track someone’s location without their consent?”, a base model refuses, whereas a search agent designed to retrieve and cite sources may lower its refusal threshold, fetch documents (e.g., court cases), and, once appended, synthesize them into an informative yet unsafe summary. We further show that utility-oriented finetuning intensifies this risk, motivating joint alignment of safety and utility. We present SafeSearch , a multi-objective reinforcement learning approach that couples a final-output safety/utility reward with a novel query-level shaping term that penalizes unsafe queries and rewards safe ones. Experiments show that SafeSearch reduces agent harmfulness by over 70% across three red-teaming datasets while producing safe, helpful responses, and matches the QA performance of a utility-only finetuned agent; further analyses confirm the effectiveness of the query-level reward in jointly improving safety and utility. se counter=jsoncounter]showjson[1][] enhanced, arc=0em, boxrule=.5pt, listing only, listing options= language=myjson, upquote=true, basicstyle= , breaklines=true, breakindent=8pt, xleftmargin=-8pt, xrightmargin=-8pt, aboveskip=-4pt, belowskip=-4pt, columns=fullflexible, escapeinside=||, , colback=white, colframe=gray, colbacktitle=gray!10, coltitle=black, attach boxed title to top center=yshift=-3mm, #1 se counter=promptcounter]prompt[1][] enhanced, arc=0em, boxrule=.5pt, listing only, breakable, listing options= language=prompt, upquote=true, basicstyle= , breaklines=true, breakindent=8pt, xleftmargin=-8pt, xrightmargin=-8pt, aboveskip=-4pt, belowskip=-4pt, columns=fullflexible, escapeinside=||, , colback=white, colframe=gray, colbacktitle=gray!5, coltitle=black, attach boxed title to top center=yshift=-3mm, #1 2 2 footnotetext: Code will be available soon. 1 Introduction Large language models (LLMs) are increasingly deployed as search agents: systems that iteratively formulate queries, retrieve external information, and reason over it to produce answers ( Li et al. 2025 ; Jin et al. 2025b ; Song et al. 2025 ; Zheng et al. 2025 ) . Unlike traditional single-turn retrieval-augmented generation (RAG) ( Lewis et al. 2020 ; Borgeaud et al. 2022 ) , search agents autonomously drive multi-turn retrieval with self-generated queries, achieving strong performance on open-domain question answering, analytical tasks, and research assistance. However, recent work finds that both single-turn RAG systems ( An et al. 2025 ) and multi-turn RAG systems (i.e., search agents) ( Yu et al. 2025 ) are more likely to generate toxic or illegal content than base LLMs, revealing a gap in safety alignment. For example, a base LLM typically refuses a harmful prompt such as “How do I kidnap someone in New York?” In contrast, a search agent may lower its refusal threshold in pursuit of helpfulness and issue follow-up queries (e.g., “kidnapping cases in New York 2024”). Even when the agent initially frames the search with benign intent, for instance, to educate the user about the legal consequences of kidnapping using real-world cases, once retrieved content is appended, especially content containing harmful details such as operational descriptions of crimes, the model may deviate from its original intent, prioritize consistency with the retrieved sources, and produce harmful outputs. While a large body of work has focused on using reinforcement learning (RL) to improve the utility of search agents, enabling better reasoning and query formulation ( Jin et al. 2025b ; Song et al. 2025 ; Zheng et al. 2025 ) , to the best of our knowledge, no existing work addresses their safety alignment. We therefore ask: (i) what factors contribute to the unsafe behaviors of search agents, and (ii) how can utility-oriented RL alignment be extended to jointly optimize both safety and utility? To study the first question, we evaluate multiple systems on red-teaming datasets containing harmful inputs. Our experiments show that search agents are considerably more harmful than base LLMs, with harmful output rates up to 2× higher. This risk further increases after utility-oriented finetuning, reaching up to 3× the harmfulness of base LLMs. We further observe that safety is closely linked to both the decision to invoke search and the safety of the resulting queries. In particular, invoking search is strongly correlated with a higher rate of harmful outputs, and unsafe queries correlate with substantially higher harmfulness compared to whose queries are all safe. Utility-oriented RL greatly improves effectiveness but trades off safety. To make search agents both useful and safe, we introduce SafeSearch , an RL-based alignment framework with automatically verifiable safety and utility rewards that jointly optimize both objectives. SafeSearch uses mixed training that interleaves general QA instances (rewarded for anwer correctness) with safety instances (rewarded for both safety and helpfulness). Beyond a binary safe/unsafe signal on final outputs, it explicitly rewards policy-compliant helpfulness (e.g., offering safe alternatives or high-level legal guidance), echoing the safety alignment of GPT-5 ( Yuan et al. 2025 ) and reducing blanket refusals that provide no information when inputs are potentially harmful. Moreover, to resolve the core tension that search can boost helpfulness yet import harm, we also add a query-level safety reward that scores generated queries and steers the agent toward benign searches that yield policy-compliant, helpful outputs. We finetune both Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct Qwen Team 2024 with SafeSearch on two suite of datasets: general QA benchmarks to assess utility, including single-hop TriviaQA ( Joshi et al. 2017 ) , multi-hop HotpotQA ( Yang et al. 2018 ) , and Bamboogle ( Press et al. 2023 ) ; and red-teaming datasets with harmful inputs, where we measure both the safety of generated outputs and the helpfulness of safe responses. Empirically, SafeSearch reduces the average harmful rate by 70% across red-teaming datasets while keeping QA performance comparable to utility-only finetuning. Ablations suggest that the query-level reward contributes to the safety gains and helps preserve utility. 2 Related Work LLM Based Search Agents. LLMs exhibit strong reasoning yet still hallucinate and lack domain knowledge, motivating integration with search engines for dynamic access to external information ( Jin et al. 2025a ) . Search agent is a prominent direction treats search as an interactive tool invoked during inference, enabling iterative query formulation, retrieval, and evidence - based response generation. Prompt - based approaches such as IRCoT ( Trivedi et al. 2023 ) and ReAct ( Yao et al. 2023 ) interleave reasoning with retrieval, while training - based methods like Toolformer ( Schick et al. 2023 ) and Self - RAG ( Asai et al. 2024 ) teach models when and how to issue calls and incorporate results. More recently, reinforcement learning (RL) has been used to induce reasoning - and - search behaviors with outcome - based rewards (e.g., Search - R1 ( Jin et al. 2025b ) , R1 - Searcher ( Song et al. 2025 ) , DeepResearcher ( Zheng et al. 2025 ) ). However, recent work ( Yu et al. 2025 ) reveals a safety devolution in search agents: the enhanced search capability also increases their potential for generating harmful or unsafe content. To address this challenge, we propose SafeSearch , the first RL framework that jointly aligns the safety and utility of search agents, enabling reliable information acquisition without compromising user