SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment

Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta · 2025

Read SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment by Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18468.txt.

Introduction

Ariful Haque and co-authors survey the risks of AI coding assistants in a 2025 paper. Tools such as Copilot and ChatGPT now write and fix code inside many teams, bringing real speed gains. The same tools can also copy insecure patterns, invent wrong code, and leak sensitive data.

The survey weighs benefits against harms across security, code quality, and ethics. It covers made-up code, called hallucinations, plus risks to private data and ownership rights. Features and performance of leading tools are compared side by side. The authors aim to help developers choose and guard these tools well, so that speed gains do not come at the cost of safety. Their guidance targets both daily practice and wider deployment policy.

Catalog record

CREATORAriful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta Abstract The integration of Large Language Models (LLMs) such as GitHub Copilot, ChatGPT, Cursor AI, and Codeium AI into software development has revolutionized the coding landscape, offering significant productivity gains, automation, and enhanced debugging capabilities. These tools have proven invaluable for generating code snippets, refactoring existing code, and providing real-time support to developers. However, their widespread adoption also presents notable challenges, particularly in terms of security vulnerabilities, code quality, and ethical concerns. This paper provides a comprehensive analysis of the benefits and risks associated with AI-powered coding tools, drawing on user feedback, security analyses, and practical use cases. We explore the potential for these tools to replicate insecure coding practices, introduce biases, and generate incorrect or non-sensical code (hallucinations). In addition, we discuss the risks of data leaks, intellectual property violations and the need for robust security measures to mitigate these threats. By comparing the features and performance of these tools, we aim to guide developers in making informed decisions about their use, ensuring that the benefits of AI-assisted coding are maximized while minimizing associated risks. A Preprint 1 Introduction Large language models have transformed coding by enabling automatic code generation from natural language descriptions. Tools such as ChatGPT, Codium, Copilot, and Cursor AI assist developers in writing, completing, refactoring, and optimizing code to enhance productivity and improve performance [ 56 ] . However, LLMs also pose security risks, as they are trained on publicly available code, which may contain insecure practices [ 32 ] . This can lead to vulnerabilities in the generated code that malicious actors could exploit [ 33 ] . Furthermore, supply chain attacks targeting third-party services, particularly popular libraries such as npm packages, have increased dramatically in recent years [ 54 ] . The use of advanced AI models in software development may unintentionally support these attacks by generating phishing messages and attack plans [ 11 ] . Despite their advantages, LLMs can also perpetuate biases and introduce bugs, which can affect performance and raise ethical concerns [ 44 ] , [ 19 ] . LLMs offer advantages in various fields, including healthcare, education, programming, improving communication, workflows, and knowledge discovery [ 51 ] , [ 55 ] . Although LLMs cannot replace human programmers, they streamline tasks such as code writing, documentation, and bug detection, making them increasingly valuable in programming [ 41 ] . However, the widespread use of LLMs raises concerns about possible security vulnerabilities and hallucinations [ 25 ] . This paper explores these trade-offs with the guidance of developers in leveraging AI’s benefits while mitigating associated risks. 1.1 Related work Heibel et al. [ 23 ] discuss Malicious Programming Prompt (MaPP) attacks, where attackers manipulate prompts to make large language models (LLMs) generate vulnerable code. Despite improvements in model capabilities, these attacks remain effective across various platforms. The authors tested MaPP on seven LLMs using HumanEval and Common Weakness Enumerations (CWEs) and found that simple prompts induced vulnerabilities without affecting code correctness. In addition, the best-performing models were more prone to malicious instructions. Their findings suggest that enhancing models alone is not enough to prevent prompt manipulation attacks. Similarly another author Derner et al. [ 18 ] analyzes the security risks of large language models (LLMs) like ChatGPT, focusing on vulnerabilities in content filters and the potential for malicious use. This paper identifies risks such as harmful text generation, data leaks, and unethical content creation. It also highlights the importance of informing policymakers and industry professionals about these issues. The authors evaluate the effectiveness of ChatGPT’s content filters and explore the ethical concerns surrounding its use. They conclude that current safeguards are insufficient, allowing for harmful outputs. The study recommends further research to strengthen safeguards and investigate the broader societal effects of LLMs. In another paper, Oviedo et al. [ 39 ] discuss the advanced AI language model ChatGPT, which is widely used for tasks like customer service and chatbots. However, it has drawbacks, including the potential to provide incorrect or unsafe information that may impact user safety. This paper also evaluates ChatGPT’s overall effectiveness in promoting safety in contexts such as phone usage while driving and stress management in the workplace. Alternatively, Jacobi et al. [ 24 ] emphasize that the increasing complexity and frequency of cyber threats necessitate updated approaches to strengthen Governance, Risk, and Compliance (GRC) frameworks. One promising approach involves the use of artificial intelligence, particularly LLMs like ChatGPT and Google Gemini, to enhance cybersecurity guidance. Research indicates that ChatGPT generally provides more relevant, accurate, and context-appropriate advice compared to Google Gemini. While both models have limitations, this study highlights the potential benefits of integrating LLMs into GRC frameworks, especially when combined with human expertise to address complex issues. In another case, Atzori et al. [ 9 ] examine on LLMs such as ChatGPT, GPT-4, Claude, and Bard can be misused to create phishing attacks. These models can generate realistic phishing emails and websites without modification, allowing attackers to easily scale their efforts using malicious prompts. To counter this threat, researchers have developed a BERT-based detection tool that effectively identifies phishing prompts across various LLMs. BERT-based detection tool that effectively identifies phishing prompts across various LLMs. Another author, Latif et al. [ 45 ] discuss previous research in code generation using Large Language Models (LLMs), which usually focus on functional correctness but overlook security. It introduces the SALLM framework, which adds security-specific prompts and metrics to evaluate secure code generation. While earlier studies mainly checked syntactic and functional accuracy. This paper improves on that by using a rule-based repair system to enhance syntactic correctness, resulting in better compilation rates, especially for GPT-4. However, the study acknowledges challenges like potential bias in manually created prompts and the lack of real-world task representation, suggesting that future research should improve datasets and security evaluation methods. Mishra et al. [ 31 ] introduced FAVA-Bench dataset for fine-grained hallucination detection in language models (LMs), featuring a comprehensive taxonomy of hallucination types. The dataset includes span-level human annotations and responses from models like Llama2-Chat and ChatGPT. It comes from a number of different sources, such as the No Robots dataset, the WebNLG dataset, and Open Assistant queries. They developed FAVA, a retrieval-augmented LM fine-tuned for detecting and mitigating hallucinations, which significantly outperformed existing systems like GPT-4. The study highlighted that over 60% of errors in model outputs were unverifiable, emphasizing the prevalence of hallucinations. While the work provides a robust framework, its focus on specific models limits generalizability, prompting future research to adapt FAVA for broader fact-checking tasks and evaluate it across diverse LM architectures. 1.2 Our Contribution Key Contributions of our SoK paper are: User-Centric Insights: Incorporates feedback from IT professionals to evaluate the impact of AI tools on productivity, error reduction, and collaboration while addressing issues like hallucinations and contextual errors. (in Sections 2-3). Evaluation of AI Coding Tools: Provides a comprehensive analysis of tools like GitHub Copilot, ChatGPT, Cursor AI, and Codeium AI, detailing their capabilities, user benefits, and limitations in improving software development

Read the complete text