Hallucination in LLM-Based Code Generation: An Automotive Case Study

Marc Pavel, Nenad Petrovic, Lukasz Mazur, Vahid Zolfaghari, Fengjunjie Pan, Alois Knoll · 2025

Read Hallucination in LLM-Based Code Generation: An Automotive Case Study by Marc Pavel, Nenad Petrovic, Lukasz Mazur, Vahid Zolfaghari, Fengjunjie Pan, Alois Knoll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11257.txt.

Introduction

Marc Pavel and co-authors audit hallucination in language models writing automotive software, where plausible nonsense can turn dangerous. Models now draft vehicle code, but their fluent errors include broken syntax, phantom references, and clashes with known interfaces. The team tests several state-of-the-art code models under three prompt loads.

Their 2025 case study moves from a bare one-line prompt, to one carrying vehicle signal specifications, to one adding a code skeleton. Error rates stay high until the richest prompt, where only the two strongest models produce a correct solution. Simpler prompts never yielded working code, even after rounds of refinement. The authors press for mitigation before such models touch safety-critical code.

Catalog record

CREATORMarc Pavel, Nenad Petrovic, Lukasz Mazur, Vahid Zolfaghari, Fengjunjie Pan, Alois Knoll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hallucination in LLM-Based Code Generation: An Automotive Case Study Marc Pavel, Nenad Petrovic, Lukasz Mazur, Vahid Zolfaghari, Fengjunjie Pan, Alois Knoll Abstract Large Language Models (LLMs) have shown significant potential in automating code generation tasks offering new opportunities across software engineering domains. However, their practical application remains limited due to hallucinations - outputs that appear plausible but are factually incorrect, unverifiable or nonsensical. This paper investigates hallucination phenomena in the context of code generation with a specific focus on the automotive domain. A case study is presented that evaluates multiple code LLMs for three different prompting complexities ranging from a minimal one-liner prompt to a prompt with Covesa Vehicle Signal Specifications (VSS) as additional context and finally to a prompt with an additional code skeleton. The evaluation reveals a high frequency of syntax violations, invalid reference errors and API knowledge conflicts in state-of-the-art models GPT-4.1, Codex and GPT-4o. Among the evaluated models, only GPT-4.1 and GPT-4o were able to produce a correct solution when given the most context-rich prompt. Simpler prompting strategies failed to yield a working result, even after multiple refinement iterations. These findings highlight the need for effective mitigation techniques to ensure the safe and reliable use of LLM generated code, especially in safety-critical domains such as automotive software systems. I Introduction Over the last few years, Large language models (LLMs) have gained tremendous popularity in diverse domains such as software engineering, natural language processing and code generation. In particular, models such as GPT-3.5 [ 1 ] , GPT-4 [ 2 ] , CodeX [ 3 ] , CodeBERT [ 4 ] , CodeGPT [ 5 ] and CodeLLaMA [ 6 ] have become increasingly valuable due to their ability to generate code snippets, refactor programs and provide programming assistance. However, despite their impressive capabilities, LLMs have been observed to produce hallucinations - instances where the model generates information that contradicts its source, cannot be verified from it, or is presented as factual despite being nonsensical or incorrect [ 7 ] [ 8 ] . The general problem with hallucinations in LLMs lie in their tendency to generate output that lacks consistency and authenticity making the generated content unrealiable and unsuitable for practical applications [ 9 ] . This issue becomes even more critical in the context of code generation. Unlike natural language processing tasks where minor inaccuracies may be overlooked or easily corrected, errors in generated code can lead to bugs, security vulnerabilities, or system failures, potentially resulting in untrustworthiness, compromised data integrity, unauthorized access and increased development and maintenance costs [ 10 ] . Hallucinations arise due to the statistical nature of LLMs, since they are trained to predict the most likely next token based on patterns in large datasets. However, they lack a true understanding of the meaning or the factual basis of the content they generate. As a result, when faced with ambiguity, sparsity of information, or requests beyond their knowledge, LLMs may fill in the gaps with plausible-sounding but incorrect or unverifiable content. Understanding how and why these failures occur is crucial for designing effective detection and mitigation strategies, especially in domains such as code generation. [ 7 ] The paper is organized as follows. The next section explores the nature and causes of LLM hallucinations in code generation. Moreover, the third section presents an automotive case study together with results, while appendix includes the list of vehicle signals used within the experiments. The last one summarizes the main findings of this work and potential future research directions. II Hallucination in LLM-Based Code Generation While hallucinations in LLMs have been extensively studied, their occurence in code generation is an emerging challenge that has only recently begun to attract focus. Due to the complexity and diversity of programming tasks, existing classifications remain inconsistent with no universally accepted taxonomy for hallucinations in code generation. Moreover, current mitigation strategies are often task-specific and struggle to address the wide variety of hallucinations that can arise highlighting the need for more comprehensive solutions. II-A Taxonomy To better understand and address hallucinations in code generation, recent studies have introduced systematic taxonomies. One such classification, proposed by Lee et al. [ 10 ] , highlights how these hallucinations can appear in various forms including bugs, syntactic errors, security vulnerabilities or even non-deterministic behavior: • Syntactic hallucinations: These refer to hallucinations that cause compile-time failures and can be further categorized into syntax violations , such as missing brackets or incorrect indentation, and incomplete code generation , where the model produces unfinished code e.g. due to output length limitations. • Runtime execution hallucinations: These refer to code that compiles correctly, but fails during execution. Runtime execution hallucinations inlcude API knowledge conflicts , which occur when the LLM misuses libraries or APIs e.g. by omitting necessary imports or using incorrect function signatures, and invalid reference errors where the generated code attempts to access or manipulate undefined elements, such as referencing undeclared variables or functions. • Functional correctness hallucinations: These refer to code that compiles and executes without errors, but fails to meet the specified requirements. They can be further divided into incorrect logical flow , which includes missing corner cases, flawed conditional logic or incorrect arithmetic operations, and requirement deviation , where the generated code diverges from the explicit requirements and functionalities described in the prompt or problem description. • Code quality hallucinations: These refer to code that introduces issues related to resource management, security or maintainability. This includes resource mishandling leading to excessive consumption or inefficient allocation of memory, security vulnerabilities , where generated code introduces weaknesses that make the system susceptible to attacks or unauthorized access and code smells referring to poorly structured or overly complex code that reducing maintainability. TABLE I: Overview of the state-of-the-art code generation LLMs, their hallucination mitigation strategies, used benchmarks and ongoing challenges. Model Hallucination Mitigation Techniques Benchmarks Challenges GPT-4.1 (2025) [ 13 ] Instruction tuning, long context up to 1 million tokens SWE-Bench, Graphwalks, MultiChallenge, etc. Limited retrieval grounding Codex (2021) [ 3 ] Fine-tuned on GPT-3 with code from GitHub HumanEval Difficulty with docstrings describing long chains of operations and with binding operations to variables Claude 3 Opus (2024) [ 11 ] Constitutional AI, RLHF, red-teaming HumanEval, MBPP, APPS No web search, factual errors and bias Code LLaMA 70B (2024) [ 6 ] Infilling, long input contexts, instruction fine-tuning HumanEval, MBPP, MultiPL-E Limited retrieval grounding DeepSeek-Coder V2 (2024) [ 14 ] long input contexts, RL, instruction fine-tuning HumanEval, MBPP+, SWE-Bench, etc. Gap in instruction-following capabilities StarCoder2 15B (2024) [ 15 ] No native mitigation HumanEval, MBPP, EvalPlus, etc. Representation bias, societal bias Phi-3 Mini (2024) [ 16 ] Post-processing including supervised fine-tuning and direct preference optimization HumanEval, MBPP, DS-1000, etc. Limited capacity to store factual knowledge due to model size WizardCoder 34B (2023) [ 17 ] Instruction tuning via Evol-Instruct HumanEval, MBPP Tendency to lose semantic and logical consistency during code generation [ 18 ] II-B Causes To understand these hallucinations more thoroughly, recent research has examined their root causes, commonly classifying them into three main categories [ 10 ] : • Training data issues: These refer to limitations in the dataset including the lack of quantity of training data and the low quality

Read the complete text