Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?

Guoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, Dan Hao · 2024

Read Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering? by Guoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, Dan Hao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02093.txt.

Description

Abstract . Large Language Models (LLMs) have significantly advanced software engineering (SE) tasks, with prompt engineering techniques enhancing their performance in code-related areas. However, the rapid development of foundational LLMs such as the non-reasoning model GPT-4o and the reasoning model o1 raises questions about the continued effectiveness of these prompt engineering techniques.

Catalog record

CREATORGuoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, Dan Hao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02093

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering? Guoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, Dan Hao Abstract . Large Language Models (LLMs) have significantly advanced software engineering (SE) tasks, with prompt engineering techniques enhancing their performance in code-related areas. However, the rapid development of foundational LLMs such as the non-reasoning model GPT-4o and the reasoning model o1 raises questions about the continued effectiveness of these prompt engineering techniques. This paper presents an extensive empirical study that reevaluates various prompt engineering techniques within the context of these advanced LLMs. Focusing on three representative SE tasks, i.e., code generation, code translation, and code summarization, we assess whether prompt engineering techniques still yield improvements with advanced models, the actual effectiveness of reasoning models compared to non-reasoning models, and whether the benefits of using these advanced models justify their increased costs. Our findings reveal that prompt engineering techniques developed for earlier LLMs may provide diminished benefits or even hinder performance when applied to advanced models. In reasoning LLMs, the ability of sophisticated built-in reasoning reduces the impact of complex prompts, sometimes making simple zero-shot prompting more effective. Furthermore, while reasoning models outperform non-reasoning models in tasks requiring complex reasoning, they offer minimal advantages in tasks that do not need reasoning and may incur unnecessary costs. Based on our study, we provide practical guidance for practitioners on selecting appropriate prompt engineering techniques and foundational LLMs, considering factors such as task requirements, operational costs, and environmental impact. Our work contributes to a deeper understanding of effectively harnessing advanced LLMs in SE tasks, informing future research and application development. CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Software defect analysis 1. Introduction Large Language Models (LLMs) ( OpenAI, 2021 ; DeepSeek-AI, 2024 ; Anthropic, 2023 ; DeepMind, 2023 ; AI, 2023 ) have achieved remarkable results across various domains ( Wu et al., 2024 ; Yan, 2024 ; Qin et al., 2023 ; Satpute et al., 2024 ; Chen et al., 2024 ) , demonstrating human-like intelligence, especially in natural language processing (NLP) ( Sun et al., 2023 ; Yi et al., 2024 ; Mo et al., 2024 ) . This success has prompted a growing number of Software Engineering (SE) researchers to integrate LLMs into solving diverse SE tasks, yielding promising outcomes ( Sun et al., 2024 ; Xia and Zhang, 2023 ; Chen et al., 2024 ; Liu et al., 2024 ; Dong et al., 2024 ; Kang et al., 2023 ; Kang et al., 2024 ) . Despite these successes, significant challenges persist in effectively using LLMs for task completion ( Satpute et al., 2024 ; Yu et al., 2024 ; Koo et al., 2023 ; Chen et al., 2024 ) . Harnessing LLMs in SE to maximize their exceptional in-context learning and reasoning capabilities ( Gao et al., 2023 ; Yao et al., 2022 ; Wei et al., 2022 ; Wang et al., 2023 ) , relies on the prompts used, the information provided, and the specific ways in which models are invoked ( Sun et al., 2024 ; Xu et al., 2023b ; Zhong et al., 2024 ) . In response, various strategies have emerged, often termed ‘‘prompt engineering techniques’’, which aim to optimize LLM performance beyond simple model calls. These techniques 1 1 1 To avoid confusion, following previous work ( Sun et al., 2024 ) , the term “techniques” in this paper specifically refers to prompt engineering techniques, while specific approaches based on these techniques will be referred to as “approaches”. include few-shot prompting ( Gao et al., 2023 ) , Chain-of-Thought (CoT) prompting ( Wei et al., 2022 ; Yao et al., 2022 ) , critique prompting ( Ma et al., 2024 ; Kim et al., 2024 ) , expert prompting ( Xu et al., 2023b ) , and so on. While these techniques have proven effective across different software development and maintenance tasks, such as code generation, code understanding, software testing, and debugging, challenges like hallucinations and inaccuracies remain ( Zhong et al., 2024 ; Dong et al., 2024 ; Pan et al., 2024 ; Chen et al., 2024 ) . To address these limitations, researchers have shifted focus to dynamic strategies ( Dong et al., 2024 ; Huang et al., 2023 ; Pan et al., 2024 ; Yang et al., 2024 ) involving continuous interaction with the LLMs, task decomposition, and result verification. Building upon these concepts, recent approaches employ prompt engineering techniques such as multi-agent systems ( Ma et al., 2024 ; Dong et al., 2024 ) , iterative refinement processes ( Ma et al., 2024 ; Yang et al., 2024 ; Zhong et al., 2024 ) , and the integration of additional contextual information to refine the LLM’s output ( Zhong et al., 2024 ; Yang et al., 2024 ) . By leveraging these prompt engineering techniques, LLMs can deliver more reliable results in complex SE tasks, paving the way for further advancements in SE research. With the rapid advancements in large language model training ( OpenAI, 2021 ; DeepSeek-AI, 2024 ; Anthropic, 2023 ; DeepMind, 2023 ; AI, 2023 ) , foundational models are being iterated and updated at an accelerated pace ( OpenAI, 2024b ; OpenAI, 2024c ; OpenAI, 2024d ) . More advanced foundational models demonstrate improved understanding and generating capabilities. When OpenAI released the GPT-4o model ( OpenAI, 2024b ) , its performance outperformed that of most prompt engineering techniques developed for earlier foundational LLMs in coding tasks ( pap, 2024 ) . The subsequent o1, o1-mini models ( OpenAI, 2024c ; OpenAI, 2024d ) integrate CoT reasoning, allowing reasoning-based LLMs to autonomously decompose complex problems into a series of simpler steps, thereby forming effective strategies for tackling intricate logical issues. However, many prompt engineering techniques for code ( Gao et al., 2023 ; Xu et al., 2023b ; Dong et al., 2024 ) were developed based on the capabilities of the earlier model, ChatGPT-3.5 ( OpenAI, 2021 ) , as it was the only option available at the time. This overlooks the enhancements offered by the more advanced GPT-4o ( OpenAI, 2024b ) and the reasoning capabilities of the o1 and o1-mini models ( OpenAI, 2024c ; OpenAI, 2024d ) . Moreover, OpenAI’s guidance indicates that using complex prompts is not recommended for reasoning LLMs ( OpenAI, 2024c ; OpenAI, 2024d ) . Thus, this raises the first question about ① the effectiveness of these prompt engineering techniques on the more advanced models . Furthermore, while it is claimed that the reasoning LLMs, i.e., o1 and o1-mini, may provide enhanced performance, ② what is its actual effectiveness compared to non-reasoning models, and what are its respective advantages and disadvantages ? Additionally, the reasoning LLMs typically incur higher operational costs, both in terms of monetary expenditure and time efficiency ( OpenAI, 2024c ; ope, 2024 ) . In addition to computational and token-based costs, the varying levels of carbon emissions are a critical consideration. This raises the third question: ③ do the benefits of utilizing these advanced models justify their increased costs? To explore these questions, this paper presents the first extensive

Read the complete text