Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions
Amirali Sajadi (corresponding author), Binh Le, Anh Nguyen, Kostadin Damevski, Preetha Chatterjee · 2025
Read Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions by Amirali Sajadi (corresponding author), Binh Le, Anh Nguyen, Kostadin Damevski, Preetha Chatterjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14202.txt.
Description
Amirali Sajadi and co-authors ask, in a 2025 study, whether chatbots that help with coding actually warn users about unsafe code. Millions of programmers now paste questions and code into large language models, so silent acceptance of flawed code could spread real harm.
The team fed three leading models questions from a coding forum that contained known flaws. A good answer would name the flaw, explain how it could be exploited, and offer a fix. In practice, the models caught only a fraction of the flaws, with detection ranging from about thirteen to forty percent. They spotted some flaw types often and others rarely, leaving much unsafe code unflagged.
Catalog record
| CREATOR | Amirali Sajadi (corresponding author), Binh Le, Anh Nguyen, Kostadin Damevski, Preetha Chatterjee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14202