Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions
Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović · 2025
Read Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions by Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13291.txt.
Description
Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, and Nenad Medvidović published a paper in 2025 on whether deep learning tools that flag buggy, hackable code are reading the right clues. Reported scores for such detectors have repeatedly failed to reproduce on new data. The authors suspect the models lean on features that merely travel with vulnerability rather than cause it.
They built a shared representation of true and spurious code features. Then they edited programs two ways. Some edits keep the flaw while changing surface features. Others remove the flaw while keeping the surface. Across five top detectors, removing the flaw still left the alarm ringing in about eighty-four percent of cases. Surface edits rarely flipped a verdict. The tools, the evidence says, mostly learn the decoration, not the defect.
Catalog record
| CREATOR | Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13291