Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions

Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović · 2025

Read Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions by Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13291.txt.

Description

Satyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, and Nenad Medvidović published a paper in 2025 on whether deep learning tools that flag buggy, hackable code are reading the right clues. Reported scores for such detectors have repeatedly failed to reproduce on new data. The authors suspect the models lean on features that merely travel with vulnerability rather than cause it.

They built a shared representation of true and spurious code features. Then they edited programs two ways. Some edits keep the flaw while changing surface features. Others remove the flaw while keeping the surface. Across five top detectors, removing the flaw still left the alarm ringing in about eighty-four percent of cases. Surface edits rarely flipped a verdict. The tools, the evidence says, mostly learn the decoration, not the defect.

Catalog record

CREATORSatyaki Das, Syeda Tasnim Fabiha, Saad Shafiq, Nenad Medvidović
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13291

Browse the full library · About the Tower of Babel