Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis Conference: Poster at ACM Secure Development Conference; July 5–6, 2026; Montreal, QC, Canada Poster at ACM Secure Development Conference (SecDev ’26), July 5–9, 2026, Montreal, QC, Canada

Jayson Ng, Amin Milani Fard · 2026

Read Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis Conference: Poster at ACM Secure Development Conference; July 5–6, 2026; Montreal, QC, Canada Poster at ACM Secure Development Conference (SecDev ’26), July 5–9, 2026, Montreal, QC, Canada by Jayson Ng, Amin Milani Fard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03140.txt.

Introduction

Jayson Ng and Amin Milani Fard test a common belief: that adding retrieved knowledge improves language model explanations of malware. Analysts increasingly use such models to summarize malicious behavior, often with VirusTotal reports as structured input.

Across several models, retrieval often made explanations worse, adding distracting context, noise, and generic filler. The authors argue malware explanation is mainly about extracting signal from evidence already present, not fetching more knowledge. They close with design advice for secure workflows, warning that retrieval can hurt in security-critical pipelines where the source evidence is already sufficient.

Catalog record

CREATORJayson Ng, Amin Milani Fard
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Retrieval-Augmented Generation for Explainable Malware Analysis Conference: Poster at ACM Secure Development Conference; July 5–6, 2026; Montreal, QC, Canada Poster at ACM Secure Development Conference (SecDev ’26), July 5–9, 2026, Montreal, QC, Canada Jayson Ng, Amin Milani Fard Abstract . Large Language Models (LLMs) are increasingly being used as security engineering tools to summarize and explain malware behavior to analysts. A common assumption is that Retrieval-Augmented Generation (RAG) improves explanation quality by injecting external security knowledge. In this work, we empirically evaluate this assumption for malware explanation using VirusTotal reports as structured input. Across multiple LLMs, we find that RAG frequently degrades explanation quality by introducing distracting or weakly related context and adding narrative noise or generic write - ups. Our results highlight a practical risk in security-critical pipelines for malware explanation that RAG can be counterproductive when structured security evidence is already sufficient. We argue that malware explanation is primarily a signal-extraction task, not a knowledge-retrieval problem, and outline design recommendations for secure development workflows. † † cc-license: by 1. Introduction Understanding why a binary is malicious is as important as determining whether it is malicious. While platforms such as VirusTotal provide rich detection labels, behavioral traces, and metadata, they do not offer analyst - ready explanations that connect low - level indicators to high - level attacker intent. These outputs lack a coherent causal narrative explaining why certain behaviors matter, how execution unfolds, or how signals indicate malicious activity. Such explainability is essential for threat hunting, incident response, and regulatory or audit requirements. Recently, LLMs have been proposed as security engineering tools to translate low-level indicators (e.g., API calls, registry edits, and network activity) into natural-language explanations that support triage, incident response, and reporting. For example, MalGPT ( 9 ) demonstrates that generative models can produce meaningful explanations from malware binaries by learning latent behavioral patterns. RAG ( 6 ) is often treated as a default enhancement for LLM-based systems, and has been applied to related pipelines such as CVE association and reasoning over decompiled artifacts ( 2 ; 7 ) . VulRAG ( 3 ) shows that incorporating structured domain knowledge improves reasoning about software vulnerabilities and reduces hallucinations. However, prior studies do not evaluate whether RAG improves explanation quality in malware analysis. Unlike domains that rely on external documentation, malware reports often encapsulate sufficient contextual evidence, raising the operational risk that indiscriminate retrieval introduces distraction or noise leading to misleading explanations in security workflows. 2. Proposed Approach We evaluate the explainability of LLMs with and without RAG for malware explanation tasks when structured evidence already exists. Figure 1. RAG LLM architecture for malware explanation. Implementation. We use LlamaIndex as the RAG orchestration framework, ChromaDB for persistent vector storage, and OpenRouter as the LLM gateway. We apply all - MiniLM - L6 - v2 and OpenAI’s text - embedding - 3 - large for embeddings. Documents are chunked into 1024 tokens with a 100 - token overlap. Retrieval employs top - k similarity search with a 0.5 threshold. The knowledge base consists of 26 documents parsed using LlamaIndex’s SimpleDirectoryReader. Our dataset is derived from the MalGPT corpus ( 9 ) with 1,702 VirusTotal reports, including structured indicators such as API calls and registry operations. MD5 hashes serve as identifiers. Our implementation is available for download 1 1 1 https://github.com/nyit-vancouver/RAG-Explain-Malware . System prompt. The prompt has 10 sections: (1) one - line verdict with confidence, (2) non - technical summary, (3) evidence bullets with JSON citations, (4) malware family mapping, (5) grouped Indicator of Compromises (IoCs) with paths, (6) behavioral summary, (7) confidence score, (8) recommended actions for technical and non - technical audiences, (9) optional FAQ, and (10) raw evidence appendix. This structure enforces consistency and traceability. Knowledge base. The knowledge base integrates four components: (1) the MITRE ATT&CK framework (Enterprise, ICS, Mobile); (2) research and best - practice guidance, including MalGPT and CISA/MITRE mapping recommendations; (3) VirusTotal technical documentation covering behavioral schemas, reports, and IoC definitions; and (4) malware intelligence datasets, including malware family taxonomies and CVE records. Evaluation. We conduct an ablation study comparing LLMs with and without RAG against GPT - 5.1 (Table 1 ). Each model runs under identical prompts using either VirusTotal JSON alone or JSON augmented with retrieved context. To isolate retrieval effects, no fine - tuning is performed. We do not compare directly with MalGPT, as it represents a different paradigm based on a trained multi - modal architecture rather than external retrieval. Since our focus is on retrieval - induced effects when explaining structured sandbox evidence, such a comparison would not isolate the impact of RAG. We measure explanation quality using BERTScore ( 10 ) , which captures semantic similarity using contextual embeddings and is robust to paraphrasing and structural variation suitable for evaluating malware explanations where equivalent meanings may be expressed using different terminology. It is also sensitive to semantic drift introduced by loosely related retrieved passages. Model # Parameters Embeddings BERTScore GPT OSS 21B None 0.8407 GPT OSS 21B all - MiniLM - L6 - v2 0.8184 GPT OSS 21B text - embedding - 3 - large 0.8493 DeepSeek-R1 671B None 0.8617 DeepSeek-R1 671B all - MiniLM - L6 - v2 0.8583 DeepSeek-R1 671B text - embedding - 3 - large 0.8582 GPT - 5.1 est. 2.5T None 1.0000 GPT - 5.1 est. 2.5T all - MiniLM - L6 - v2 0.8778 GPT - 5.1 est. 2.5T text - embedding - 3 - large 0.9047 Table 1. Malware explanation compared to GPT - 5.1 w/o RAG. 3. Discussion Results analysis. Based on commonly observed ranges in the literature, BERTScores 0.9-1 indicate excellent alignment, scores between 0.85 and 0.9 reflect strong semantic similarity, and scores between 0.8 and 0.85 indicate moderate quality. Differences as small as 0.01–0.02 are meaningful in technical domains. Excluding GPT - 5.1, the highest score is achieved by DeepSeek-R1 without RAG (0.8617), indicating that retrieval does not effectively improve explanation quality. As shown in Table 1 , the all - MiniLM - L6 - v2 retriever degrades performance for both models, particularly GPT - OSS - 20B. While text - embedding - 3 - large improves GPT - OSS - 20B slightly, it does not benefit DeepSeek-R1. Compact embeddings often retrieve semantically similar but low - utility passages, such as generic malware descriptions, which distract the generator. Prior work formalizes this distracting effect and shows that weakly relevant retrieval degrades performance ( 1 ) . Long - context RAG studies similarly observe performance decline as retrieved context accumulates hard negatives ( 4 ) . These findings align with our observations. VirusTotal JSON reports already provide tightly scoped behavioral evidence. Injecting external descriptions shifts model attention toward plausible but irrelevant narratives, reducing semantic alignment. For strong models such as DeepSeek-R1, additional context appears redundant or mildly contradictory ( 5 ) . In this setting, malware explainability is primarily a signal - extraction problem rather than a knowledge - retrieval one. Retrieval thresholds were intentionally permissive to expose potential distractions. Security engineering implications. Our findings highlight important implications for secure system design. The risk of automation bias increases when RAG-based explanations that drift from the underlying evidence are not manually verified. Analysts benefit more from

Read the complete text