AutoDFBench 1.0: A Benchmarking Framework for Digital Forensic Tool Testing and Generated Code Evaluation Journal: DFRWS EU 2026

Akila Wickramasekara, Tharusha Mihiranga, Aruna Withanage, Buddhima Weerasinghe, Frank Breitinger, John Sheppard, Mark Scanlon · 2025

Read AutoDFBench 1.0: A Benchmarking Framework for Digital Forensic Tool Testing and Generated Code Evaluation Journal: DFRWS EU 2026 by Akila Wickramasekara, Tharusha Mihiranga, Aruna Withanage, Buddhima Weerasinghe, Frank Breitinger, John Sheppard, Mark Scanlon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16965.txt.

Introduction

Akila Wickramasekara and coauthors introduce AutoDFBench 1.0, a framework that automates testing of digital forensic tools under the NIST Computer Forensic Tool Testing programme, the de facto standard for validating such tools. No shared automated bench covered its diverse tasks before.

The bench spans five task areas: string search, deleted file recovery, file carving, Windows registry recovery, and SQLite recovery. Ground truth holds 63 test cases and 10,968 scenarios, scored through a REST API with precision, recall, and F1 outputs whose average forms the AutoDFBench Score. The design lets vendors, researchers, and standards bodies compare tools and AI-written code fairly and repeatably.

Catalog record

CREATORAkila Wickramasekara, Tharusha Mihiranga, Aruna Withanage, Buddhima Weerasinghe, Frank Breitinger, John Sheppard, Mark Scanlon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoDFBench 1.0: A Benchmarking Framework for Digital Forensic Tool Testing and Generated Code Evaluation Journal: DFRWS EU 2026 Akila Wickramasekara, Tharusha Mihiranga, Aruna Withanage, Buddhima Weerasinghe, Frank Breitinger, John Sheppard, Mark Scanlon Abstract The National Institute of Standards and Technology (NIST) Computer Forensic Tool Testing (CFTT) programme has become the de facto standard for providing digital forensic tool testing and validation. However to date, no comprehensive framework exists to automate benchmarking across the diverse forensic tasks included in the programme. This gap results in inconsistent validation, challenges in comparing tools, and limited validation reproducibility. This paper introduces AutoDFBench 1.0, a modular benchmarking framework that supports the evaluation of both conventional DF tools and scripts, as well as AI-generated code and agentic approaches. The framework integrates five areas defined by the CFTT programme: string search, deleted file recovery, file carving, Windows registry recovery, and SQLite data recovery. AutoDFBench 1.0 includes ground truth data comprising of 63 test cases and 10,968 unique test scenarios, and execute evaluations through a RESTful API that produces structured JSON outputs with standardised metrics, including precision, recall, and F1 score for each test case, and the average of these F1 scores becomes the AutoDFBench Score . The benchmarking framework is validated against CFTT’s datasets. The framework enables fair and reproducible comparison across tools and forensic scripts, establishing the first unified, automated, and extensible benchmarking framework for digital forensic tool testing and validation. AutoDFBench 1.0 supports tool vendors, researchers, practitioners, and standardisation bodies by facilitating transparent, reproducible, and comparable assessments of DF technologies. 1 Introduction The rapid growth of digital crimes, accelerated by the emergence of generative Artificial Intelligence (AI) technologies, has significantly increased the workload and complexity faced by digital forensic (DF) investigators [ 18 , 5 ] . To address these challenges, there is a growing demand for robust, accurate, and consistent digital forensic tools that can ensure reliability and reproducibility in evidence analysis. However, one of the most persistent issues in this domain remains the lack of standardisation in tool validation and evaluation methodologies [ 3 ] . Recent advancements in large language models (LLMs) and autonomous AI agents have shown great potential in enhancing the efficiency and productivity of digital investigations [ 20 , 15 , 10 ] . These systems can generate digital forensic scripts, automate repetitive tasks, and assist in interpreting complex data. Nevertheless, despite their promise and growing interest from the digital forensic research community [ 1 ] , there is still no appropriate or standardised mechanism to validate and benchmark the accuracy, reliability, and forensic soundness of AI-generated source code and scripts. This research addresses that gap by focusing on the significant enhancement and expansion of a comprehensive unified benchmarking framework for evaluating both conventional and AI-generated digital forensic tools and scripts, AutoDFBench. AutoDFBench builds upon the principles of the National Institute of Standards and Technology (NIST) Computer Forensics Tool Testing (CFTT) programme and provides an automated, extensible, and standardised mechanism for reproducibly evaluating forensic tools across multiple investigation scenarios. This paper advances the framework from an initial proof-of-concept to a 1.0 release version, covering the breadth of CFTT software test suites needed for computer forensic software tool testing and validation. This work makes the following contributions: 1. Enhance and expand the AutoDFBench framework through the incorporation of an improved API layer, a new CSV input layer, and the expansion of the set of test suites from one to five. The framework is released open source under the Apache 2.0 License via the project’s GitHub repository ( https://github.com/akila-UCD/AutoDFBench ). 2. The preparation and integration of comprehensive ground truth datasets derived from the NIST CFTT programme, encompassing 63 test cases and 10,968 unique test scenarios across five test suites. 3. The experimental evaluation of the framework using NIST provided datasets demonstrates its reliability, accuracy, and consistency in benchmarking digital forensic tools and AI-generated code. Table 1: Forensic String Search Test Cases Test Case Description FT-SS-01 Test the ability to identify a plain ASCII string consisting of a single word. FT-SS-02 Assess whether a tool can locate an ASCII string regardless of upper or lowercase variations. FT-SS-03 Verify that a tool matches whole words exactly and does not detect substrings, and ignores case. FT-SS-04 Evaluate searches that require two different terms to be present within the same file. FT-SS-05 Check if a tool can detect the presence of at least one string from a given set of words. FT-SS-06 Confirm whether a search can identify a string while excluding content containing another specified term. FT-SS-07 Determine support for searching text in non-English scripts, including Asian characters, Hangul, Kana, Cyrillic, Latin with accents, and right-to-left languages. FT-SS-08 Assess built-in search functions for structured data types such as email addresses, phone numbers, or identification numbers. FT-SS-09 Measure performance in scenarios such as formatted documents, fragmented files, inaccessible storage areas, NTFS metadata, substrings in file names, and word stemming. FT-SS-10 Examine the handling of pattern-based searches using regular expressions, including hexadecimal matching and simple character sets. 2 Computer Forensics Tool Testing Programme The CFTT programme 1 1 1 https://www.nist.gov/itl/ssd/software-quality-group/computer-forensics-tool-testing-program-cftt provides a structured methodology for evaluating digital forensic tools, aiming to ensure reliable, repeatable, and standardised results that withstand legal scrutiny. It includes formal test specifications and publishes tool testing results that assess performance across tasks, including string search, file carving, deleted file recovery, registry analysis, database extraction, write blocking, cloud data extraction, media preparation, and mobile devices. These suites are widely recognised as de facto standards for independent DF tool validation. In this work, CFTT underpins AutoDFBench 1.0, supplying the benchmark datasets and specifications required for automated, reproducible, and comparable evaluations of forensic tools and scripts. 2.1 Forensic String Search (FSS) In FSS, NIST emphasises two main requirements for a string search tool. One requirement is that the tool returns exact matches based on the given query, and the other is that it should support at least one character representation for searching. To address these requirements, forensic string search is accompanied by ten primary test cases defined by the NIST CFTT programme, which are explained in Table 1 . These ten test cases are subdivided into sub-test cases, as each test case consists of different string values, file statuses (active, deleted, and unallocated), and the disk operating system. Table 2: Deleted File Recovery Core Test Cases Test Case Description DFR-01 Evaluates the recovery of a single non-fragmented file. DFR-02 Assesses the recovery of a file consisting of two fragments. DFR-03 Examines the recovery of a file that has multiple fragments. DFR-04 Tests recovery of several non-fragmented files with non-Latin character filenames. DFR-05 Evaluates the recovery of two fragmented files. DFR-06 Check the recoverability of a single large file. DFR-07 Tests recovery of one file that has been overwritten. DFR-08 Examines the recovery of multiple overwritten files. DFR-09 Evaluates recovery when a large number of files are deleted without overwriting. DFR-10 Assesses recovery of a large number of files that were deleted with partial overwriting. DFR-11 Tests recovery of files deleted from a single directory. DFR-12 Assesses recovery of files deleted across multiple directories. DFR-13 Checks recovery of chaotic file system activity. DFR-14 Evaluates recovery of non-standard or other file system objects. 2.2 Deleted File Recovery

Read the complete text