Machine Learning Experiences A story of learning AI for use in enterprise software testing that can be used by anyone

Michael Cohoon, Debbie Furman · 2025

Read Machine Learning Experiences A story of learning AI for use in enterprise software testing that can be used by anyone by Michael Cohoon, Debbie Furman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22064.txt.

Introduction

Michael Cohoon and Debbie Furman (2025) tell the story of a software testing team learning machine learning from scratch, and turn it into a guide anyone can follow. The team applied learning methods to enterprise software testing. Their route mirrors the CRISP-DM process used across data projects. The steps run in order: gather the data, clean it, engineer features, split data into training and testing sets, choose a model, train it, then test and evaluate.

The paper's value is its plainness. Rather than presenting a novel algorithm, it documents how a real team worked through a standard workflow. The authors claim that following this sequence lets anyone apply machine learning effectively to their own project. For testing teams eyeing automation, the story offers a map drawn by people who walked it without a research lab behind them.

Catalog record

CREATORMichael Cohoon, Debbie Furman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Machine Learning Experiences A story of learning AI for use in enterprise software testing that can be used by anyone Michael Cohoon, Debbie Furman Abstract 1 Abstract This paper details the machine learning (ML) journey of a group of people focused on software testing. It tells the story of how this group progressed through a ML workflow (similar to the CRISP-DM process). This workflow consists of the following steps and can be used by anyone applying ML techniques to a project: gather the data; clean the data; perform feature engineering on the data; splitting the data into two sets, one for training and one for testing; choosing a machine learning model; training the model; testing the model and evaluating the model performance. By following this workflow, anyone can effectively apply ML to any project that they are doing. 2 Why You Should Read This Paper We are a group working in the software testing space and we were curious as to how machine learning could apply to our roles. We decided to write this paper to share our journey. We started by defining our problem statement and then we identified what data existed that would help us. Along the way we learned about different machine learning algorithms and we figured out how to model our data to fit into these algorithms. We continuously learned what worked and what did not work for our problem statement. We are not experts in the field of machine learning. We are not trained data scientists. We are experts in the domain of software testing. We applied this workflow to our domain, however this workflow can be applied to any domain (ie. a domain other than software testing), and therefore any learning project. As we were educating ourselves on machine learning, we came across terms that we never heard before, whether they were statistical terms or machine learning terms. In this paper we will introduce these terms and attempt to define them in a simple to understand way, giving you an understanding of terminology that we did not know or understand at the beginning of our journey. If you too are in the early stages of learning about machine learning techniques, then this paper is for you. We will identify pitfalls that we ran into and successes to help you in your own journey. The following figure (figure 1 ) is a workflow of the steps needed to apply machine learning to any project. We will start by describing the activity of gathering data. Next we will move on to describe what it means to clean the data. Part of cleaning the data is something called feature engineering. We will then touch upon how and why you will need to split the data into a training and a test set. The next step is to explore the available models and choose one to experiment with your dataset. Once a model is chosen, it will need to be trained using your specific data. After the model is trained, then we talk about testing the model and evaluating the model performance. It will be useful to keep the following ML process flow (see figure 1 ) in mind to understand the rest of the paper. 3 Disclaimer In this paper, we will demonstrate our journey through understanding the basics in machine learning by sharing resources, methods, ideas, and code snippets. Please understand that these examples are not endorsements. There may be alternative or better ways that we do not discuss. If you are beginning your machine learning journey, we hope that this paper is but one resource you can learn from. Start Craft Problem Statement Craft Solution Strategy Gather Data Choose and Clean Data Identify Relevant Features Feature Engineering Choose Model Split Data Train/Test Train Model Test Model Evaluate Model Performance Model approach still believed adequate Model performance acceptable End no no yes yes Figure 1: Machine Learning Process Flow 4 Problem Statement As software testers, we started by asking: what problems do we currently have? One of the problems that is challenging and complex is handling defects that are found both during the development cycle and in the field. For the field defects, we wondered if we could use machine learning to help predict when a fix for a field defect would have an error in it. For the defects found during the development cycle, we wondered if we could use machine learning to prune out duplicate or invalid defects. Throughout this paper, we will be referencing both these problem statements to give examples of how we approached our education of machine learning techniques. We start our journey with the first lesson: namely, that machine learning is all about the data. 5 It’s all about the Data Let’s look at the ”Gather Data” stage in the ML process flow (see figure 2 ). Start Craft Problem Statement Craft Solution Strategy Gather Data Choose and Clean Data Identify Relevant Features Feature Engineering Choose Model Split Data Train/Test Train Model Test Model Evaluate Model Performance Model approach still believed adequate Model performance acceptable End no no yes yes Figure 2: Gather Data For the application of machine learning, it is important to have both a problem to solve (or a business objective) and the data to assist in solving that problem. In figure 2 we show a workflow of steps we used in our machine learning journey, from identifying the problem we wanted to solve to moving through different stages of manipulating the data needed to assist our use of machine learning to solve the problem. We first asked questions around what problems we currently had in the testing space that we would like to solve. This defined our business objective. We then determined what data was available to help us solve these problems. For the use cases that we will go through in this paper, we have defect data both from the field after the software has been released to customers and defect data that is gathered during the software development cycle prior to being released to customers. We say that it is important to have both a problem and data. Even if we have a problem to solve, machine learning is impossible if we don’t have the data that supports the problem being looked at. This is because machine learning requires a large amount of data in order to train the machine learning algorithm to solve a problem. That is why we say it is all about the data. Example 1. Suppose we have a business objective of predicting if a code fix needs special testing or review. It would not be valuable to answer this question with machine learning if we did not have labeled data that captures whether or not a previous code fix introduced an issue. While it is important to have the appropriate data to build a machine learning system, it is often the case that data exists without a known business objective. Have you ever worked on a software project where you were asked to collect data without any knowledge of how the data was going to be used? In the software testing space, we collect information regarding defects and fixes,

Read the complete text