Improving QA Efficiency with DistilBERT: Fine-Tuning and Inference on mobile Intel CPUs

Ngeyen Yinkfu · 2025

Read Improving QA Efficiency with DistilBERT: Fine-Tuning and Inference on mobile Intel CPUs by Ngeyen Yinkfu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22937.txt.

Introduction

Ngeyen Yinkfu fine-tunes a compact language model for question answering on an ordinary laptop processor, in a 2025 paper. Big transformer models answer well but demand heavy hardware. Many real uses run on modest machines. DistilBERT, a slimmed BERT, promises a workable middle ground. The study tests that promise end to end.

Training used the standard SQuAD question set. Data exploration and augmentation preceded careful fine-tuning. The model reached a solid score while answering each question in a fraction of a second. A rule-based baseline scored far lower. Full BERT models cost much more compute for their gains. Developers of offline assistants gain a recipe that fits consumer hardware.

Catalog record

CREATORNgeyen Yinkfu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving QA Efficiency with DistilBERT: Fine-Tuning and Inference on mobile Intel CPUs Ngeyen Yinkfu Abstract This study presents an efficient transformer-based question-answering (QA) model optimized for deployment on a 13th Gen Intel i7-1355U CPU, using the Stanford Question Answering Dataset (SQuAD) v1.1. Leveraging exploratory data analysis, data augmentation, and fine-tuning of a DistilBERT architecture, the model achieves a validation F1 score of 0.6536 with an average inference time of 0.1208 seconds per question. Compared to a rule-based baseline (F1: 0.3124) and full BERT-based models, our approach offers a favorable trade-off between accuracy and computational efficiency. This makes it well-suited for real-time applications on resource-constrained systems. The study includes systematic evaluation of data augmentation strategies and hyperparameter configurations, providing practical insights into optimizing transformer models for CPU-based inference. 1 Introduction Question answering (QA) systems have become a cornerstone of natural language processing (NLP), enabling machines to extract precise answers from textual contexts. The Stanford Question Answering Dataset (SQuAD) v1.1 ( Rajpurkar et al., 2016 ) is a widely adopted benchmark for evaluating QA models, comprising over 87,000 training examples of context-question-answer triples. While transformer-based models like BERT ( Devlin et al., 2019 ) have achieved state-of-the-art performance on SQuAD, their computational complexity often demands GPU acceleration, limiting deployment on resource-constrained devices like mid-range CPUs. This study addresses the challenge of developing a transformer-based QA model optimized for inference on a 13th Gen Intel i7-1355U CPU, a 10-core processor with a 5.0 GHz turbo frequency. We focus on DistilBERT ( Sanh et al., 2020 ) , a lightweight transformer, to balance performance—measured by F1 score and accuracy—with inference speed. Our contributions include: • Comprehensive exploratory data analysis (EDA) of SQuAD v1.1 to inform modeling decisions. • Data augmentation strategies to enhance model robustness to low-overlap question-context pairs. • Fine-tuning and optimization of DistilBERT, achieving a validation F1 score of 0.6536 and an average inference time of 0.1208 seconds per question. • Comparative evaluation against a rule-based baseline and BERT-based configurations, highlighting efficiency trade-offs. This paper details the dataset, methodology, results, and implications for deploying QA systems in resource-constrained environments. 2 Related Work The advent of transformer-based architectures has significantly advanced question-answering (QA) systems, particularly on benchmarks like SQuAD v1.1 ( Rajpurkar et al., 2016 ) . Below, we discuss key QA models, their architectures, performance, and optimization strategies, situating our work within this landscape. 2.1 Transformer-Based QA Models The introduction of BERT ( Devlin et al., 2019 ) marked a breakthrough in QA, achieving an F1 score of 88.5 on SQuAD v1.1 through bidirectional pre-training on large corpora. BERT’s 12-layer (BERT-Base) architecture with 110 million parameters leverages self-attention to capture contextual relationships, making it highly effective for span-based QA tasks. However, its computational complexity poses challenges for deployment on resource-constrained devices. Subsequent models have built on BERT’s foundation. RoBERTa ( Liu et al., 2019 ) enhances BERT by optimizing pre-training with larger datasets and longer training, achieving an F1 score of 91.5 on SQuAD v1.1. ALBERT ( Lan et al., 2020 ) reduces parameters through factorized embeddings and cross-layer parameter sharing, reaching an F1 score of 90.9 with lower memory demands. XLNet ( Yang et al., 2020 ) integrates autoregressive pre-training, outperforming BERT with an F1 score of 89.9 by modeling permutations of input sequences. T5 ( Raffel et al., 2023 ) frames QA as a text-to-text task, achieving competitive performance (F1: 90) by treating all NLP tasks as sequence generation. ELECTRA ( Clark et al., 2020 ) introduces a discriminative pre-training objective, where a smaller model detects replaced tokens, yielding an F1 score of 90.8 with improved efficiency over BERT. While these models achieve high performance, their large parameter counts (e.g., RoBERTa: 125M, T5: up to 11B) and computational requirements make them less practical for CPU-based inference, motivating our focus on lightweight models like DistilBERT ( Sanh et al., 2020 ) . DistilBERT, with 66 million parameters, is a distilled version of BERT, retaining 97% of its performance (F1: 85 on SQuAD) while reducing inference time, as demonstrated in Wolf et al. (2020) . MobileBERT ( Sun et al., 2020 ) further optimizes for mobile devices, achieving an F1 score of 84.3 with 25 million parameters, but its applicability to standard CPUs remains underexplored. 2.2 Optimization for Efficiency To address computational constraints, several optimization techniques have been proposed. Knowledge distillation, as used in DistilBERT ( Sanh et al., 2020 ) , transfers knowledge from a larger model (e.g., BERT) to a smaller one, reducing parameters while preserving performance. Quantization, explored by Shen et al. (2021) , compresses model weights to lower precision (e.g., 8-bit integers), reducing inference time on CPUs. Pruning techniques, such as those in Michel et al. (2019) , remove redundant attention heads or layers, further enhancing efficiency. For example, Zafrir et al. (2019) applied 8-bit quantization to BERT, achieving near-original performance with faster inference. However, these studies primarily target edge devices or GPUs, with limited focus on mid-range CPUs like the Intel i7-1355U. 2.3 Data Augmentation in QA Data augmentation enhances QA model robustness, particularly for low-overlap question-context pairs. Yang et al. (2019) employed synonym replacement and back-translation, increasing SQuAD training data diversity and improving generalization. Similarly, Wei and Zou (2019) proposed easy data augmentation (EDA) techniques, including synonym substitution and random insertion, which improved performance on smaller datasets. These strategies align with our approach of using WordNet-based paraphrasing to augment questions and contexts, addressing the moderate overlap (mean: 0.52) observed in SQuAD. 2.4 Positioning Our Work Our study builds on these advancements by fine-tuning DistilBERT for SQuAD v1.1, emphasizing CPU inference on a mid-range processor. Unlike high-parameter models (e.g., RoBERTa, T5), we prioritize efficiency, achieving a competitive F1 score (0.6536) with an inference time of 0.1208 seconds per question. Compared to Shen et al. (2021) and Michel et al. (2019) , our work focuses on standard CPU hardware, and our augmentation strategies extend Yang et al. (2019) by combining question and context paraphrasing. This positions our work as a practical solution for real-time QA in resource-constrained environments, addressing a gap in deploying transformer-based QA models on CPUs. 3 Dataset Description The SQuAD v1.1 dataset ( Rajpurkar et al., 2016 ) contains 87,599 training examples and 10,570 validation examples, each comprising a Wikipedia-sourced context, a question, and an answer span. Its diversity and scale make it a standard benchmark for QA tasks. 4 Exploratory Data Analysis We conducted exploratory data analysis (EDA) on SQuAD v1.1’s training split to guide model design. Key statistics include: • Training Set Size : 87,599 examples; Validation Set Size : 10,570 examples. • Question Length : Mean of 10.06 words, standard deviation of 4.32, maximum of 40 words. • Context Length : Mean of 119.76 words, standard deviation of 52.14, maximum of 653 words. • Answer Length : Mean of 3.16 words, standard deviation of 2.87, maximum of 43 words. • Question-Context Overlap : Mean proportion of 0.52, standard deviation of 0.19. Visualizations (Figures 1 , 2 , 3 ) reveal right-skewed distributions for question and context lengths, short answer spans (81% between 1–5 words), near-uniform answer positions, and moderate question-context overlap (62% between 0.4–0.7). These findings informed a maximum sequence length of 384 tokens, data augmentation for low-overlap examples, and the choice

Read the complete text