Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms’ “Typo” Correction

Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, Marcus Botacin · 2025

Read Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms’ “Typo” Correction by Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, Marcus Botacin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11622.txt.

Introduction

Eavesdroppers can in principle recover typed text from the sound of keystrokes, but real rooms are noisy and current models falter there. Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, and Marcus Botacin attack the problem from two sides. Vision transformers read the sound spectrograms and capture longer context than the older convolutional models, beating the previous benchmark at classifying keystrokes. A large language model then fixes the resulting typos, using the fact that people type words that fit the conversation rather than random strings.

The authors show the two strategies are complementary and together make acoustic side-channel attacks practical under realistic noise, a result with clear security implications.

Catalog record

CREATORSeyyed Ali Ayati, Jin Hyun Park, Yichen Cai, Marcus Botacin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms’ “Typo” Correction Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, Marcus Botacin Abstract Abstract The large integration of microphones into devices increases the opportunities for Acoustic Side-Channel Attacks (ASCAs), as these can be used to capture keystrokes’ audio signals that might reveal sensitive information. However, the current State-Of-The-Art (SOTA) models for ASCAs, including Convolutional Neural Networks (CNNs) and hybrid models, such as CoAtNet, still exhibit limited robustness under realistic noisy conditions. Solving this problem requires either: (i) an increased model’s capacity to infer contextual information from longer sequences, allowing the model to learn that an initially noisily typed word is the same as a futurely collected non-noisy word, or (ii) an approach to fix misidentified information from the contexts, as one does not type random words, but the ones that best fit the conversation context. In this paper, we demonstrate that both strategies are viable and complementary solutions for making ASCAs practical. We observed that no existing solution leverages advanced transformer architectures’ power for these tasks and propose that: (i) Visual Transformers (VTs) are the candidate solutions for capturing long-term contextual information and (ii) transformer-powered Large Language Models (LLMs) are the candidate solutions to fix the “typos” (mispredictions) the model might make. Thus, we here present the first-of-its-kind approach that integrates VTs and LLMs for ASCAs. We first show that VTs achieve SOTA performance in classifying keystrokes when compared to the previous CNN benchmark. Second, we demonstrate that LLMs can mitigate the impact of real-world noise. Evaluations on the natural sentences revealed that: (i) incorporating LLMs (e.g., GPT-4o) in our ASCA pipeline boosts the performance of error-correction tasks; and (ii) the comparable performance can be attained by a lightweight, fine-tuned smaller LLM (67 times smaller than GPT-4o), using Low-Rank Adaptation (LoRA). Our results and findings highlight the practical viability of our solution toward effective ASCA. 1 Introduction Acoustic Side-Channel Attacks (ASCAs) exploit the acoustic emanations from electronic devices to infer sensitive information without directly accessing the targeted systems. This adversarial method represents a substantial cybersecurity risk, particularly today as microphones become ubiquitous components integrated into common consumer devices, such as smartphones, laptops, digital assistants, and conferencing equipment. Although initially perceived as a novel theoretical threat, practical acoustic attacks have emerged to compromise sensitive information in various contexts [ 50 , 10 , 11 , 35 ] . Previous research has demonstrated the feasibility of recovering typed passwords and PINs covertly by analyzing keystroke sounds [ 50 , 4 , 8 , 3 , 25 ] , reconstructing confidentially printed text from dot-matrix printer noises in sensitive environments such as medical and banking facilities [ 6 ] , and extracting cryptographic information or CPU operation details from subtle hardware acoustic signals [ 40 , 36 ] . Among these, keyboard-focused acoustic attacks have been proven to be especially threatening due to their widespread applicability and ability to compromise passwords and confidential communications effectively [ 50 , 4 , 25 , 23 ] . Early ASCA methodologies primarily depended on straightforward statistical analyses and signal-processing techniques, leveraging acoustic features like cepstrum coefficients and inter-keystroke timing patterns for keystroke recovery [ 50 ] . Despite enhancements through cross-correlation analysis and time-based localization techniques that improved keystroke identification [ 8 , 14 ] , these approaches exhibited significant limitations when faced with realistic scenarios involving ambient noise. Machine-Learning (ML) methods such as Hidden Markov Models (HMMs) and Support Vector Machines (SVMs) subsequently improved accuracy by modeling sequential structures and key feature distinctions [ 6 , 46 ] . However, these early systems remained vulnerable to real environmental conditions, where significant amounts of ambient noise and recording artifacts could drastically degrade results—potentially causing incorrect keystroke identifications. Previous research has shown accuracy reductions of 30–50% in noisy environments [ 23 , 25 ] . Our experiments confirm this, with accuracy dropping over 50% under high ambient noise. Such inaccuracies are critical, particularly when attackers aim to recover sensitive information such as passwords, where even minor incorrect inferences make full password reconstruction near-impossible. Recent Deep-Learning (DL) approaches, notably Convolutional Neural Networks (CNNs) and architectures like CoAtNet [ 12 ] , have further advanced acoustic classification accuracy by enabling effective hierarchies of acoustic feature extraction [ 25 ] . Despite these latest improvements, the fundamental challenge–vulnerability to noisy environmental conditions–remains largely unaddressed in the existing literature. Moreover, incorrect keystroke predictions under such conditions render traditional ASCA pipelines significantly less practical and reliable. Therefore, solutions for the noisy problem are still warranted. Our key observation and insight about it is that solving this problem requires either: (i) an increased model’s capacity of inferring contextual information from longer sequences, allowing the model to learn that an initially noisily typed word is the same as a futurely collected non-noisy word, making the model learn a noisy representation for the characters directly from noisy data; or (ii) an approach to fix misidentified information from the contexts, exploiting the fact that one does not type random words, but the ones that best fit the conversation context. In this case, we would train the model with non-noisy data and allow the model to make a wrong prediction from the noisy spectrogram, a spectrogram “typo”, but we would fix the prediction based on the context. However, technical solutions to deploy these strategies were not available in the literature until recently. At the same time we identified the above limitations and needs, transformer architectures have emerged as a powerful class of neural network solutions capable of capturing complex long-range correlations and contextual relationships within data. Originally developed for sequence modeling in the language domain, transformers have demonstrated remarkable performance across multiple modalities—including language (LLMs), vision (VTs), and more recently audio—by leveraging their ability to globally correlate data patterns. In particular, vision transformers (e.g., ViT [ 18 ] and Swin [ 33 ] ) successfully model spatial correlations in images and spectrogram representations, while large language models (e.g., GPT-3 [ 9 ] and GPT-4 [ 1 ] ) excel at contextual inference, error detection, and text correction tasks. These advantages naturally position transformers as candidate solutions for deploying our above-mentioned strategies. Despite these advantages, to the best of our knowledge, no research has yet leveraged transformer architectures—either vision or language variants—to handle noisy acoustic keystroke data. This position our work as the first-of-its-kind in making ASCAs practical. Considering the above, we postulate three key scientific hypothesis: (1) The global modeling capability of visual transformers, particularly their unparalleled skill in capturing long-range contextual relationships, might effectively mitigate errors introduced by acoustic noise and improve ASCA reliability, making them the most suitable candidate for solving the problem via the first above-pointed strategy (i) ; and, similarly (2) Large Language Models (LLMs) have significant error correction (e.g., rewritting, passphrase, translation, and so on) capabilities, which makes them the most suitable solutions to solve the problem via the second above-mentioned strategy (ii) ; finally, (3) These solutions can complement each other and be used in tandem to solve the problem at a higher scale. To test our hypothesis in practice, we developed a novel transformer-driven framework for enhancing acoustic side-channel attacks on keyboards and conducted experiments with it on reference

Read the complete text