Detecting Avalanche Effect in Adversarial Settings: Spotting the Encryption Loops in Ransomware

Nanqing Luo \affilnum 1, Xusheng Li \affilnum 2, Haizhou Wang \affilnum 1, Shuangyi Zhu \affilnum 3, Yuan Ma \affilnum 4, Peng Liu \affilnum 1 · 2026

Read Detecting Avalanche Effect in Adversarial Settings: Spotting the Encryption Loops in Ransomware by Nanqing Luo \affilnum 1, Xusheng Li \affilnum 2, Haizhou Wang \affilnum 1, Shuangyi Zhu \affilnum 3, Yuan Ma \affilnum 4, Peng Liu \affilnum 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24131.txt.

Introduction

Luo and five colleagues spot encryption loops in binary-only ransomware by checking the avalanche effect itself. Earlier work only checked a ripple effect, a necessary but insufficient sign, which attackers can fake. Their record-and-replay method tolerates sloppy input and output identification and resists evasion, backed by Shapiro-Wilk normality guarantees. Tests show no missed cases and 1.1 percent false alarms, and the tool analyzes every sample drawn from ten representative ransomware families.

Catalog record

CREATORNanqing Luo \affilnum 1, Xusheng Li \affilnum 2, Haizhou Wang \affilnum 1, Shuangyi Zhu \affilnum 3, Yuan Ma \affilnum 4, Peng Liu \affilnum 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67384
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24131

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting Avalanche Effect in Adversarial Settings: Spotting the Encryption Loops in Ransomware Nanqing Luo \affilnum 1, Xusheng Li \affilnum 2, Haizhou Wang \affilnum 1, Shuangyi Zhu \affilnum 3, Yuan Ma \affilnum 4, Peng Liu \affilnum 1 Abstract Abstract Spotting encryption loops in binary-only ransomware is a critical reverse engineering task. Since the existence of avalanche effect, an intrinsic characteristic of any secure encryption algorithms, is unavoidable during a victim data encryption attack, it is a very promising direction to spot encryption loops through avalanche effect detection. Unfortunately, no existing work in this direction ensures that the being-checked effect is the avalanche effect itself . Although CipherXRay 19 is inspired by avalanche effect, it only checks whether a “ripple effect” ( i . e ., a necessary but non-sufficient condition) of avalanche effect exists, allowing a straightforward counterattack to succeed. In this work, we present a new approach that checks the avalanche effect itself. Because the detection is conducted in adversarial settings ( e . g ., the ransomware author may obfuscate the code), a viable approach must tolerate inaccurate input & output identification and must be resilient to adversarial evasion. These challenges are addressed by a novel record-and-replay detection mechanism that takes advantages of the statistical guarantees provided by the Shapiro–Wilk normality test. The experiment results show that our approach achieves achieves 0.0% false negative rate and 1.1% false positive rate. When our tool is employed to reverse engineer real-world ransomware samples, it succeeds in analyzing all the ransomware samples selected from ten representative families. 1 Introduction Ransomware attack is a massive cyberattack that encrypts victims’ data for ransom. WannaCry 8 in May 2017 is the first high-profile ransomware attack. WannaCry leveraged “EternalBlue”, a highly effective exploit against Server Message Block (SMB), a protocol that allows machines to share files over a Windows network. Within 30 days, this worldwide attack had compromised over 200,000 systems. In recent years, ransomware attacks have become an increasingly more serious societal threat due to not only the increasing amount of security loss caused by real-world enterprise networks, but also the increasing amount of disruptions caused to critical infrastructures. For example, the Colonial Pipeline ransomware attack 3 in May 2021 has brought one of the largest and most vital oil pipelines in the United States to a shutdown for one week. (The pipeline comprises more than 5,500 miles of pipes.) Figure 1: Encrypted file structure analysis through loop body of encryption process in TeslaCrypt . Being a critical aspect of malware analysis, reverse engineering plays an indispensable role in defending against ransomware attacks. Besides such standard malware reverse engineering tasks as disassembly 22 , type inference 6 , function recognition 14 , memory dependence analysis, and data/control dependence analysis 2 , one unique task in ransomware reverse engineering is to identify and understand the program code used to encrypt victims’ data. The importance of this unique task is illustrated in Figure 1 , where a simplified encryption loop body of the TeslaCrypt ransomware 27 is shown on the left, and the (simplified) structure of a file encrypted by TeslaCrypt is on the right. What is important here are the connections between the left side and the right side, represented by the directed edges. For example, the first line of code on the left can guide the analyst to pinpoint the inst_id field in the encrypted file: this field tells the analyst the family/type of ransomware. The second line on the left can guide the analyst to pinpoint the GLOB_PUB field: this field tells the analyst the public key for asymmetric encryption ( e . g ., RSA). The third line on the left can guide the analyst to pinpoint the encrypted AES key in the encrypted file. Although this unique reverse engineering task is crucial, fulfilling this task in the real world involves arduous manual labor due to several reasons, including but not limited to the following ones: (Gap A) The existing code-similarity-based methods ( e . g ., 7 ; 31 ) detect the encryption loop with high false positives. (Gap B) Existing pattern-based ( i . e ., signature/heuristic-based) methods ( e . g . 21 ; 17 ) are not scalable when handling newly emerged encryption algorithms. (Gap C) Existing methods inspired by the avalanche effect 19 suffer from adversarial evasion which will be described in Section 3.1 Data flow based counterattack against CipherXRay . This work seeks to develop an analysis method that can automatically spot the encryption loops in ransomware. Although this work is not the first avalanche-effect-inspired attempt to spot encryption loops, it is the first work that aims to achieve the needed resilience against adversarial evasion ( i . e ., the ransomware author can adversarially obfuscate the code in a variety of ways) than the existing approaches. Besides achieving resilience against adversarial evasion, our method aims to achieve high accuracy, thus addressing the primary limitation of code-similarity-based methods, and aims to be scalable, thus addressing the primary limitation of pattern-based methods. Our method is based on two key observations: (1) The limitations identified in Gap A and Gap B can be effectively addressed through the application of avalanche effect detection, and (2) Gap C can be addressed by integrating statistical testing with snapshot-based replay mechanisms. Because the avalanche effect is an intrinsic characteristic of any secure encryption algorithms, the existence of avalanche effect is unavoidable during a victim data encryption attack. Informally, avalanche effect means that “each output bit is flipped with a probability of 0.50 when one bit in input is flipped”. In principle, if an attacker is using secure ciphers, the avalanche effect cannot be avoided even if the attacker adopted unknown ciphers or code obfuscation. Unfortunately, there are several daunting challenges when one intends to check whether the avalanche effect exists in adversarial settings. By “adversarial settings”, we mean that the ransomware author is aware of the detection mechanism and strongly motivated to adversarially change the program code and/or behavior in such a way that confuses ( e . g ., triggers lots of false positives) the detection mechanism. First, in adversarial settings the input and output data (i.e., bits and bytes) of an encryption loop cannot be guaranteed to be identified at 100% accuracy, and therefore the avalanche effect detection mechanism must tolerate inaccurately identified input & output. Second, in order to robustly detect avalanche effect, the phenomenon of “flipped with a probability of 0.50” needs to be confirmed with statistical guarantees. That means we will need to reproduce this phenomenon through running the encryption loop multiple times in order to reach a statistical significant conclusion. However, no real-world malware analysis procedure takes this (statistical test) requirement into consideration. Third, since we are detecting avalanche effect in adversarial settings, we must achieve resilience against adversarial evasion. Instead of directly addressing these challenges, CipherXRay 19 , the only existing work in adversarial settings, takes the strategy of checking whether certain “ripple effect” ( i . e ., a necessary but non-sufficient condition) of the avalanche effect exists. Specifically, it checks the following ripple effect: if the avalanche effect exists, then every bit in the ciphertext should have been tainted from every bit in the

Read the complete text