Gibbs randomness-compression proposition: An efficient deep learning

M. Süzen · 2025

Read Gibbs randomness-compression proposition: An efficient deep learning by M. Süzen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23869.txt.

Introduction

Süzen proposes a link between randomness and compression in trained networks. The 2025 paper builds the link on Gibbs entropy over the weights left after a lossy squeeze. Task scores serve as the probe: as a model is squeezed and retrained in cycles, its score tracks how much was lost.

Three squeezes were tested on a standard vision task, from random pruning to a new dual method based on compressed sensing. Within a small drop in compression, score and entropy moved together in step. The author reads this as a computable tie between randomness and squeezing. The framing borrows from the physics of coarse-graining in statistical physics, where detail is shed in stages.

Catalog record

CREATORM. Süzen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gibbs randomness-compression proposition: An efficient deep learning M. Süzen Abstract A proposition that connects randomness and compression put forward via Gibbs entropy over set of measurement vectors associated with a compression process. The proposition states that a lossy compression process is equivalent to directed randomness that preserves information content. The proposition originated from the observed behaviour in newly proposed Dual Tomographic Compression (DTC) compress-train framework. This is akin to tomographic reconstruction of layer weight matrices via building compressed sensed projections, so called weight rays . This tomographic approach is applied to previous and next layers in a dual fashion, that triggers neuronal-level pruning. This novel model compress-train scheme appear in iterative fashion and act as smart neural architecture search, Experiments demonstrated utility of this dual-tomography producing state-of-the-art performance with efficient compression during training, accelerating and supporting lottery ticket hypothesis. However, random compress-train iterations having similar performance demonstrated the connection between randomness and compression from statistical physics perspective, we formulated so called Gibbs randomness-compression proposition , signifying randomness-compression relationship via Gibbs entropy. Practically, DTC framework provides a promising approach for massively energy and resource efficient deep learning training approach. I Introduction Connections between randomness and data compression is noted earlier within the algorithmic information framework of Solomonoff-Kolmogorov-Chaitin Solomonoff 1964 ; Kolmogorov 1965 ; Chaitin 1966 (SKC). In summary these works are aiming at reaching a definition of randomness from model selection perspective Chaitin 1975 , i.e., algorithmic entropy or a complexity measure. Striking connection to algorithmic entropy and Gibbs entropy is also explored in from fundamental conceptual point of view Bennett 1982 ; Zurek 1989 ; Machta 1999 . Statistical mechanics connections and information established between Gibbs and Boltzmann entropies Jaynes 1965 ; Jaynes 1957 ; Buonsante et al. 2016 ; Rondoni and Cohen 2000 . However, relationship among compression process, entropy, information and randomness is always made quite generic in SKC framework. This manifestation was by design quite generic without strict mathematical conditition , i.e., finding the shortest algorithm or explanation that can generate the random sequence. This is similar in entropy production studies Süzen 2025 . In this work, we proposed a mathematical conditions among these concepts in a more formal manner inspired from newly proposed deep learning compress-train framework. We describe a technique called inverse compressed-sensing in dual fashion, Dual Tomographic Compression (DTC) compress-train for deep learning in an iterative manner as an example of Gibbs randomness-compression proposition . Such interplay between physics and deep machine learning, now it is well-established that, constitute physics research Hopfield 1982 ; Ackley et al. 1985 ; Süzen 2022 ; Süzen 2023 ; Süzen 2024 . II Background Our purpose is establishing a mathematical relationship between randomness and complexity , whereby complexity appears as a compression process. In this direction, we leverage couple of distinct research areas, randomness and complexity, compressed sensing and neural network architecture compression. The definition of randomness Compagner 1991 and quality of generating random numbers Melissa E. O’Neill 2014 appears to be connected subjects within the complexity, compressing data Zurek 1989 , empirically Chang et al. 2010 and SKC algorithmic randomness context. This framework commonly known as Kolmogorov Complexity (KC), algorithmic complexity or algorithmic randomness and its applications are vast Li and Vitányi 2008 ; Zurek 1991 . KC or SKC framework, makes a generic statement about information content of sequence s of binary digits of length N , K(s) , defined to be the size of the smallest program in bits that generates s , the sequence. The connection to compression is implicit here and its mathematical definition to compression process appears as only a single compression step. It is hinted that there is a positive correlation between compression ratio and randomness Li and Vitányi 2008 . Moreover, the interest of randomness in this manner also explored from biological evolution stand point Wagner 2012 , which is more relevant in our case due to proposed train-compress framework having mutation-like process. Compressed sensing (CS) framework shown huge leap in statistical signal processing and information theory as it shows one can reconstruct an unknown signal with much less measurements under sparsity constraints Donoho 2006 ; Candès et al. 2006 ; Candes and Tao 2006 ; Baraniuk 2007 ; Eldar and Kutyniok 2012 . CS involves in random sampling and it provides an excellent tool to investigate the relationship between compression and randomness. Reducing size of neural networks has attracted research interest since their inception, such as earlier works on Hopfiled networks Janowsky 1989 in statistical physics community. This pioneering work identify clipping as removing existing connections and pruning re-learning, i.e., fine-tuning after clipping. Similar clipping approach first applied in computer science by LeCun et. al LeCun et al. 1989 , fun title of brain damage is used, reflecting the performance degradation due to clipping, hinting at magnitude pruning, so called small saliency that small weights doesn’t effect the performance that much. With the advent of deep learning LeCun et al. 2015 ; Schmidhuber 2015 and recently generative models producing ever increasing size large deep learning architectures due to attention Vaswani et al. 2017 and diffusion Sohl-Dickstein et al. 2015 mechanisms, necessitates producing much smaller deep learning models for both training and inference efficiency, as energy consumption and ethical use is raised as the most important aspect in industry and academic practice Menghani 2023 . More recently, lottery ticket hypothesis Frankle and Carbin 2018 , provides a deeper perspective in compressing or finding smaller deep learning architectures, via magnitude pruning, i.e., brain damage, stating that subnetworks are possible that gives similar performance as the initial deep architecture due to initialization chance, so the name lottery ticket. Applying the lottery ticket approach, iteratively pruning during training, leads to train-compress procedure, a compression process, which is the main example tool for our quest for finding relationship between randomness and compression. Such process of train-compress as a meta algorithm has been already proposed Han et al. 2015 ; Carreira-Perpinán and Idelbayev 2018 , before lottery tickets proposed. Effectiveness of random only pruning also reported recently Gadhikar et al. 2023 . The contributions of the paper is presented in the following sections Inverse Compressed Sensing , Dual Tomographic Compression and the main proposition, Gibbs randomness-compression proposition . These are supported by in-depth state-of-the-art (SoTA) numerical experiments on canonical vision deep learning test system. Figure 1: DTC accuracy over different sparsity levels in train-compress iterative pruning. Figure 2: DTC accuracy over different sparsity levels in train-compress iterative pruning. Close up to higher sparsity. III Inverse Compressed Sensing We apply compressed sensing on the learned weights. However, as weights are known, instead of measuring the signal (weights), we do have it. This implies so called inverse compressed sensing , not to be confused with inverse problem nature of the original compressed sensing framework. Inverse here implies, we construct an hypothetical measurement process from known signal, and try to reconstruct the signals sparse version via conventional compressed sensing, i.e.,so called weight rays . In case of deep learning layers, total weights associated to neuron’s input and output will be used instead of single CS, dual nature of tomographic process. Essentially, we found a sparse projection of the learned weights, which might be sparser due to compressed sensing

Read the complete text