Kryptonite-N: Machine Learning Strikes Back

Nathan Bailey*, Kira Kim*, Albus Li*, Will Sumerfield* · 2024

Read Kryptonite-N: Machine Learning Strikes Back by Nathan Bailey*, Kira Kim*, Albus Li*, Will Sumerfield* (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20588.txt.

Description

Abstract Quinn et al propose challenge datasets in their work called “Kryptonite-N”. These datasets aim to counter the universal function approximation argument of machine learning, breaking the notation that machine learning can “approximate any continuous function” [ 22 ] .

Catalog record

CREATORNathan Bailey*, Kira Kim*, Albus Li*, Will Sumerfield*
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Kryptonite-N: Machine Learning Strikes Back Nathan Bailey*, Kira Kim*, Albus Li*, Will Sumerfield* Abstract Quinn et al propose challenge datasets in their work called “Kryptonite-N”. These datasets aim to counter the universal function approximation argument of machine learning, breaking the notation that machine learning can “approximate any continuous function” [ 22 ] . Our work refutes this claim and shows that universal function approximations can be applied successfully; the Kryptonite datasets are constructed predictably, allowing logistic regression with sufficient polynomial expansion and L1 regularization to solve for any dimension N. † † affiliationnotice: Equal contribution 1 Introduction Recent advancements in machine learning, namely LLMs like GPT-4, have gained not only popularity from its success in solving various complex problems [ 5 ] but also some scepticism in its capabilities, as suggested by the “Kryptonite-N” paper [ 22 ] . Quinn and Luther state despite applying binary logistic regression with a GPT-driven basis expansion technique, they still fail to achieve high-quality performance on the datasets. In this paper we refute the claims of the “Kryptonite-N” paper by showing that several models yield good results on the dataset, therefore proving the validity of Universal Function Approximation . At first a comprehensive data exploration is performed to inform the experimental design. Based on the experiments, we discuss insights gained from the results followed by an analysis on sustainability of model lifecycle. Furthermore, we demonstrate the flaws in their GPT approaches and explain observed poor performance. Finally, we outline the process where we determined the construction of the datasets . 2 Data Exploration Initial data exploration using quantitative metrics and visual representation is crucial to gain insights for data manipulation and modelling steps. We exclusively examined the training portion of the datasets, to ensure that we didn’t bias our models. After assessing metrics from each dimension, we discovered unusual results: each dimension in each K-N dataset has an approximate mean of 0.5, standard deviation of 0.5, and range of [0-\epsilon,1+\epsilon] . This prompted us to evaluate the data distribution by plotting the Probability Mass Function of each dimension for the K-9 Dataset . We learned several very important things from this graph. Firstly, each dimension appears to come from a multimodal distribution with peaks at at 0 and 1. Each peak within a dimension takes the same shape, with there being three different distributions seen across the dimensions, which we classify as ”Burst-like”, ”Gaussian-like”, and ”Spread-like”. Secondly, we see that there appears to be no correlation between which peak a data-point is sampled from and the label. This made us curious about the correlation between the dimensions and the labels, which we hypothesized to be low based on this graph. A correlation matrix shows our hypothesis is correct - there is close to zero correlation across dimensions. This implies the labels must be based on a non-linear combination of features. We visualised the data via a 2-Dimensional Principal Component Analysis to better understand the data’s relationship with the labels, but that resulted in a circle of data points with little clustering of labels, especially at higher dimensions. However, if we only visualise as slices in 2D , we can extrapolate that the points fall in k^{k} distinct clusters, evenly spaced over k -Dimensional space. This reminded us of the XOR problem, a dataset which Multi-Layer Perceptrons fail to solve without non-linearities. The XOR function has points at each corner, with a non-linear relationship to the labels dependent on each input feature (which are also exactly uncorrelated). It seemed to us that this dataset is similar to the K-N datasets, and therefore that we might use similar methods to solve them, such as polynomial features and basis expansion. This also leads us to the idea of discretizing the dataset, by mapping v<0.5\Rightarrow 0,v>0.5\Rightarrow 1 for each dimension. This method assumes the distributions of each dimension are inherently noisy, which is backed by the fact that the labels seem to fall evenly across the distribution. 3 Methodology We tackle the Kryptonite-N dataset in the same way as the original work; a classification task with a labelled dataset D=\{(x^{(i)},y^{(i)})\}_{i=1}^{N} . We train a parametric model f with parameters \theta . The training function will seek to optimise these parameters with respect to a loss function L and return the model with the parameters producing the smallest loss. 3.1 Data Standardisation Standardisation is a common pre-processing step to place attributes in uniform scale. The mean \mu and standard deviation \sigma of each feature j are computed for the dataset and each value is scaled such that each feature has a mean of 0 and a standard deviation of 1. \hat{x}_{ij}=\frac{x_{ij}-\mu_{j}}{\sigma_{j}} It is especially important for models using linear combinations of weights and inputs, \hat{y}=\theta^{T}x and optimizing using gradient descent. Consider a single layer neural network, \frac{\partial L}{\partial W}=x\frac{\partial L}{\partial z} where z is the output of the layer. Therefore, all partial derivatives are scaled by the input features. If some features are centred around a larger mean, they will cause larger updates to their corresponding weights. Standardising data counters against this better ensures convergence. It also enables relative comparison of polynomial coefficients for polynomial features which will be essential in understanding underlying patterns across feature values. 3.2 Neural Networks Neural Networks are universal function approximations and therefore are a good solution to experiment with to solve the Kryptonite-N dataset. Given an input vector x , a single layer l of a neural network will map this to output using a matrix of weights W_{l} and a vector of biases b_{l} . The number of rows in W corresponds to the number of neurons in the layer, with each neuron connecting to all inputs with a corresponding weight. Mathematically: A_{l}=\sigma_{l}(W_{l}x^{T}+b_{l} ). Where \sigma_{l} is a non-linear activation function of the layer. An activation function is key to introduce non-linearity into the network, allowing it to approximate non-linear functions. Layers of a neural network can be stacked to produce a multi-layer network, defined as: \hat{y}=\sigma_{L}(W_{L}(\sigma_{L-1}(...\sigma_{0}(W_{0}x^{T}+b_{0}))+b_{L}) . Where L is the final layer in the network. In a multi-layer neural network, there are 3 hyper-parameters to optimize for: the number of layers, the number of neurons in each layer and the activation function used in each layer. 3.2.1 Mini-Batch Gradient Descent For a given weight in a neural network, the update rule is given by w\leftarrow w-\alpha\nabla_{w}L . Stochastic gradient descent will compute the loss and the weight for every sample in the dataset. This can introduce noise into the weight updates, making the training process unstable as every derivative can point in different directions. Batched gradient descent, on the other hand, computes a loss value after processing every sample in the dataset, making one weight update for every pass of the dataset. This has the disadvantage of slow convergence to optimal, as it needs a full pass of the data for a single update. Mini-batch gradient descent represents a middle ground, where the loss and derivative are calculated over a subset of the samples in the dataset. This helps to reduce the noise from stochastic gradient descent, whilst speeding up convergence compared to batched gradient descent. The batch size hyperparameter B can be altered to adjust the trade-off between convergence

Read the complete text