Comparison of Autoencoders for tokenization of ASL datasets

Vouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad · 2025

Read Comparison of Autoencoders for tokenization of ASL datasets by Vouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06942.txt.

Introduction

Vouk Praun-Petrovic, Aadhvika Koundinya, and Lavanya Prahallad compare ways to compress sign language images into tokens in a 2025 study. Their data set holds tens of thousands of American Sign Language images across a set of hand sign classes. Three autoencoder families compete to encode each image compactly and rebuild it faithfully: plain feedforward, convolutional, and diffusion based.

The diffusion autoencoder wins, with the lowest reconstruction error and the highest objective quality score among the three. Good tokenizers are the foundation for generative models of signing, which could power translation and teaching tools for the Deaf community.

Catalog record

CREATORVouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-06942

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparison of Autoencoders for tokenization of ASL datasets Vouk Praun-Petrovic, Aadhvika Koundinya, Lavanya Prahallad Abstract 1 Introduction Generative AI powered by large language models (LLMs) has revolutionized a wide array of applications, ranging from chatbots and virtual assistants to the generation of sound, audio, and video. With advancements in multimodal AI, LLMs are increasingly capable of handling tasks involving text, audio, images, and video [ 1 ] . Among these, applications such as sign language detection and generation have gained prominence, leveraging LLMs’ ability to process and integrate data across different modalities. These multimodal models employ encoder-decoder mechanisms as essential components to preprocess input data and post-process outputs for various tasks. These mechanisms ensure that raw input data—whether visual, audio, or textual—is converted into a tokenized form that the LLM can process, and then decoded into meaningful output. The quality and precision of encoder-decoder mechanisms play a pivotal role in the success of multimodal applications. An effective encoder-decoder framework not only preserves the semantic and structural information of the input during encoding but also ensures high-fidelity reconstruction during decoding. In multimodal applications, such as sign language generation, inaccuracies in encoding can lead to the loss of critical contextual or spatial information, while errors in decoding can result in outputs that fail to convey the intended message. Thus, designing and evaluating robust encoding and decoding systems is essential to enable multimodal LLMs to perform reliably across diverse tasks. This work focuses specifically on developing encoder-decoder models for an American Sign Language (ASL) image dataset—a challenging yet essential task for enabling effective communication between individuals with hearing impairments and digital systems. The ASL dataset presents a rich set of hand sign images representing 29 distinct classes, including letters, space, delete, and nothing. By addressing this dataset, we aim to create models that can efficiently tokenize image data for LLMs and reconstruct images with high accuracy. Through this work, we contribute to the advancement of encoder-decoder architectures tailored for visual modalities, a critical component in multimodal AI systems [ 8 ] . To evaluate the effectiveness of different approaches, we compare three encoder-decoder techniques: a simple feedforward network, a convolutional network, and a diffusion network. Each technique represents a distinct methodology for encoding and decoding image data [ 10 ] . The feedforward network provides a baseline with fully connected layers operating on flattened image inputs. The convolutional network leverages spatial hierarchies in image data through convolutional and transposed convolutional layers, making it more effective for image reconstruction. The diffusion network introduces probabilistic modeling and iterative noise prediction into the encoding and decoding process, adding robustness to noise and improving generative capabilities [ 7 ] . Our study offers a comprehensive analysis of these techniques through both objective and subjective evaluations. Objective metrics such as reconstruction error and accuracy provide quantifiable insights into the performance of each model, while subjective evaluations assess the perceptual quality of the reconstructed images. By comparing these approaches, this work highlights the strengths and weaknesses of each technique and provides valuable insights for developing robust encoder-decoder systems for multimodal AI applications, particularly in the context of sign language recognition and generation. 2 Data 2.1 Data Description The dataset used in this study is the ASL Alphabet Dataset sourced from Kaggle, comprising 87,000 images across 29 classes of hand signs. These classes represent the letters A-Z, along with signs for “space” and “delete,” as well as a ”nothing” class of blank images. Each class contains 3,000 images, ensuring diverse representations with variations in hand sizes, lighting conditions, and backgrounds. This diversity makes the dataset well-suited for training robust models capable of recognizing hand signs in varying real-world conditions. To facilitate training and evaluation, the dataset was split into an 80/20 ratio, yielding 69,600 images for training and 17,400 for validation. Additionally, a test set containing a single image for each class, excluding ”delete” (28 images total), was created to evaluate the model’s performance on unseen data. 2.2 Data Pre-Processing Prior to feeding the data into the autoencoders, several preprocessing steps were performed to ensure consistency and suitability for training. The images were first resized to a fixed resolution of 200×200 pixels to match the input requirements of the models. Each image was then converted into a tensor, a multidimensional data structure used by PyTorch, and normalized to scale pixel values to the range [0, 1]. This normalization step ensures uniformity across the dataset, preventing issues arising from varying pixel intensity scales. These normalized images were then separated into training and validation datasets, with an 80/20 split. By applying these preprocessing techniques, the dataset was prepared to efficiently train the autoencoders while preserving critical features necessary for the accurate representation and reconstruction of hand signs [ 5 ] . 3 Models 3.1 Feedforward Autoencoder The Feedforward Autoencoder is a fully connected neural network designed to compress high-dimensional input data into a lower-dimensional latent representation and reconstruct it back to its original form. It treats the input as a flattened vector, making it suitable for simpler datasets or tasks that do not require spatial feature extraction. Its straightforward structure makes it a strong baseline for understanding autoencoding techniques. 3.2 Details The Feedforward Autoencoder is a fully connected neural network designed to compress high-dimensional input data into a lower-dimensional latent representation and reconstruct it back to its original form. It treats the input as a flattened vector, making it suitable for simpler datasets or tasks that do not require spatial feature extraction. Its straightforward structure makes it an effective baseline for understanding autoencoding techniques. Figure 1: Visual Depiction of Feedforward Autoencoder architecture 3.3 Convolutional Autoencoder The Convolutional Autoencoder utilizes convolutional layers to encode and decode image data while preserving the spatial structure of the input. It is particularly well-suited for image-related tasks, as convolutional operations hierarchically extract spatial features such as edges, textures, and complex patterns. 3.4 Details The encoder consists of three convolutional layers, each using 4×4 filters with a stride of 2 and padding of 1. These layers progressively reduce the spatial dimensions of the input image from 200×200 to 100×100, 50×50, and finally 25×25, while increasing the number of feature maps from 32 to 64 and 128. After this, the encoder flattens the 25×25×128 tensor and applies a linear layer to reduce its size to 64 dimensions. The decoder reverses this process using another linear layer followed by transposed convolutional layers, restoring the latent representation to the original image size. Each convolutional and transposed convolutional layer employs ReLU activations, except for the final output layer, which uses a sigmoid activation function. This network contains approximately 16 million parameters, making it significantly more efficient than the feedforward autoencoder due to parameter sharing in the convolutional layers. Its ability to capture hierarchical spatial features makes it highly effective for image reconstruction tasks. Figure 2: Visual Depiction of Convolutional Autoencoder architecture 3.5 Diffusion Autoencoder The Diffusion Autoencoder extends the convolutional autoencoder by incorporating a diffusion process, adding probabilistic noise, and performing denoising during the encoding phase. This architecture combines principles of variational autoencoders with iterative noise prediction, making it robust for tasks involving noise or generative data modeling [ 2 , 5 ] . 3.6 Details The encoder retains the structure of

Read the complete text