Architectural Trade-offs in Small Language Models Under Compute Constraints

Shivraj Singh Bhatti - [email protected] · 2025

Read Architectural Trade-offs in Small Language Models Under Compute Constraints by Shivraj Singh Bhatti - [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20877.txt.

Introduction

Shivraj Singh Bhatti measures what each design choice buys a small language model when computing power is tight. Starting from a bare linear next-token predictor, he adds one ingredient at a time: nonlinearity, self-attention, then full multi-layer transformers. Each variant trains on Tiny Shakespeare at character level, plus two word-level datasets.

Scoring weighs accuracy against parameter count and training cost. Attention models beat plain feed-forward networks in accuracy per unit of compute, even at tiny sizes. Extra depth or context length hurt when the training budget could not support them. One caution stands out: rotary position embeddings, a hit in large models, did not reliably transfer to small ones.

Catalog record

CREATORShivraj Singh Bhatti - [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Architectural Trade-offs in Small Language Models Under Compute Constraints Shivraj Singh Bhatti - [email protected] Abstract We present a systematic empirical study of small language models under strict compute constraints, analyzing how architectural choices and training budget interact to determine performance. Starting from a linear next-token predictor, we progressively introduce nonlinearities, self-attention, and multi-layer transformer architectures, evaluating each on character-level modeling of Tiny Shakespeare and word-level modeling of Penn Treebank (PTB) and WikiText-2. We compare models using test negative log-likelihood (NLL), parameter count, and approximate training FLOPs to characterize accuracy–efficiency trade-offs. Our results show that attention-based models dominate MLPs in per-FLOP efficiency even at small scale, while increasing depth or context without sufficient optimization can degrade performance. We further examine rotary positional embeddings (RoPE), finding that architectural techniques successful in large language models do not necessarily transfer to small-model regimes. Code and experiment logs: . 1 Introduction This work studies how architectural choices and compute budget affect the quality of small language models. We begin with a simple linear next-token predictor and progressively introduce nonlinearities, self-attention, and multi-layer transformer blocks [ 1 ] . All models are trained on Tiny Shakespeare (character-level) and evaluated using negative log-likelihood (NLL) on held-out test data. We then select the best-performing architecture and train word-level models on PTB [ 4 ] and WikiText-2 [ 3 ] . Throughout, we compare models in terms of test NLL, parameter count, and an approximate measure of training FLOPs to understand compute–performance trade-offs in a constrained regime. 2 Experimental Setup 2.1 Datasets and Tokenization For the Tiny Shakespeare experiments we use the provided train/validation/test split (about 1 MB of text in total). Tokenization is character-level: we build a vocabulary of all distinct characters across the three splits and map each character to an integer ID. PTB and WikiText-2 are treated as word-level datasets with whitespace tokenization and a fixed vocabulary built from the training split only. Given a context length T (“block size”), training examples for Tiny Shakespeare are constructed by sliding a window of T characters over the corpus. Each example consists of an input sequence x_{1:T} and target character x_{T+1} ; models predict the distribution of x_{T+1} given x_{1:T} . 2.2 Model Architectures All Tiny Shakespeare models share the same input representation: characters are embedded into a d_{\text{model}}=128 dimensional space via a learnable embedding matrix. Linear model. The linear baseline flattens the embedded context into a vector of dimension Td_{\text{model}} and applies a single linear layer to predict logits over the vocabulary. This is multinomial logistic regression over concatenated embeddings. Multi-layer perceptron (MLP). The MLP model flattens the embedded context and passes it through two hidden layers with ReLU activations and dropout, followed by a linear output layer. Hidden dimensions \{128,256,512\} are explored; the main configuration uses hidden size 256. Self-attention model. The self-attention model adds learned positional embeddings to token embeddings and feeds the sequence into a single transformer-style block: causal multi-head self-attention (upper-triangular mask), a position-wise feedforward network, and layer norms with residual connections [ 1 ] . The final time step is projected to vocabulary logits. We sweep heads \{1,2,4\} ; the main configuration uses 4 heads. Multi-layer transformer. The transformer stacks L blocks with embedding size 128 and feed-forward width 256. We sweep L\in\{2,3,4\} ; L=3 is used as the primary model. 2.3 Training Procedure All models are trained with Adam (learning rate 3\times 10^{-4} , default \beta ) and batch size 64. For Tiny Shakespeare we cap training positions per epoch at 50k for speed and use 10k positions each for validation and test. Epoch counts are intentionally small because Tiny Shakespeare converges quickly in this regime. We train linear and MLP models for 3 epochs, and the attention and transformer models for 4 epochs. Validation loss is monitored each epoch and the best checkpoint is retained for testing and generation. To compare models under different budgets, training FLOPs are approximated as \text{FLOPs}\approx 2\times(\#\text{parameters})\times(\#\text{training tokens}), which roughly corresponds to one forward and one backward pass per parameter per token. While coarse, this estimate is sufficient for relative comparisons across model families. 3 Character-Level Modeling: Tiny Shakespeare 3.1 Learning Curves Figure 1 shows training and validation NLL across epochs for the four architectures. Figure 1 : Training and validation NLL for Tiny Shakespeare across architectures. More expressive models achieve lower loss while still avoiding major overfitting in this setting. The linear model reaches a test NLL of about 3.05. Adding nonlinear hidden layers in the MLP reduces test NLL to 2.32, and adding self-attention further improves it to 2.13. The multi-layer transformer achieves the best performance at roughly 2.01 NLL. 3.2 Hyperparameter Sweeps We examine performance as a function of one design dimension per family, training each configuration for two epochs on a fixed subset for fast comparison. Linear: context length. Figure 2 plots test NLL as a function of context length T\in\{32,64,128\} . The linear model performs best with the shortest context and degrades as T grows, consistent with increasing parameter count without increasing modeling power. Figure 2 : Linear model: test NLL vs. context length. Longer contexts increase dimensionality and parameters, but the model remains linear and appears under-trained at fixed optimization budget. MLP: hidden dimension. We vary hidden width h\in\{128,256,512\} (Figure 3 ). Test NLL decreases with h , but gains diminish from 256 to 512, suggesting 256 is an efficient choice. Figure 3 : MLP: test NLL vs. hidden dimension. Wider layers help, but marginal gains shrink as parameter count grows. Self-attention: number of heads. We vary heads H\in\{1,2,4\} (Figure 4 ). With fixed embedding size, 4 heads performs best in this regime. Figure 4 : Self-attention: test NLL vs. number of heads. Four heads yields the best performance, consistent with learning multiple attention subspaces. Transformer: number of layers. We sweep depth L\in\{2,3,4\} (Figure 5 ). The 3-layer model slightly outperforms shallower and deeper variants, suggesting a good capacity–compute balance. Figure 5 : Transformer: test NLL vs. layers. Depth L=3 performs best here; L=4 likely suffers from insufficient optimization steps at fixed budget. 3.3 Compute vs. Performance Table 1 summarizes Tiny Shakespeare configurations. Figure 6 visualizes test NLL against approximate training FLOPs (log scale). Table 1 : Tiny Shakespeare models: parameter count, approximate training FLOPs, and test NLL. Model Params Train FLOPs Test NLL Linear 1,073,345 4.1\times 10^{13} 3.05 MLP 4,285,377 1.6\times 10^{14} 2.32 Self-Attn 231,617 1.2\times 10^{13} 2.13 Transformer 430,785 2.2\times 10^{13} 2.01 Figure 6 : Test NLL vs. approximate training FLOPs (log scale). Attention-based models are markedly more efficient per FLOP than the MLP in this setup. Two patterns stand out: (i) nonlinear models dramatically outperform the linear baseline, and (ii) self-attention/transformers provide better NLL per FLOP than the MLP. 3.4 Qualitative Samples Each trained model generates a 100-character continuation of the prompt HAMLET: . Sample quality correlates with test NLL: the linear model produces scrambled characters, the MLP begins to capture word-like structure, and attention-based models produce clearer formatting (speaker tags, line breaks), with the transformer being most coherent. 4 Word-Level Modeling on PTB and WikiText-2 We reuse the 3-layer Tiny Shakespeare transformer (128-dimensional embeddings, 4 heads, feed-forward width 256) for word-level modeling on PTB and WikiText-2. 4.1 Datasets and Tokenization We switch to word-level tokenization.

Read the complete text