Revisiting the Shape Convention of Transformer Language Models Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi, Da-shan Shiu Abstract Dense Transformer language models have largely adhered to one consistent architectural shape: each layer consists of an attention module followed by a feed-forward network (FFN) with a narrow–wide–narrow MLP, allocating most parameters to the MLP at expansion ratios between 2 and 4. Motivated by recent results that residual wide–narrow–wide (hourglass) MLPs offer superior function approximation capabilities, we revisit the long-standing MLP shape convention in Transformer, challenging the necessity of the narrow–wide–narrow design. To study this, we develop a Transformer variant that replaces the conventional FFN with a deeper hourglass-shaped FFN, comprising a stack of hourglass sub-MLPs connected by residual pathways. We posit that a deeper but lighter hourglass FFN can serve as a competitive alternative to the conventional FFN, and that parameters saved by using a lighter hourglass FFN can be more effectively utilized, such as by enlarging model hidden dimensions under fixed budgets. We confirm these through empirical validations across model scales: hourglass FFNs outperform conventional FFNs up to 400M and achieve comparable performance at larger scales to 1B parameters; hourglass FFN variants with reduced FFN and increased attention parameters show consistent improvements over conventional configurations at matched budgets. Together, these findings shed new light on recent work and prompt a rethinking of the narrow-wide-narrow MLP convention and the balance between attention and FFN towards efficient and expressive modern language models. † † affiliationnotice: Equal contribution Figure 1 : Performance frontiers of Transformers with hourglass (wide-narrow-wide) versus conventional (narrow-wide-narrow) FFNs ( 37 ) . We revisit the shape convention of Transformer by replacing the narrow-wide-narrow FFN with a hourglass FFN, composing stacks of wide-narrow-wide sub-MLPs connected by residuals. We observe that Hourglass FFNs achieve comparable performance to the conventional design up to 1B parameters. Here we also show a conventional variant trained based on OLMo-2 architecture. Only the non-embedding parameters are accounted for the FLOPs. 1 Introduction Despite rapid advances in scale and training methodology, the architectural shape of dense Transformer language models has remained remarkably stable since early scaling studies ( 16 ) . Modern models consistently adopt a narrow–wide–narrow multilayer perceptron (MLP) in feed-forward network (FFN), expanding the model dimension d_{\text{model}} to an intermediate width d_{h} before projecting back, with expansion ratios typically fixed between 2 and 4 ( 38 ; 16 ) . This design choice has become a de facto standard across contemporary dense LLMs ( 37 ; 26 ; 36 ; 25 ) . At the same time, the FFN dominates parameter allocation in Transformers, accounting for a majority of model parameters relative to attention. As a result, the MLP shape implicitly determines how capacity is distributed between depth, width, and attention. Figure 2 : Overview: revisiting the shape convention of Transformer through studying the relaxation of MLP shape in FFN. Inspired by ( 21 ; 3 ) , we compare transformer architectural variants with conventional FFN and hourglass FFN. (a) Conventional Transformer Block with L^{\prime} layers, consisting of an attention module and a conventional FFN with a narrow-wide-narrow MLP. (b) Hourglass Transformer Block with L layers, consisting of an attention module followed by an hourglass FFN with K hourglass-shaped MLP sub-blocks. We explore the design space by tuning parameters such as d_{\text{model}} , d_{h} , K , and L , allowing the Hourglass layer count L to differ from the baseline L^{\prime} . While the shape convention in Transformer is effective, we raise a fundamental question: is the narrow–wide–narrow MLP shape in FFN essential, or merely one convenient instantiation of a residual-compatible transformation? Recent results suggest that this convention may be unnecessarily restrictive. Theoretical studies have shown that residual MLPs with wide–narrow–wide (hourglass) structures can act as optimal function approximators, including universality results for networks with extremely narrow intermediate layers ( 20 ; 21 ) . From an expressivity perspective, such hourglass architectures permit deeper residual compositions at fixed parameter budgets, a setting known to favor rapid growth in linear regions and representational complexity ( 8 ; 15 ) . Empirical evidence has also begun to challenge conventional MLP shape assumptions. Motivated by high-dimensional representation learning, 3 demonstrates that hourglass-shaped MLPs outperform standard designs in generative vision models. Related work further shows that expert-routed FFNs with hourglass MLPs in Mixture-of-Experts Transformers can surpass dense baselines ( 39 ) at scale. Together, these results indicate that residual hourglass MLPs constitute a strong class of building blocks, motivating a re-examination of FFN shape conventions in Transformer language models. In this work, we revisit the shape convention of Transformer and study the design space by substituting the conventional narrow-wide-narrow FFN with hourglass FFN variant, consisting of a stacked wide–narrow–wide MLPs with residuals. Such variant offers several advantages. Embedding representations into higher-dimensional spaces before and after narrow transformations increases flexibility for linear separability, consistent with classical results such as Cover’s theorem and random feature methods for kernel approximation ( 5 ; 27 ) . Moreover, hourglass MLPs allow parameters to be allocated more efficiently across depth and width, a principle that has been successfully exploited in convolutional networks ( 42 ; 44 ) and other deep architectures ( 30 ; 10 ) . We therefore hypothesize that FFN with wide-narrow-wide MLPs can outperform the conventional counterpart in Transformer language models . We experimentally validate the hypothesis by comparing hourglass FFN transformers with conventional FFN baselines at various parameter sizes between 113M and 1B. We provide extensive architecture search results to further isolate the contributions of model width, block layer depth, and hourglass FFN structure. To summarize, our contributions are as follows: • We challenge the standard ”narrow-wide-narrow” FFN shape in conventional Transformer, proposing an efficient ”wide-narrow-wide” Hourglass FFN that maintains expressivity with hourglass sub-MLPs connected by residuals. • We demonstrate a parameter reallocation strategy where redirecting parameters from FFN width to attention width or internal depth ( K>1 ) yields superior performance at 113M. • We provide architectural insights by identifying a U-shaped width-depth trade-off, finding optimal d_{h}/d_{\text{model}} and d_{\text{model}}/L ratios and robust deeper Hourglass FFN structures. • Based on the architectural insights at 113M, we validate the scalability of the Hourglass FFN from 113M to 1B, demonstrating it is a competitive and efficient alternative to standard baselines. 2 Background and Related Works 2.1 Narrow-wide-narrow MLP in Transformer FFN In Transformer architectures, the FFN serves as a position-wise non-linear transformation applied independently to each token representation. Formally, given a hidden state z\in\mathbb{R}^{d_{\text{model}}} , the conventional MLP with residual in FFN computes \text{FFN}(z)=z+W_{2}\sigma(W_{1}\text{norm}(z)) (1) where \sigma(\cdot) is a non-linear activation function, \text{norm}(\cdot) is a normalization function, W_{1}\in\mathbb{R}^{d_{h}\times d_{\text{model}}} , W_{2}\in\mathbb{R}^{d_{\text{model}}\times d_{h}} , and d_{h}>d_{\text{model}} . This expansion–projection structure has been retained across most modern dense LLMs ( 37 ; 26 ; 36 ) . Beyond functional form, the FFN plays a central role in determining the overall shape of a Transformer. It accounts for a majority of parameters and floating-point operations per layer, often exceeding those of the attention module. As a result, the choice of the shape expansion ratio d_{h}/d_{\text{model}} , together with the model dimension d_{\text{model}} and the number of block layers L , implicitly governs how capacity is allocated across width, depth, and attention. While prior work has explored variations in activation functions ( 31 ) , normalization strategies ( 41 ; 12 ) , and parameter sharing ( 18 ; 19 ) , the overall narrow–wide–narrow shape of the FFN and its interaction with Transformer scaling ( 16 ) have remained largely unchanged. 2.2 Revisiting Shape Through Hourglass MLPs Theoretical analyses of wide-narrow-wide (hourglass) 1 1 1 Prior works often use the term - bottleneck - to describe the wide-narrow-wide shape while we use hourglass here interchangeably. MLP with residual connections have shown its universal approximation capability of continuous function. When sufficient network depth is allowed, ( 20 ) establishes that residual networks with extremely narrow hidden layers retain universal approximation capability while ( 21 ) further shows that residual hourglass MLP with a narrow constant width can even achieve optimal approximation of Lebesgue-integrable functions. Empirically, while we focus on the FFN of transformer, several work focusing on efficient architecture has explored employing hourglass architecture with residual connections. The computer vision literature on networks such as U-Net ( 29 ) , MobileNet ( 44 ) , and Wide ResNets ( 42 ) utilize bottleneck structures and separable convolutions in depth to optimize the parameter-compute ratio. These works illustrate that carefully structured bottlenecks can reduce computational cost while preserving, or even enhancing, representational power through increased depth or improved gradient flow. In the context of language modeling, low-rank and bottleneck-inspired techniques have primarily appeared in the form of parameter-efficient methods. Approaches such as LoRA ( 10 ; 22 ) introduce low-rank projections within linear layers to reduce the number of trainable parameters during fine-tuning, effectively imposing a bottleneck on weight updates. Mixture-of-Experts (MoE) ( 30 ; 24 ) scales capacity without a proportional increase in inference cost by replacing the dense FFN with multiple sparse expert networks. Moreover, historically, the hourglass MLP shape in dense Transformer FFN is shown to be worse than the narrow-wide-narrow counterpart in the earlier work of scaling law by ( 16 ) . However, recent work in general MLP study on generative vision tasks paints it differently ( 3 ) , where residual-connected hourglass MLPs, which iteratively refine representations in expanded feature spaces, outperform conventional MLPs. While prior research focused on general MLP properties, our work studies hourglass MLPs in FFN as an alternative for the conventional FFN in Transformer, enabling a reconsidering of model shape and parameter alllocation between attention and FFN. We validated such hourglass FFN’s effectiveness as a parameter-efficient alternative to the de facto narrow-wide-narrow standard across language model scales. 3 Transformer with Hourglass FFN To relax and study the shape constraints of conventional Transformer, we propose a Transformer variant that replaces the conventional wide-expansion MLP with a narrow-contracted MLP in FFN (Hourglass FFN) as illustrated in Figure 2 . Based on theoretical and empirical background discussed in Section 2 , we posit that Hourglass FFN can achieve comparable performance to conventional FFN under the same parameter budget. 3.1 Network Architecture The architecture follows the conventional LLaMA style Transformer backbone ( 37 ) consisting of input embeddings, stacked model layers L , and a final output projection. Hourglass Transformer Layer Each layer consists of an attention module followed by an Hourglass FFN. The attention module performs global information aggregation across the sequence, while the Hourglass FFN performs local feature refinement within each token. Both components are wrapped with residual connections and layer normalization. Specifically, given the input to the l -th layer \mathbf{z}^{(\ell)} , the intermediate representation \mathbf{u}^{(\ell)} is computed via the attention mechanism (Attn): \displaystyle\mathbf{u}^{(\ell)} \displaystyle=\mathbf{z}^{(\ell)}+\text{Attn}(\text{norm}(\mathbf{z}^{(\ell)})) (2) Then, we set \mathbf{h}_{0}^{(\ell)}=\mathbf{u}^{(\ell)} as the input to the Hourglass FFN. Hourglass Feed-Forward Network The Hourglass FFN refines the representation through K stacked hourglass-shaped MLP sub-blocks. For i=0,\dots,K-1 : \displaystyle\mathbf{h}_{i+1}^{(\ell)} \displaystyle=\mathbf{h}_{i}^{(\ell)}+\text{MLP}_{i}(\mathbf{h}_{i}^{(\ell)}) (3) where \text{MLP}_{i} denotes the i -th hourglass sub-block. Finally, the output of the layer is \mathbf{z}^{(\ell+1)}=\mathbf{h}_{K}^{(\ell)} . While the convention expands the hidden dimension d_{\text{model}} to a wider d_{h} , the Hourglass FFN utilizes a compression-expansion structure with a bottleneck dimension d_{h}1 ), which appears to offer a better representational balance than width alone. 4.2.3 Can the parameters be redistributed more to the attention module? We first compare (K,L)\in\{(2,12),(2,6),(4,12),(4,6)\} to study the interaction between FFN depth K and Transformer layer count L , as well as the distribution of parameters between attention and FFN modules. As shown in Table 2 , all configurations are matched to a total model size of approximately 113M parameters, yet achieve very similar performance: validation perplexity differences range from 0.05 to 1.34, and validation losses differ by at most 0.046 across settings. Surprisingly, the best result is obtained with (K=4,L=6) , yielding a validation perplexity of 35.101, closely followed by (K=2,L=12) at 35.346. Crucially, this architecture inverts the conventional parameter allocation: the attention module becomes the dominant consumer of parameters, surpassing the FFN (e.g., 66M vs 47M in the K=2,L=12 setting). This shift is significant because it suggests that for small-scale models, contextual processing (mediated by attention) is a more valuable resource than static factual retrieval (mediated by FFNs). The hourglass FFN allows us to ”buy back” parameters from the FFN and reinvest them into a wider, more capable attention mechanism. 4.2.4 How does hourglass FFN change the Transformer width and depth landscape? Figure 4 : Validation loss versus d_{\text{model}}/L ratio for different Hourglass FFN configurations at 113M parameters. The validation loss is minimized when the ratio d_{\text{model}}/L is around 110 for K=4 ; around 180 for K=2 ; around 144 for K=1 . Encouraged by the positive results in Section 4.2.3 , we conducted a comprehensive parameter search to understand the optimal shape of the hourglass Transformer. We varied the ratio d_{\text{model}}/L (effectively trading network width for depth) while keeping the total parameter count fixed at 113M. We evaluated multiple hourglass configurations with different internal depths ( K\in\{1,2,4\} ) and width ratios ( d_{h}/d_{\text{model}} ), selected best on best results from Figure 3 . The results, summarized in Figure 4 , reveal several key insights. First, rather than a flat plateau, the width-depth trade-off exhibits a distinct U-shaped curve. Across all configurations, there is a clear ”sweet spot” for the d_{\text{model}}/L ratio, generally falling between 100 and 250. This indicates that neither extreme depth (small ratio) nor extreme width (large ratio) is optimal; instead, a balanced allocation where the model dimension is roughly 100 to 250 times the number of layers yields the lowest validation loss. Second, increasing the hourglass depth K consistently improves performance and robustness. As shown in Figure 4 , the curve for K=4 (green squares) lies below the curves for K=2 (orange triangles) and K=1 (blue circles/red diamonds) across the entire sweep. In fact, the K=4 configuration outperforms the conventional baseline (purple dashed line) across nearly the entire tested range of width-depth ratios. This suggests that a deeper internal FFN structure ( K>1 ) is a more parameter-efficient way to gain expressivity than simply reshaping the global attention-layer topology. Finally, the optimal configurations for the Hourglass architecture—specifically those with K=2 or K=4 in the optimal width-depth range—significantly outperform the conventional baseline. This confirms that reallocating parameters from the FFN’s ”brute-force” width to a combination of increased attention width and deeper, narrower FFN stacks is a preferable design strategy for this scale. 4.2.5 Scalability Across Model Scales Table 3 : Configurations of different model sizes for conventional baselines and hourglass variants. We follow ( 2 ) for setting model parameters of conventional baselines and ( 9 ) for learning parameters. We construct hourglass variants from these baselines with approximately matched parameter budgets within less than 0.001\% difference. The scaled model configurations of the hourglass variants are chosen based on searched configurations at 113M described in Section 4.2.2 and 4.2.4 . Model Size Attention Size FFN Size d_{\text{model}} d_{h} L K LR Batch (Tokens) Tokens (B) Conventional 113M 28M 85M 768 3072 12 1 6\times 10^{-4} 0.5M 2.5 Hourglass 51M 62M 1032 418 12 4 Conventional 403M 101M 302M 1024 4096 24 1 3\times 10^{-4} 0.5M 7 Hourglass 150M 253M 1376 557 24 4 Conventional 906M 227M 679M 1536 6144 24 1 2.5\times 10^{-4} 2M 16 Hourglass 415M 491M 2080 819 24 4 Conventional 1074M 269M 805M 2048 8192 16 1 4\times 10^{-4} 4M 21 Hourglass 649M 425M 2848 2486 20 1 Table 4 : Performance comparison of Hourglass vs. Conventional FFN Transformers across model scales (113M–1B). On validation sets, the Hourglass architecture achieves lower loss at smaller scales (up to 906M) and comparable performance at the 1B parameter scale. For downstream tasks, Hourglass models show higher accuracy in all reasoning and QA tasks below 403M and comparable performance up to the 1B parameter scale. The configurations of the models are from Table 3 . The Conventional (OLMo-2) is trained from configuration of OLMo2-1B . Downstream Tasks are report in accuracy (higher is better unless noted as PPL). Validation Downstream Tasks Size Model Val Loss Val PPL Arc Easy HellaSwag PIQA SciQ CommonsenseQA TriviaQA (PPL) NaturalQS (PPL) 113M Conventional 3.464 36.441 0.453 0.283 0.611 0.683 0.293 1.914 1.604 Hourglass 3.426 35.335 0.456 0.284 0.622 0.696 0.296 1.877 1.537 403M Conventional 3.087 25.398 0.518 0.351 0.666 0.768 0.332 1.586 1.427 Hourglass 3.064 24.909 0.521 0.349 0.640 0.777 0.333 1.591 1.386 906M Conventional 2.943 22.473 0.553 0.385 0.676 0.788 0.362 1.505 1.325 Hourglass 2.934 22.282 0.542 0.383 0.678 0.798 0.361 1.489 1.324 1074M Conventional 2.810 20.002 0.554 0.407 0.684 0.806 0.369 1.408 1.311 Hourglass 2.810 20.082 0.577 0.403 0.682 0.825 0.359 1.422 1.272 Conventional (OLMo-2) 2.833 20.466 0.568 0.405 0.669 0.810 0.373 1.428 1.323 Having validated the potential of Hourglass FFN under fixed attention size at the 125M scale, we now extend our evaluations to multiple model sizes (113M, 403M, 906M, and 1B) under matched total parameter budgets, where baseline configurations are listed in Table 3 . These comparisons allow d_{\text{model}} to vary, enabling each architecture to optimize its parameter allocation while maintaining the same overall capacity. For Hourglass FFN, we scaled the model by searching for configurations following the guidance presented in Section 4.2.2 and 4.2.4 . For 113M, 403M, and 906M models, we select configurations with d_{\text{model}}/L ratios between 57 and 87 and d_{h}/d_{\text{model}}\approx 0.4 following results in Figure 3 ; for 1B, we select the configuration with d_{\text{model}}/L\approx 142,d_{h}/d_{\text{model}}\approx 0.85,K=1 , and L=20 , the setup with the lowest validation perplexity on Figure 4 . Configurations and results of the setup are presented in Table 3 and Table 4 . Scalability of the Shape. Hourglass FFN variants consistently outperform conventional baselines in validation loss and perplexity at 113M, 403M, and 906M scales (see Table 4 ). For instance, at 906M, validation perplexity improves from 22.473 to 22.282. This advantage is largely maintained in downstream evaluations, where Hourglass models show particular strength in reasoning tasks, improving accuracy on Arc Easy, HellaSwag, PIQA, SciQ, and CommonsenseQA. At the 1B scale (1074M), the Hourglass variant performs on par with the strong narrow-wide-narrow baseline (Val PPL 20.082 vs 20.002) and surpasses the OLMo2 configuration (20.466). The diminishing relative gain at 1B compared to smaller scales may be attributed to the limit of how much we can compress the FFN capacity, suggesting that a minimal FFN capacity is still required for larger models. Parameter Distribution. The Hourglass architecture enables a flexible reallocation of the parameter budget. As shown in Table 3 , while conventional Transformers are consistently FFN-dominated (allocating \approx 3\times more parameters to FFN than Attention), our optimized Hourglass configurations shift this balance. At 906M, the parameters are roughly balanced ( 415 M Attn vs 491 M FFN). Notably, at the 1B scale, the configuration discovered by our search strategy at K=1 favors an Attention-dominated allocation ( 649 M Attn vs 425 M FFN). This extreme shift aligns with our findings at smaller scales: contextual processing (Attention) is a highly valuable resource, and the Hourglass FFN allows us to reallocate budget to it efficiently. 4.3 Ablation Studies We conduct a series of ablation studies on the 113M hourglass FFN Transformer to investigate the impact of key architectural hyperparameters. All experiments use the same dataset, tokenization, and training setup as described in Section 4.1 . Table 5 : Impact of increasing hourglass depth K . Performance (validation loss and perplexity) improves as depth K increases from 1 to 8. We fixed d_{\text{model}}=1032 , d_{h}=418 , and L=12 . K Model Size Attention Size FFN Size Val Loss Val PPL 1 67M 51M 16M 3.551 40.153 2 82M 51M 31M 3.489 37.632 4 113M 51M 62M 3.426 35.335 6 144M 51M 93M 3.391 34.051 8 175M 51M 124M 3.357 32.832 4.3.1 Varying K With d_{\text{model}}=1032 , d_{h}=418 , and L=12 fixed, we vary K\in\{1,2,4,6,8\} to study the effect of Hourglass FFN depth. As shown in Table 5 , increasing K consistently improves performance: validation perplexity drops from 40.153 at K=1 to 32.832 at K=8 , with corresponding reductions in both training and validation loss. The gains are most pronounced when moving from shallow configurations ( K=1 or K=2 ) to moderate depth ( K=4 or K=6 ), after which improvements become more incremental. Notice that under this setting, larger K also increases model size from 67M at K=1 to 175M at K=8 , suggesting a trade-off between depth and efficiency. This indicates that while deeper Hourglass FFNs yield better performance, practical deployments may prefer moderate K values (e.g., K=4 or K=6 ) to balance accuracy and parameter budget. Table 6 : Impact of varying d_{h} . The validation perplexity increases gradually as the ratio decreases from 0.8 to 0.1. We fixed d_{\text{model}}=1032 , K=4 , and L=12 . d_{\text{h}} d_{h}/d_{\text{model}} Model Size Attention Size FFN Size Val Loss Val PPL 836 0.8 175M 51M 124M 3.355 32.788 627 0.6 144M 51M 93M 3.384 33.747 418 0.4 113M 51M 62M 3.426 35.335 209 0.2 82M 51M 31M 3.500 38.075 103 0.1 66M 51M 15M 3.541 39.741 4.3.2 Varying d_{h} With d_{\text{model}}=1032 , L=12 , and K=4 fixed, we vary d_{h}/d_{\text{model}}\in\{0.8,0.6,0.4,0.2,0.1\} to study the effect of reducing the intermediate FFN dimension relative to the hidden dimension. As shown in Table 6 , decreasing the ratio leads to smaller model sizes from 175M at ratio =0.8 down to 66M at ratio =0.1 while performance degrades gradually. Validation perplexity increases from 32.788 at ratio =0.8 to 39.741 at ratio =0.1 , with the largest jumps occurring when the ratio drops below 0.4 . These results characterize the trade-off between FFN width and model efficiency under a fixed architectural configuration. Reducing d_{h} leads to substantial parameter savings in the FFN, while validation performance degrades gradually over a broad range of ratios. In particular, configurations with d_{h}/d_{\text{model}}\geq 0.4 maintain comparable performance despite significant reductions in model size. Below this range, performance degradation becomes more pronounced, indicating a transition where further FFN compression is less effective. Overall, the results delineate a practical operating regime for FFN width reduction in hourglass architectures. 5 Discussions and Future Work Our investigation challenges the long-standing ”narrow-wide-narrow” shape constraint in Transformer feed-forward networks (FFNs). We demonstrate that this design constraint is largely redundant: by constricting the FFN into a narrower, deeper ”hourglass” bottleneck, we can achieve competitive or superior performance while drastically reducing the parameter count of the FFN itself. This efficiency gain allows for a fundamental shift in resource allocation. Unlike standard Transformers where the FFN dominates the parameter budget, the Hourglass FFN enables the reallocation of parameters to the attention mechanism, allowing it to become the dominant component (e.g., utilizing 66M parameters for attention vs. 47M for FFN at the 113M scale). This inversion suggests that for smaller-scale models, enhancing contextual processing capacity is more valuable than expanding the static factual memory typically associated with wide FFNs. Furthermore, our results show that this performance is robust across various depth allocations ( K vs. L ) and intermediate ratios ( d_{h}/d_{\text{model}}\approx 0.4 – 0.6 ), highlighting the architectural flexibility of our proposal. While these results are promising, specific limitations remain. First, our comprehensive parameter search was constrained to the 113M scale due to compute resources, with findings extrapolated to models up to 1B parameters with limited ablation to confirm the extrapolation. Consequently, the scalability of the Hourglass FFN to tens or hundreds of billions of parameters remains an open empirical question. Second , this study isolated the FFN modification while fixing the attention module to MHA, leaving the interaction between Hourglass FFNs and advanced attention mechanisms unexplored. Alternative attention module such as Group-Query Attention ( 11 ) or Multi-head Latent Attention ( 6 ) could change the attention-FFN ratio landscape. Future work would prioritize verifying these findings at larger scales. Specifically, continued scaling of model dimensions ( d_{\text{model}} and K ) is a promising avenue, as similar manifold-constrained architectures have shown benefits from expanded width ( 40 ) . However, as models deepen, careful attention must be paid to signal propagation challenges, known as the ”curse of depth” ( 33 ) . Deeper hourglass FFN may result in accumulation of residual noise, causing diminishing gradient updates at the later layers. Ultimately, we envision the Hourglass FFN as a step towards more efficient architectures that could break free from historical design conventions to lift language models towards greater representational limits. Impact Statement This work revisits the established architectural conventions of Transformer language models, specifically the design of Feed-Forward Networks. By demonstrating that alternative ”Hourglass” structures can improve parameter and computational efficiency without sacrificing performance, our findings contribute to the ongoing effort to make large language models more efficient. This research has potential positive impacts by reducing the computational resources and energy consumption required for training and deploying models, thereby lowering the environmental footprint of AI development. Additionally, improved architectural efficiency may help democratize access to powerful language models. We do not foresee immediate negative societal consequences specific to this architectural modification, though the general ethical considerations of Large Language Models remain relevant. Appendix A Appendix A.1 Details of Experiment Settings A.1.1 Experimental Environment and Training Corpus All experiments are conducted based on the official training scripts provided in the OLMo codebase 2 2 2 https://github.com/allenai/OLMo and are executed on NVIDIA RTX 6000 Ada and B200 GPUs. The fixed random seed 6198 is used in all experiments. For training data, we adopt the Stage 1 pre-training corpus used to train the original OLMo-2 1B checkpoint. To control for the effects of stochasticity arising from data shuffling, we replicate the exact data ordering employed during the Stage 1 pre-training of OLMo-2 1B. From this fixed-order dataset, we select the first 2.5B, 7B, 16B, and 21B tokens to train the 113M, 403M, 906M, and 1074M models, respectively. A.1.2 Validation Datasets We conduct validation using the following datasets: Dolma Common Crawl ( 32 ) , Dolma The Stack ( 32 ) , M2D2 ( 28 ) , and WikiText ( 23 ) . The validation splits follow the official OLMo-2 1B configuration. For both cross-entropy loss and perplexity, we report the average scores computed across these four datasets. Complete validation results are provided in Section A.2 . A.1.3 Downstream Evaluation Datasets We conduct downstream task evaluation on the following benchmark datasets: Arc Easy ( 4 ) , HellaSwag ( 43 ) , PIQA ( 1 ) , SciQ ( 13 ) , CommonsenseQA ( 35 ) , TriviaQA ( 14 ) , and NaturalQS ( 17 ) . Among these, TriviaQA and NaturalQS are evaluated using perplexity, while all remaining tasks are assessed using accuracy as the evaluation metric. A.1.4 Training Setting Details Across all experiments, we use AdamW as the optimizer and adopt the cross-entropy loss augmented with an auxiliary softmax loss as the training objective. For AdamW, the hyperparameters are set to \beta_{1}=0.9 , \beta_{2}=0.95 , and \epsilon=1\times 10^{-8} , with a weight decay coefficient of \lambda=0.1 . SwiGLU is employed as the activation function for all models. Rotary Position Embedding (RoPE) is utilized for positional encoding in both the Transformer architecture and the proposed hourglass blocks. A cosine learning rate scheduler with warmup is applied in all experiments, where the number of warmup tokens varies according to model size. Additional hyperparameters and model-size-dependent configurations are reported in Table 3 and Table 7 . Table 7 : Additional experimental hyperparameters for different model sizes , including learning rates, warmup tokens, attention heads, and sequence lengths. Model Size Learning Rate WarmUp Tokens Attention Heads Max Sequence Size 113M 6\times 10^{-4} 50M 12 2048 403M 3\times 10^{-4} 50M 16 2048 906M 2.5\times 10^{-4} 50M 16 2048 1074M 4\times 10^{-4} 200M 16 4096 A.2 Detailed Validation Results Table 8 : Detailed validation loss and perplexity on our 4 validation datasets , including Dolma Common Crawl, Dolma The Stack, M2D2 and WikiText. Validation Loss Validation Perplexity Size Model Dolma Common Crawl Dolma The Stack M2D2 WikiText Dolma Common Crawl Dolma The Stack M2D2 WikiText 113M Conventional 3.86 2.477 3.873 3.646 47.448 11.907 48.09 38.321 Hourglass 3.833 2.414 3.858 3.6 46.188 11.184 47.384 36.583 403M Conventional 3.523 2.064 3.568 3.195 33.874 7.875 35.442 24.403 Hourglass 3.512 2.026 3.548 3.169 33.502 7.583 34.756 23.793 906M Conventional 3.412 1.837 3.463 3.062 30.317 6.278 31.924 21.371 Hourglass 3.41 1.829 3.457 3.04 30.27 6.226 31.719 20.915 1074M Conventional 3.306 1.668 3.394 2.871 27.275 5.301 29.775 17.655 Hourglass 3.321 1.659 3.396 2.865 27.676 5.256 29.849 17.547 OLMo2 3.333 1.686 3.407 2.905 28.032 5.397 30.174 18.262 Table 8 presents the complete validation loss and perplexity results for all four model sizes (113M, 403M, 906M, and 1074M) evaluated across four distinct validation datasets.