CoFrGeNet: Continued Fraction Architectures for Language Generation

Amit Dhurandhar, Vijil Chenthamarakshan, Dennis Wei, Tejaswini Pedapati, Karthikeyan Natesan Ramamurthy, Rahul Nair · 2026

Read CoFrGeNet: Continued Fraction Architectures for Language Generation by Amit Dhurandhar, Vijil Chenthamarakshan, Dennis Wei, Tejaswini Pedapati, Karthikeyan Natesan Ramamurthy, Rahul Nair (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21766.txt.

Introduction

Amit Dhurandhar and co-authors look for an alternative to the Transformer for generating language. Transformers are the default architecture for this task, but the team takes inspiration from a classic math object. Continued fractions, where a value is built from nested layers of simpler parts.

Their 2026 preprint introduces a new family of functions for generative modeling, and the networks that implement it, called CoFrGeNets. The idea is to see whether this nested structure can produce language with different strengths or costs than attention-based models. Work of this kind matters because nearly all modern language systems share one design, so a credible second family would widen the field's options considerably.

Catalog record

CREATORAmit Dhurandhar, Vijil Chenthamarakshan, Dennis Wei, Tejaswini Pedapati, Karthikeyan Natesan Ramamurthy, Rahul Nair
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21766

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoFrGeNet: Continued Fraction Architectures for Language Generation Amit Dhurandhar, Vijil Chenthamarakshan, Dennis Wei, Tejaswini Pedapati, Karthikeyan Natesan Ramamurthy, Rahul Nair Abstract Transformers are arguably the preferred architecture for language generation. In this paper, inspired by continued fractions, we introduce a new function class for generative modeling. The architecture family implementing this function class is named CoFrGeNets - Continued Fraction Generative Networks. We design novel architectural components based on this function class that can replace Multi-head Attention and Feed-Forward Networks in Transformer blocks while requiring much fewer parameters. We derive custom gradient formulations to optimize the proposed components more accurately and efficiently than using standard PyTorch-based gradients. Our components are a plug-in replacement requiring little change in training or inference procedures that have already been put in place for Transformer-based models thus making our approach easy to incorporate in large industrial workflows. We experiment on two very different transformer architectures GPT2-xl (1.5B) and Llama3 (3.2B), where the former we pre-train on OpenWebText and GneissWeb, while the latter we pre-train on the docling data mix which consists of nine different datasets. Results show that the performance on downstream classification, Q& A, reasoning and text understanding tasks of our models is competitive and sometimes even superior to the original models with 2 3 \frac{2}{3} to 1 2 \frac{1}{2} the parameters and shorter pre-training time. We believe that future implementations customized to hardware will further bring out the true potential of our architectures. Figure 1: Above we see a Transformer block consisting of attention and FFN layers. We propose candidate CoFrNet architectures for Transformer (causal) attention and FFN layers. The circles with the blue curves denote the \frac{1}{x} non-linearity in our architectures. The zoomed out image on the far right shows the mapping between the pictorial representation and the actual equations. Details of the architectures are discussed in section 4 . 1 Introduction Since OpenAI’s ChatGPT release at the end of 2022, Large Language Models (LLMs) ( 33 ) have been getting increasingly infused into multiple user applications and platforms across the world. The most prevalent architecture behind these models is the Transformer architecture ( 46 ) , which consists of an (multi-head) Attention block and a Feed Forward Network (FFN) with single large hidden layer. In this paper, we propose novel architectural components based on a radically different function class inspired by continued fractions. Taking inspiration from ( 31 ) , where continued fraction architectures CoFrNets were introduced for the supervised setting, we build new architectures for the generative setting providing alternatives for attention and FFN in Transformer blocks. Given a canonical form for continued fractions a_{0}+\frac{1}{a_{1}+\frac{1}{a_{2}+\cdots}} (ladder like structure) where, a_{k} s are complex numbers, CoFrNets ( 31 ) were introduced for supervised learning problems where in place of the a_{k} s, linear functions of the input x\in\mathbb{R}^{p} are computed by taking the inner product of x with weight vector w_{k}\in\mathbb{R}^{p} in each layer k (or also referred to as step of the ladder). 1 1 1 A constant term is assumed to be absorbed in x . The reciprocal of the function thus far is applied as a nonlinearity in each layer leading to the following kind of form for a single CoFrNet ladder: \displaystyle w_{0}x+\frac{1}{w_{1}x+\frac{1}{w_{2}x+\cdots}} (1) Here w_{k} s are the learnable parameters. Essentially, the input x is passed to each layer which gets multiplied by the corresponding parameter vectors and the reciprocal of the values of the previous layer are added to this. This simple architecture was shown to have universal approximation capabilities when we ensemble enough of these ladders. However, the above contributions were for the supervised setting and it is not clear if such architectures can also be built for representation learning and sequence generation, where we: i) Need to produce multi-dimensional outputs, ii) learn richer functions and iii) model sequences causally i.e. learning parameters that depend only on prior tokens. Moreover, the \frac{1}{x} non-linearity is inefficient to compute in forward and backward passes especially when the depth d and number of ladders L is large. This is because one has to compute the inverse d\times L times and it is known that division is many times slower than multiplication in modern hardware. We address the above challenges in this paper by making the following contributions that distinguish it significantly from ( 31 ) : 1) We propose novel continued fraction architectures for (causal) attention and FFNs as depicted in Figure 1 . We call our architecture with both components replaced as Co ntinued Fr action Ge nerative Net work (CoFrGeNet). We report results replacing either FFN or attention or both offering the possibility to the user of replacing only one or both of the components for their application. Even replacing one component can offer significant parameter and training time savings as seen in our experiments. 2) We propose an alternative representation for the ladders and derive custom formulas for the gradients that reduces the number of divisions from d to a constant of just 1 for a d -depth ladder. This greatly enhances both training and inference efficiency. 3) We propose a custom training schedule to update CoFrGeNet parameters. This is described in section 5 . 4) We pre-train our models on OpenWebText (OWT) ( 12 ) , GneissWeb ( 11 ) and the docling data mix ( 43 ) showing that our models are competitive or outperform the corresponding Transformer models. We compare with Transformers since we are replacing its components making it a fair comparison. For an apples-to-apples comparison with other model architectures such as Mamba ( 14 ) one would want to replace its hidden state function with novel (to be designed) CoFrNet components, which would be a significant independent contribution in itself that we leave for future work. 2 Preliminaries We introduce notation and also discuss some of properties of continued fractions. The generalized form for a continued fraction is a_{0}+\frac{b_{1}}{a_{1}+\frac{b_{2}}{a_{2}+\cdots}} , where a_{k} s and b_{k} s can be complex numbers. If none of the a_{k} or b_{k} are zero \forall k\in\mathbb{N} , then using equivalence transformations ( 19 ) , one can create simpler equivalent forms where either the b_{k}=1 or the a_{k}=1 \forall k\in\mathbb{N} , with a_{0}=0 in the latter form. A more concise way to write these two forms is as follows: i) a_{0}+\frac{1}{a_{1}+\frac{1}{a_{2}+\cdots}}\equiv a_{0}+\frac{1}{a_{1}+}\frac{1}{a_{2}+\cdots} and ii) \frac{b_{1}}{1+\frac{b_{2}}{1+\cdots}}\equiv\frac{b_{1}}{1+}\frac{b_{2}}{1+\cdots} . Form i) is known as the canonical form . One of the nice properties of continued fractions is that in representing any real number with natural number parameters a_{k},b_{k}\in\mathbb{N} , the rational approximations formed by any of its finite truncations (termed convergents ) are closer to the true value than any other rational number with the same or smaller denominator. A continued fraction is therefore the best possible rational approximation in this precise sense ( 19 ; 29 ) . In this work, we consider continued fractions in canonical form, with partial numerators b_{k}=1 for k=1,\dots,d and depth d . We thus view continued fractions as functions f of the partial denominators, where we separate a_{0} from the others and use a\coloneqq(a_{1},\dots,a_{d}) as a shorthand. Hence we write \displaystyle f(a_{0},a)=a_{0}+\frac{1}{a_{1}+}\frac{1}{a_{2}+}\cdots\frac{1}{a_{d-1}+}\frac{1}{a_{d}}=a_{0}+\tilde{f}(a), (2) where we also define \tilde{f}(a) as the “fractional part” of f(a_{0},a) . Another way of representing a continued

Read the complete text