A Pāṇinian Foundation for Indic Language Processing CCS: Computing methodologies Machine translation CCS: Computing methodologies Speech recognition CCS: Computing methodologies Phonology / morphology CCS: Computing methodologies Language resources CCS: Computing methodologies Discourse, dialogue and pragmatics CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Lexical semantics

Ritwik Banerjee, Lav R. Varshney · 2026

Read A Pāṇinian Foundation for Indic Language Processing CCS: Computing methodologies Machine translation CCS: Computing methodologies Speech recognition CCS: Computing methodologies Phonology / morphology CCS: Computing methodologies Language resources CCS: Computing methodologies Discourse, dialogue and pragmatics CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Lexical semantics by Ritwik Banerjee, Lav R. Varshney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24172.txt.

Introduction

Banerjee and Varshney argue that language technology for over a billion speakers of Indic languages is fragmented for a fixable reason. Tools and benchmarks are built language by language. Each new language then restarts the work of analysers, parsers, and datasets. Yet two millennia of convergence around Sanskrit gave these languages a shared grammatical frame. The ancient grammarian Pāṇini formalised it in his Aṣṭādhyāyī. It cuts across family lines.

That shared frame, the authors contend, can supply the unifying design the field lacks. Benchmarks grounded in it would judge systems against the common structure, not isolated language quirks. Banerjee and Varshney sketch effects across machine translation, speech recognition, morphology, and information extraction. Systems built on the shared base should prove more accurate and more data-efficient. Knowledge learned for one Indic language can transfer to its neighbours, instead of being rebuilt from scratch each time.

Catalog record

CREATORRitwik Banerjee, Lav R. Varshney
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Pāṇinian Foundation for Indic Language Processing CCS: Computing methodologies Machine translation CCS: Computing methodologies Speech recognition CCS: Computing methodologies Phonology / morphology CCS: Computing methodologies Language resources CCS: Computing methodologies Discourse, dialogue and pragmatics CCS: Computing methodologies Information extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Lexical semantics Ritwik Banerjee, Lav R. Varshney Abstract . More than a billion people communicate in Indic languages, yet the natural language processing infrastructure serving them remains fragmented and underdeveloped. The cause is structural: the field organizes its tools and benchmarks around individual languages or small subsets of genealogical language families, building separate analyzers, parsers, and datasets for each language and starting over for the next. This overlooks a deep regularity. Through more than two millennia of convergence around Sanskrit, Indic languages came to share a morphosyntactic architecture formalized in Pāṇini’s grammar, the Aṣṭādhyāyī. This cuts across genealogical lines, uniting languages through a common framework. We argue that this Pāṇinian framework supplies a unifying computational architecture the field has lacked, and that benchmarks grounded explicitly in it would make Indic language systems more accurate, more data-efficient, and more transferable, effectively merging many apparently disparate and sparse Indic language resources into a single high-resource metalanguage bedrock. We propose a four-part benchmark suite to render this shared architecture explicit, measurable, and ready to be leveraged for practical applications. Moreover, we underscore the question it raises for interpretability research: whether neural models trained on these languages come to represent Pāṇini’s categories on their own. \correspondingauthor One Metagrammar for a Billion Voices: Benchmarks and Architecture 1. Introduction Over a billion people across South Asia and beyond communicate daily in Indic languages — Hindi, Bengali, Tamil, Telugu, Marathi, Santali, and dozens more — yet the computational infrastructure serving these speakers/writers remains critically underdeveloped relative to its global significance. As AI-powered tools for translation, accessibility, education, and information retrieval become central to modern life, the gap between what NLP can do for English or Chinese and what it can do for Indic languages carries real costs, both human and economic. Closing that gap is an important engineering challenge facing the computing community today. Progress, however, has been slow. The primary reason is not a lack of data or talent, but a structural problem in how the field has organized itself. The longstanding emphasis on genealogical divisions in the taxonomy of Indic languages has led to a fragmentation of computational approaches: separate morphological analyzers, parsers, and annotated datasets for each language family, or even each individual language. The result is redundant engineering effort, acute resource constraints, and benchmark infrastructure that cannot transfer across languages. The scale of this duplication is concrete: the Universal Dependencies ( 17 ) collection alone, as of release 2.18 in May 2026 ( 58 ) , maintains 24 separately annotated treebanks across 18 Indic languages 1 1 1 universaldependencies.org , each built as an independent effort, and conversions between their annotation schemata are frequently lossy ( 50 ; 49 ) . Every new language effectively requires building from scratch. This is not a denial of the value or the tremendous engineering achievements of massively multilingual models ( 41 ) . But their design objective is breadth, not depth: covering hundreds of languages optimizes translation fluency and coverage, not the morphosyntactic and semantic structure that distinguishes Indic languages, and no current benchmark would reveal the difference. Such models do achieve cross-lingual transfer, but the units that carry it across Indic languages --- the shared and borrowed vocabulary, the parallel inflectional patterns, and the subwords that encode them --- are shared precisely because these languages are organized by a common Pāṇinian architecture 2 2 2 Pāṇini was a Sanskrit grammarian from around the fifth century BCE; his Aṣṭādhyāyī (“Eight Chapters”) specifies the language in some 4,000 ordered rules built from a formal metalanguage and explicit rule-precedence mechanisms. The parallel between this rule system and modern formal grammars was drawn early in the history of modern computing: writing in the Communications of the ACM, 24 proposed that Backus-Naur form be renamed “Pāṇini-Backus form”. The correspondence is at best cosmetic, however — the Pāṇinian rule formalism is in fact strictly more powerful than the context-free grammars described by BNF ( 47 ) . , whether through inheritance, sustained contact, or independent convergence. The transfer is therefore already running on Pāṇinian rails; the models merely exploit them implicitly and incompletely, through incidental surface signals rather than the deeper structure reflected by those signals. Empirical evidence bears the fingerprints of this shared architecture: cross-lingual transfer is stronger within Indic languages than between Indic and non-Indic languages even when script differences are controlled for ( 3 ; 39 ) , and morphological analyzers built on Pāṇinian principles transfer across language families rather than merely within them ( 19 ; 21 ) . Scale alone, however, quickly reaches a real ceiling. A fixed-capacity model spread across many languages dilutes per-language quality, with the gains gravitating towards high-resource languages ( 15 ) , leaving most Indic languages underserved. Furthermore, generic subword tokenizers fragment morphologically rich Indic words rather than recovering their morphemes ( 11 ) . Making the shared substrate explicit converts this transfer into an interpretable and designed mechanism, so that the meaning of a lexeme can be transparently tracked as it is inflected and compounded across languages, rather than lost to subword fragmentation determined by ad hoc statistics. Naïve data sharing across Indic languages, with no Pāṇinian grounding, already increases tagging accuracy ( 46 ) , suggesting that there is ample room for empirical improvements through explicit grounding. Since the transfer behavior of multilingual models already reveals that they implicitly learn a structural prior commonly held across languages, it is likely that giving Indic languages an interpretable grammar already known to be a common foundation will boost language comprehension across the board. The fragmented, language-by-language approach and the opacity of multilingual models err in opposite directions around the same fact: the structural unity of Indic languages is real — the former ignores it and pays in redundant effort, whereas the latter gains from it without ever recognizing its central role. That unity is not a modeling artifact but the structural commons observed consistently by native multilingual users, across the boundaries of Indic languages and genealogies: remarkable parallels in morphological patterns such as verb conjugation, case marking, and agglutination; syntactic structures including postpositions and participial relatives; and shared frameworks for expressing agency, causation, aspect, and evidentiality. Far from superficial lexical borrowings, these phenomena reflect deep architectural similarities engendered by over two millennia of linguistic convergence where Sanskrit has functioned as a formal intellectual “metalanguage” across South Asia and beyond. This is not an isolated observation. In other linguistic traditions, Latin shaped the prestige registers and grammatical self-conception of English ( 10 ; 35 ) . In the case of Indic languages, however, the shared architecture runs considerably deeper: Tamil grammarians, working within their own tradition, arrived at structural conclusions that converge with Pāṇinian categories --- a finding that is computationally significant precisely because it transcends genealogical boundaries. Dialectal and register variations, too, operate along predictable continua rather than discrete breaks: dialects typically differ in phonological realization or lexical choice while preserving fundamental

Read the complete text