TALL : A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages
Moshe Ofer, Orel Zamler, Amos Azaria · 2025
Read TALL : A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages by Moshe Ofer, Orel Zamler, Amos Azaria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05057.txt.
Introduction
Moshe Ofer, Orel Zamler, and Amos Azaria set out TALL, a trainable build that lifts large language models in tongues with scant training data. Their paper is from two thousand twenty-five.
Such models shine in rich tongues but lag in poor ones. TALL joins a large model to two translation models that work both ways. Text in a poor tongue is turned into a rich form. The large model's strength can then be used, while traits of the first tongue are kept by match layers and custom parts. Tests on Hebrew beat plain use, naive translation, and fine-tuning. Most parts stay frozen; just light adapters train.
Catalog record
| CREATOR | Moshe Ofer, Orel Zamler, Amos Azaria |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05057
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TALL : A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages Moshe Ofer, Orel Zamler, Amos Azaria Abstract Large Language Models (LLMs) excel in high-resource languages but struggle with low-resource languages due to limited training data. This paper presents TALL ( T rainable A rchitecture for Enhancing L LM Performance in L ow-Resource Languages), which integrates an LLM with two bilingual translation models. TALL transforms low-resource inputs into high-resource representations, leveraging the LLM’s capabilities while preserving linguistic features through dimension alignment layers and custom transformers. Our experiments on Hebrew demonstrate significant improvements over several baselines, including direct use, naive translation, and fine-tuning approaches. The architecture employs a parameter-efficient strategy, freezing pre-trained components while training only lightweight adapter modules, balancing computational efficiency with performance gains. 1 1 1 Our code is available at https://github.com/MosheOfer1/TALL . 1 Introduction Large Language Models (LLMs) have significantly advanced natural language processing (NLP) for high-resource languages like English, but their performance in low-resource languages remains limited due to sparse data, linguistic complexity, and the scarcity of annotated datasets and pre-trained models. Efforts to address these challenges have primarily focused on transfer learning Pan and Yang 2010 , which leverages knowledge from high-resource languages to enhance performance in low-resource settings, though notable limitations persist due to linguistic diversity and persistent data scarcity. This paper introduces TALL , a novel architecture that harnesses the strengths of LLMs trained on high-resource languages to improve performance on low-resource languages. The key innovation lies in integrating three pre-trained models: an LLM trained primarily on a high-resource language, and two translation models that bridge the gap between high and low-resource languages. Through carefully designed dimension alignment adapters Rebuffi et al. 2017 ; Houlsby et al. 2019 and custom transformer Vaswani et al. 2017 components, TALL ensures that the linguistic knowledge embedded in high-resource LLMs benefits low-resource language processing. We validate TALL through experiments on Hebrew, chosen for its rich morphology, complex syntax, and limited available datasets. Our results demonstrate significant improvements in accuracy compared to direct use of LLMs, naive translation approaches, soft prompting, and fine-tuning, highlighting the effectiveness of our architecture. 2 Related Work Our work builds upon several approaches to improving LLM performance in low-resource settings: Cross-lingual Transfer Prior research has explored transferring knowledge from high-resource to low-resource languages through various methods. Cross-lingual in-context learning Cahyawijaya et al. 2024 leverages examples from high-resource languages to improve performance in low-resource contexts. However, these approaches often lack the structural integration that TALL provides. Parameter-Efficient Adaptation Adapter-based approaches Pfeiffer et al. 2020a ; Pfeiffer et al. 2020b insert small, trainable modules into frozen pre-trained models to adapt them to new tasks or languages. Our dimension alignment adapters follow this principle, but specifically focus on aligning representations between different language spaces. Translation-based Strategies Recent work has introduced creative translation-based strategies. Notable examples include TaCo Upadhayay and Behzadan 2024 , which integrates translation into reasoning; CoTR Deshpande et al. 2024 , which translates inputs into a high-resource language for processing; and TALENT Guo et al. 2024 , which employs a curriculum-inspired approach to grammar learning before translation. Concept-level Processing Large Concept Models LCM-team et al. 2024 operate on sentence-level "concepts" in a unified multilingual embedding space, offering an alternative to token-level prediction. Similar to TALL , they aim to create cross-lingual representations that preserve semantic meaning across languages. 3 Model Architecture 3.1 Architecture Overview TALL strategically integrates pre-trained components with lightweight, trainable adapters to efficiently process low-resource language inputs. As illustrated in Figure 1 , the architecture consists of seven main stages (numbered 1-7 in the figure): Figure 1: Overview of the TALL architecture with numbered components (1-7) corresponding to the processing stages described in Section 3.1 . * For input high-resource sentences, the last word is removed before translation during training; ** For target sentences, teacher forcing is used during training while outputs are generated auto-regressively during inference. 1. Source Language Encoding: The input sentence in the low-resource language is processed by a pre-trained low-to-high-resource (LR-HR) encoder, generating hidden states that represent the intermediate semantic form. 2. Dimension Alignment (Source to LLM): A trainable dimension alignment adapter transforms the encoder’s hidden states to match the dimensional requirements of the LLM, ensuring seamless integration. 3. Custom Transformer (LR-HR): This trainable component receives the LLM tokenized translated sentence embeddings (during training, the last word is removed before the sentence is translated) and applies cross attention Gheini et al. 2021 to the aligned hidden states, ensuring the LLM receives representations enriched with information from the original input. 4. LLM Integration: The processed sequence is passed to the frozen LLM for next-token prediction, leveraging its strong language modeling capabilities in the high-resource language. 5. Dimension Alignment (LLM to Target): A second alignment adapter transforms the LLM’s hidden states to match the dimensional requirements of the second translator. 6. Custom Transformer (HR-LR): This component processes the aligned hidden states from the LLM, preparing them for the final decoding stage. 7. Target Language Decoding: The HR-LR decoder generates the final text in the target low-resource language by applying cross-attention between the encoded representation and the target language embeddings. The implementation details for each component are as follows: Translation Models We use pre-trained Marian MT models Junczys-Dowmunt et al. 2018 for translation between Hebrew and English, leveraging their robust bilingual capabilities. Dimension Alignment Adapters These adapters consist of trainable fully-connected layers with layer normalization and GELU activations. They project hidden states between the different dimensional spaces required by each pre-trained component. Custom Transformers We implement two custom transformer modules derived from Marian configurations: (1) a transformer that conditions on high-resource tokens and cross-attends to dimension-aligned low-resource encoder states, and (2) a transformer that refines representations before the final decoding stage. Parameter Efficiency A key advantage of TALL is its parameter efficiency. By freezing the large pre-trained components (translation models and LLM) and training only the lightweight alignment adapters and custom transformers, we minimize computational overhead while maximizing performance gains. In our implementation with bloomz-560m Muennighoff et al. 2023 , only 14.35% of the total parameters (126.8M out of 883.5M) are trainable, while for QWEN2.5-0.5b Qwen-Team 2024 , 13.47% of the total parameters (107.7M out of 799.2M) are trainable. Dataset Approach Bloomz-560m QWEN2.5-0.5b WIKI-SLVM Direct Hebrew 0.63 3.77 Fine-tuned 0.16 2.75 Naive 2.11 2.85 Soft prompt 2.50 1.96 From Scratch 2.93 3.99 TALL 5.59 5.15 HE-LYRICS Direct Hebrew 0.14 2.11 Fine-tuned 0.36 2.22 Naive 1.83 2.20 Soft prompt 2.05 1.90 From Scratch 2.08 2.92 TALL 5.44 4.77 Table 1: Accuracy comparison (%) for predicting the missing Hebrew word. TALL, soft prompt, fine-tuning, and from-scratch were trained on the news dataset. A detailed breakdown of the TALL parameters, including overall statistics and module-wise counts for both bloomz TALL and Qwen TALL, is provided in Tables 2 , 3 , and 4 in the Appendix. 3.2 Training Procedure We trained TALL on 256,000 Hebrew sentences from news articles. The dataset underwent standard preprocessing, including removal of duplicates and filtering for sentences between 5-30 words in length. The model is trained using teacher forcing, where ground-truth target tokens are provided as decoder inputs. Importantly, the loss is computed only for the final missing token in the sequence. This limitation is inherent to the TALL