Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups
Aditya Pratap Singh · 2026
Read Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups by Aditya Pratap Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29024.txt.
Introduction
Singh adapts a strong open translation system, IndicTrans2, so it can handle casual conversation in its 21 Indic languages. The system was trained on general text and sounds stiff on chat and subtitles. Using only public data, Singh fine-tunes it for conversational register. Then he fights the side effect: plain fine-tuning improves casual scores but makes the model forget the general domain.
Mixing general data back into training, called experience replay, and then averaging the tuned weights with the base model, called a model soup, removes the trade-off. The adapted model beat the original on conversational scores in all 21 languages, while matching it on FLORES. Tests confirm the casual gains are real and general scores are not significantly worse. Singh is candid about scope: human and model judges did not confirm a quality gain, so it may be register matching, not better translation.
Catalog record
| CREATOR | Aditya Pratap Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups Aditya Pratap Singh Abstract Abstract IndicTrans2 is the strongest open English to Indic translation system, but like most systems it is trained on general text and tends to sound stiff on casual, conversational input. We adapt IndicTrans2-1B to conversational register across all 21 Indic languages using only public data (OpenSubtitles, BPCC-H-Daily, Tatoeba). Plain fine-tuning improves conversational chrF but forgets the general domain (it drops 3.9 chrF on FLORES for Hindi). Mixing general data back into training (experience replay) and then averaging the fine-tuned weights with the base (model souping) removes that trade-off: the resulting model beats IndicTrans2-1B on conversational chrF in every one of the 21 languages (mean +6.2) while matching it on FLORES (mean change -0.17, all within 0.7 chrF). Paired bootstrap tests confirm the conversational gains are significant (p \leq 0.004) and that FLORES is not significantly degraded. We are deliberate about scope: these are chrF gains, and a blind human plus multi-model LLM check does not confirm them as a perceived quality improvement, so we treat the conversational gain as largely a register match to the references rather than proof of better translation. The techniques are not new; the contribution is the honest, end-to-end study in the Indic conversational setting. 1. Introduction Most translation systems are tuned and measured on general benchmarks like FLORES-200. But a lot of real translation is conversational: chat, subtitles, voice commands. That text is shorter, more idiomatic, and more elliptical than news, and a model that scores well on FLORES can still produce stilted output on a casual line. For Indian languages, IndicTrans2 is the best open option, trained on the 230M-pair BPCC corpus. It is a general-domain model, so we asked a simple question: Can we adapt a strong multilingual model to conversational register across many languages, without hurting its general quality, using only public data? By chrF the answer looks like a clear yes. Plain fine-tuning causes the expected catastrophic forgetting, and experience replay plus model souping fixes it while delivering large conversational gains across all 21 languages with no significant FLORES loss (§5). We did not stop there. The whole point is a claim about translation quality , not about chrF, so we also ran a blind human and LLM evaluation (§6). That check does not confirm a quality gain. We report both results honestly: the recipe works by the metric, but the metric overstates how much it helps in conversation. Contributions. 1. A reproducible study of conversational adaptation for English to Indic translation across 21 languages, built entirely from public data, with a working anti-forgetting recipe and significance tests. 2. An honest account of the recipe’s limits: forgetting under plain fine-tuning, inflated gains on easy in-domain test sets, a metric that overstates conversational quality, and a floor set by the base model on the lowest-resource languages. The techniques we use, model soups and replay, are established. The value here is the systematic study and its honest evaluation in a setting nobody had measured this way. 2. Related Work Domain adaptation for NMT. Fine-tuning a general model on in-domain data is standard, and its main failure mode is forgetting the general domain. Mixing general data back in during fine-tuning is a known fix and a form of experience replay [Chu et al., 2017; Robins, 1995]. Model soups. Averaging the weights of fine-tuned models improves accuracy [Wortsman et al., 2022a]; WiSE-FT averages a fine-tuned model with its starting point to balance in-domain and out-of-domain performance [Wortsman et al., 2022b]. We apply that base-to-fine-tune averaging to translation. Metrics versus human judgment. chrF and BLEU reward overlap with the reference and can diverge from human assessment [Callison-Burch et al., 2006; Mathur et al., 2020; Freitag et al., 2021]. When adaptation shifts a model toward the reference style, the metric can rise without quality rising. Our result is a concrete case of this in conversational Indic translation. Indic MT. IndicTrans2 [Gala et al., 2023] and BPCC are the base model and a data source. WAT hosts Indic translation shared tasks [Nakazawa et al., 2024]. 3. Data Base model. ai4bharat/indictrans2-en-indic-1B (1.1B parameters). Conversational data (English to Indic, each pair tagged with its target language): • OPUS OpenSubtitles for the five Indic languages with substantial subtitle data (hi, ml, ta, te, ur), capped at 40k per language. • BPCC-H-Daily, human-written everyday and assistant utterances, available for all 21 languages (about 1.8k to 11k pairs each). • Tatoeba everyday sentences (Hindi). After filtering for length, length ratio, exact duplicates, and script, the conversational set is 294,119 pairs across 21 languages (train 283,753 / dev 4,183 / test 6,183). The Hindi-only runs also apply a LaBSE cosine filter at 0.70 to drop misaligned subtitle pairs. General anchor (for replay): BPCC-H-Wiki, a general register, sampled roughly 1:1 with the conversational data (254,469 pairs). The mixed training set is therefore 538,222 pairs. Conversational volume is very uneven. The five OpenSubtitles languages hold 30k to 48k pairs each; the other sixteen rely on BPCC-H-Daily (about 2k to 11k). This skew is baked into the available data and it shapes the results (§7). 4. Method We compare three setups, all fine-tuning IndicTrans2-1B (IndicProcessor preprocessing with per-example target-language tags, bf16, Adafactor, single NVIDIA A10G GPU): 1. Plain FT. Fine-tune on conversational data only. 2. Mixed FT (replay). Fine-tune on conversational plus the general anchor, about 1:1. 3. Model soup. Average the base and a fine-tuned model, soup = (1 - a) * base + a * finetuned , sweeping the mixing weight a and picking the best conversational-to-general trade-off. We stream the two checkpoints’ tensors so memory stays bounded. Hindi-only models use 146k conversational pairs (2 to 3 epochs). The multilingual models use the full 294k conversational and 538k mixed sets (2 epochs and 1 epoch, matched on examples seen). We score chrF2 (sacreBLEU) on a held-out conversational test and on FLORES-200 devtest, per language, with paired bootstrap significance for representative languages. 5. Results By the metric, the recipe works. These numbers are real; §6 then asks whether they mean what they appear to. 5.1 Hindi case study Table 1 isolates the mechanism on Hindi. Figure 1 shows the same story. Table 1: Hindi (chrF2). Model conv FLORES IndicTrans2 base 51.6 61.7 Plain conv-FT 57.0 57.8 (forgetting) Mixed FT 56.0 60.6 Model soup (a=0.6) 54.1 61.8 Plain fine-tuning gains 5.4 conversational chrF but loses 3.9 on FLORES, which is clear forgetting. Mixed FT recovers most of the general quality, and the soup lands at or above the base on both axes. Sweeping a from 0.3 to 0.7 puts the knee at 0.6. Figure 1: On Hindi, plain fine-tuning lifts conversational chrF but drops FLORES below the base. Replay keeps most of the general quality, and averaging with the base restores FLORES while staying ahead on conversation. 5.2 All 21 languages The mixed soup improves conversational chrF in every language (mean +6.2) and barely moves FLORES (mean -0.17, all within 0.7). Figure 2 shows the per-language gains; Figure 3 shows that the conversational gain comes with essentially no change in general quality. Table 2: All 21 languages, base to mixed-soup (chrF2). Lang conv (delta) FLORES (delta) asm 61.9