Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task

Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Christan Grant, Daisy Wang · 2026

Read Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task by Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Christan Grant, Daisy Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20626.txt.

Introduction

Aashish Dhawan and four colleagues report the University of Florida Gators entry to a shared task on cultural image captioning in Indigenous languages. Their two-stage pipeline first writes a Spanish caption, then produces the target caption with retrieval-augmented many-shot prompting. Gains over the baseline reached 164.1 percent for Bribri, 131.7 percent for Guarani, and 122.6 percent for Orizaba Nahuatl on development data. The entry won overall, placing second of five finalists in human judging.

Catalog record

CREATORAashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Christan Grant, Daisy Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37104
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20626

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Retrieval-Augmented Long-Context Translation for Cultural Image Captioning: Gators submission for AmericasNLP 2026 shared task Aashish Dhawan, Christopher Driggers-Ellis, Dzmitry Kasinets, Christan Grant, Daisy Wang Abstract We present the University of Florida Gators submission to the AmericasNLP 2026 shared task on cultural image captioning for Indigenous languages. Our two-stage pipeline generates a Spanish intermediate caption with Qwen2.5-VL, then produces the target-language caption using retrieval-augmented many-shot prompting with Gemini 2.5 Flash. We achieve 164.1%, 131.7%, and 122.6% improvements over the shared task baseline for Bribri, Guaraní, and Orizaba Nahuatl captioning, respectively, in our dev set evaluation and maintain >150% improvements for the Bribri and Orizaba Nahuatl languages in the test set evaluation. We find retrieval is highly language-dependent, beneficial only for large, in-domain corpora, and that synthetic data augmentation accounts for around 28 chrF++ of the dev set Guaraní performance gain. Our submission is the overall winner of the shared task, placing second out of five finalist submissions in human evaluations of target-language captions. Code and prompts are available on GitHub. 1 1 1 https://github.com/dhawan98/AmericasNLP2026-Gators-Submission 1 Introduction The AmericasNLP 2026 3 consists of generating culturally grounded captions for images in Indigenous languages of the Americas. We identify three major challenges. First, the target languages are low-resource. Second, the captions are culturally specific rather than generic visual descriptions. Third, the task requires not only lexical transfer, but also stylistic control. Successful systems must produce short, natural captions that match the reference register expected by the organizers. While the cultural image captioning task is new to AmericasNLP 2026, we find that current vision-language models (VLMs) are not able to directly caption images in the target languages. This is unsurprising given the limited training data available for the target languages. We therefore formulate the problem as a compound of image captioning in a high-resource language followed by machine translation from that language into one of the shared task targets, which mirrors the shared task’s baseline approach. We first attempt to build on the work of 7 by providing intermediate Spanish (Es) VLM captions to an mBART-based machine translation (MT) model. 7 show that synthetic parallel data and language-specific preprocessing improve low-resource Indigenous MT, including Es-Guaraní (Grn) translation. In the 2026 shared task, however, a standard neural machine translation pipeline proves insufficient. When we apply the existing Es-Grn translation model to Spanish intermediate captions produced from the dev images, the resulting captions are often fluent enough at the sentence level but do not match the target caption register. In other words, the performance bottleneck shifts from generic translation quality to domain adaptation and culturally grounded caption style. To address this mismatch, we move from sequence-to-sequence translation toward retrieval-augmented in-context translation with Large Language Models (LLMs). The core idea is simple. Instead of relying on a single model fine-tuned on mixed-domain low-resource language corpora, we retrieve Es-low-resource (LoRes) examples that are similar to the current caption and provide them as in-context exemplars. This design lets the decoder adapt its lexical choices and stylistic register at inference time. We evaluate multiple OpenAI and Gemini models, vary the number of retrieved training examples and development exemplars, and test prompt variants and retrieval heuristics. Our experiments lead to three main findings. First, among OpenAI models, many-shot direct translation outperforms both our mBART baseline and LLM post-correction, but gains are modest and highly sensitive to prompt composition. Second, Gemini 2.5 Flash 4 is dramatically stronger in Es-LoRes in-context translation than the GPT-family models. Third, development references are useful as in-context exemplars for matching the expected caption register, but they must be interpreted carefully because same-pool development prompting can inflate development-set scores. Our primary contributions are as follows. We present a retrieval-augmented LLM caption translation pipeline for low-resource cultural image captioning. We document an extensive set of negative and positive ablations, including model substitutions, prompt revisions, and development-exemplar hyperparameter searches. Finally, we highlight evaluation design as a central methodological issue for dev-conditioned in-context learning in low-resource captioning. 2 Background Machine translation for Indigenous languages of the Americas has advanced largely through the AmericasNLP shared tasks, which have established evaluation benchmarks and made parallel corpora available for many low-resource languages 12 ; 9 ; 10 . Strong systems in these shared tasks have typically relied on multilingual pretrained models such as mBART, M2M-100, or NLLB-200, often combined with synthetic data generation, multilingual transfer, and additional corpus collection 11 ; 18 ; 5 . A recurring pattern in this literature is that performance improvements come not only from larger models, but also from better alignment between model capacity, augmentation strategy, and the target domain. Character-based evaluation metrics are also especially relevant in this area. chrF and chrF++ are widely used for morphologically rich languages because they are more robust than BLEU to inflectional variation and spelling differences 14 ; 15 . The AmericasNLP 2026 organizers likewise use chrF++ as the first-stage ranking metric for the shared task. Finally, our work relates to the broader use of in-context learning for low-resource generation. Instead of relying exclusively on fixed model parameters after fine-tuning, retrieval-augmented prompting can adapt the decoder to the current example at inference time. In our setting, this is especially attractive because the target outputs are short and stylistically constrained: parallel demonstrations can serve not only as semantic guides, but also as direct evidence of the desired caption register. 3 Dataset and Methodology 3.1 Baseline System We compare our retrieval-augmented LLM pipeline against the Qwen3VL-8B 16 Captioning and Sheffield 2023 11 MT method, which is the stated baseline for the current shared task on image captioning. We apply this baseline to our submission results for dev set captioning in each of the applicable target languages and report percent improvement over the baseline scores. For MT in Guaraní captioning, we also compare against the mBART-based Es-Grn model from 7 , trained on curated and synthetic parallel data. This serves as the strongest conventional translation baseline in our pipeline and allows us to assess whether retrieval-augmented in-context generation improves over standard sequence-to-sequence translation. We evaluate several retrieval-augmented GPT-4-family ( 13 ) model variants, summarized in Table 2 : GPT-4o-mini, GPT-4.1-mini, GPT-4.1, along with the competing Gemini 2.5 Flash. Across these models, we vary the number of retrieved training examples r , development exemplars d , and prompting strategy. 3.2 Task Data The official development set \mathcal{D} (dev set) contains 50 examples, and the organizers release the data in JSONL format paired with images. The pilot set includes Spanish captions for reference, but the organizers explicitly note that these are pilot-only and will not be present in development or test. Our final submission pipeline is two-stage. Stage 1 produces a Spanish caption from the image using Qwen 2.5B 2 . Stage 2 utilizes Gemini 2.5 Flash 4 for Es-LoRes translation. 3.3 Datasets by Language The shared task covers five target languages: Guaraní, Yucatec Maya, Orizaba Nahuatl, Bribri, and Wixárika. As shown in Table 1 , the available retrieval data \mathcal{R} differs substantially across languages, which motivates treating the retrieval size r and the number of development exemplars d as inference-time hyperparameters that vary across target language submissions. Guaraní We use the largest retrieval bank in our setup. The retrieval Es-Grn bank of 53,183 pairs contains AmericasNLP 2023

Read the complete text