When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation CCS: Computing methodologies Natural language processing CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence

Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas · 2025

Read When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation CCS: Computing methodologies Natural language processing CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence by Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16809.txt.

Description

Amirkia Rafiei Oskooei and three co-authors test whether flooding a language model with examples helps it translate code between programming languages. Modern models accept huge prompts, so many-shot prompting, with hundreds of worked examples, looks attractive. The team ran over ninety thousand translations to check that assumption.

Example counts ran from none to six hundred twenty-five, with prompts reaching about eight hundred thousand tokens. Surface similarity scores rose slightly with more examples. Working correctness, though, peaked with only five to twenty-five examples and often fell as examples piled up. The authors call this a many-shot paradox. Their advice: a few well-chosen examples beat sheer volume for code translation.

Catalog record

CREATORAmirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16809

Browse the full library · About the Tower of Babel