When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation CCS: Computing methodologies Natural language processing CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence
Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas · 2025
Read When Many-Shot Prompting Fails: An Empirical Study of LLM Code Translation CCS: Computing methodologies Natural language processing CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence by Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16809.txt.
Description
Amirkia Rafiei Oskooei and three co-authors test whether flooding a language model with examples helps it translate code between programming languages. Modern models accept huge prompts, so many-shot prompting, with hundreds of worked examples, looks attractive. The team ran over ninety thousand translations to check that assumption.
Example counts ran from none to six hundred twenty-five, with prompts reaching about eight hundred thousand tokens. Surface similarity scores rose slightly with more examples. Working correctness, though, peaked with only five to twenty-five examples and often fell as examples piled up. The authors call this a many-shot paradox. Their advice: a few well-chosen examples beat sheer volume for code translation.
Catalog record
| CREATOR | Amirkia Rafiei Oskooei, Kaan Baturalp Cosdan, Husamettin Isiktas, Mehmet S. Aktas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16809