Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis

Greta Dolcetti 1 , Vincenzo Arceri 2 , Eleonora Iotti 2 , Sergio Maffeis 3 , Agostino Cortesi 1 , and Enea Zaffanella 2 · 2024

Read Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis by Greta Dolcetti 1 , Vincenzo Arceri 2 , Eleonora Iotti 2 , Sergio Maffeis 3 , Agostino Cortesi 1 , and Enea Zaffanella 2 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14841.txt.

Description

Greta Dolcetti and co-authors build in 2024 a loop that lets language models fix their own code using test results and static analysis. Developers now ask models for code daily, which speeds work but can plant bugs, smells, and security holes. Past studies showed mainstream models often emit unsafe code.

Their framework grades each draft by running tests and analyzers, then feeds the complaints back to the model as guidance for a rewrite. The study used general purpose open source models, not tuned coding giants. Iterated feedback steadily lifts the share of code that passes. Such loops could sit inside editors, quietly hardening generated code before it ships. The authors report where the gains stop.

Catalog record

CREATORGreta Dolcetti 1 , Vincenzo Arceri 2 , Eleonora Iotti 2 , Sergio Maffeis 3 , Agostino Cortesi 1 , and Enea Zaffanella 2
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14841

Browse the full library · About the Tower of Babel