M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation

Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu · 2024

Read M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation by Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20127.txt.

Description

Zhaopeng Feng and a team of co-authors let language models debate each other to grade machine translations. Their framework appeared in twenty twenty-four. Single-model judges have lagged behind trained automatic metrics, despite sounding human.

The system, M-MAD, splits the standard human checklist into separate dimensions first. Specialist agents then argue each dimension among themselves, pooling their reasoning. A final synthesis merges the dimension scores into one verdict. Across test sets, the debate beat every earlier language-model judge. It even rivaled the best reference-based metrics while running on a modest model. Ablations credit the design, not luck. Code and data are public for other teams to stress-test.

Catalog record

CREATORZhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20127

Browse the full library · About the Tower of Babel