MAR: Multi-Agent Reflexion Improves Reasoning Abilities in LLMs

Onat Ozer, Grace Wu, Yuchen Wang, Daniel Dosti, Honghao Zhang, Vivi De La Rue · 2025

Read MAR: Multi-Agent Reflexion Improves Reasoning Abilities in LLMs by Onat Ozer, Grace Wu, Yuchen Wang, Daniel Dosti, Honghao Zhang, Vivi De La Rue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20845.txt.

Introduction

Onat Ozer and five co-authors try to fix a failure of self-correction in language models. A model that reviews its own mistakes can improve, but reviewing itself repeatedly often loops: it repeats the same error while knowing it is wrong. The authors call this degeneration of thought.

Their method, MAR, replaces the lone reviewer with several debating personas that generate reflections together. Debate produced more varied feedback, and the acting model used it better. Scores reached forty-seven percent exact match on the HotPotQA questions and about eighty-three percent on HumanEval coding tasks. Both beat single-model reflection, suggesting that outside perspectives, even simulated ones, keep reasoning honest.

Catalog record

CREATOROnat Ozer, Grace Wu, Yuchen Wang, Daniel Dosti, Honghao Zhang, Vivi De La Rue
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20845

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAR: Multi-Agent Reflexion Improves Reasoning Abilities in LLMs Onat Ozer, Grace Wu, Yuchen Wang, Daniel Dosti, Honghao Zhang, Vivi De La Rue Abstract Large Language Models (LLMs) have shown strong capabilities in reasoning and program synthesis, yet enabling them to improve through experience remains a central challenge. Reinforcement learning offers one opportunity for agent improvement, but updating large models is computationally expensive and requires significant environment interaction. The Reflexion framework introduced a lightweight alternative in which an LLM agent generates natural language reflections about its failures and stores them as episodic memory that guides future attempts. Reflexion demonstrated meaningful gains on multi hop question answering and code generation without modifying model parameters. However, our replication shows that the single agent design of Reflexion leads to systematic shortcomings. The same model generates actions, evaluates its own behavior, and produces reflections, which often results in repeated reasoning errors, confirmation bias, and limited corrective feedback. To address these limitations, we reproduce Reflexion on HotPotQA and HumanEval using GPT 3.5 and GPT 4, and we analyze failure modes through detailed logging of trajectories and reflections. We consistently observe that self reflections tend to repeat earlier misconceptions and do not introduce new reasoning paths on difficult examples. Motivated by these findings, we introduce Multi Agent Reflexion (MAR), a structured multi agent extension that incorporates diverse reasoning personas and a judge model that synthesizes their critiques into a unified reflection. MAR separates the processes of acting, diagnosing, critiquing, and aggregating, which reduces shared blind spots and prevents the repeated reinforcement of earlier mistakes. On both benchmarks, MAR improves directly over Reflexion. MAR increases HotPotQA exact-match accuracy by 3 points, from 44 to 47, and improves HumanEval pass@1 from 76.4 to 82.6, a 6.2-point gain. MAR also reduces the stagnation observed in single agent Reflexion, indicating that multi agent interaction offers a simple and effective path toward more reliable LLM agents. Introduction Large Language Models (LLMs) has evolved to generate strong reasoning traces and high-quality code, but enabling them to improve through experience remains an open problem. Traditional reinforcement learning offers one mechanism for agent improvement, but updating billion-parameter policies is computationally costly and data-intensive. In-context learning ( 1 ) , where an LLM adapts using examples directly in the prompt, offers one path toward improving model performance without additional training. Building on this idea, prompting strategies such as chain-of-thought (CoT), expert-style prompting, and ReAct have been shown to enhance LLM reasoning and decision-making. Reflexion ( 11 ) integrates these advances while adding a mechanism for the model to self-reflect on its own mistakes: after each attempt, the agent converts feedback into natural-language guidance for the next trial, analogous to how reinforcement learning algorithms use reward signals to update a policy ( 14 ) . Reflexion reports substantial gains over baseline LLM and CoT/ReAct agents on HotPotQA ( 17 ) and HumanEval ( 2 ) , improving exact-match accuracy by roughly 20 points on HotPotQA and pass@1 accuracy by 11 points on HumanEval. We focus on these benchmarks for replication since multi-hop question (questions where a model must combine information across several steps of retrieveal) answering and program synthesis are the two high-impact domains where iterative self-improvement most meaningfully enhance LLM agent performance, and covers the majority of use cases. Despite the gains of the Reflexion framework, in our replication, we consistently observed that Reflexion is vulnerable to degeneration-of-thought ( 8 ) , where the agent repeats the same flawed reasoning across iterations even when explicit failures are identified. To mitigate this limitation, we introduce Multi-Agent Reflexion (MAR) , a framework that replaces single-agent self-critique with a structured debate among diverse persona-based critics. These multiple critics generate richer reflections that more effectively guide the agent toward improved strategies. Our full codebase, including replication scripts, logging instrumentation, and the Multi-Agent Reflexion implementation, is available at: https://github.com/danpro1011/reflexion . Related Work Recent research explores improving LLM reasoning through self-generated feedback, iterative refinement, and multi-agent interaction. Self-critique methods such as Self-Refine and Reflexion show that verbal feedback can improve performance without updating model weights, though they often rely on a single agent with fixed reasoning patterns. Multi-agent debate introduces structured disagreement and persona diversity, helping reduce shared blind spots; our approach builds on this idea by integrating diverse critic personas into an episodic memory framework that supports more stable and transferable improvements across tasks. Self-Generated Feedback and Iterative Refinement: Several works explore letting LLMs critique and revise their own outputs. Self-Refine, for example, iteratively generates output, asks the model to provide targeted feedback, and then uses that feedback to produce a revised answer ( 10 ) . Multi-Agent Debate and Divergent Reasoning: Multi-agent debate methods instantiate multiple LLM agents that propose answers, critique each other’s reasoning, and then aggregate the resulting arguments into a final decision ( 4 ) . Recent work shows that structured disagreement can improve factual accuracy and robustness on challenging reasoning benchmarks by promoting divergent reasoning and reducing shared blind spots ( 8 ) . Other multi-agent frameworks introduce persona-driven or role-based heterogeneity among agents and coordinate their outputs through majority voting or a centralized judge model ( 7 ) . Chain-of-Thought and Tool-Augmented Reasoning: Chain-of-thought (CoT) prompting shows that LLMs can solve substantially harder reasoning problems when encouraged to generate intermediate natural-language steps before answering ( 16 ) . CoT is purely a single-pass prompting strategy. The model produces one reasoning trace and does not explicitly learn across episodes. ReAct extends this idea by interleaving reasoning tokens with actions, so that an LLM can plan, call tools or APIs, and react to new observations in text-based environments ( 18 ) . Background Reflexion Framework Reflexion ( 11 ) wraps a language model in a simple loop that turns past failures into natural-language guidance for future attempts. Conceptually, the framework decomposes the agent into three prompted roles plus an episodic memory. Actor ( M_{a} ). The Actor is the LLM prompted to behave as the agent’s policy. It generates actions, intermediate reasoning, and final answers. Reflexion is compatible with multiple prompting styles; the original paper focuses on: • Chain-of-Thought (CoT) ( 16 ) prompting, which encourages structured step-by-step reasoning. • ReAct ( 18 ) , which interleaves reasoning steps with explicit actions taken in an external environment. Evaluator ( M_{e} ). The Evaluator judges whether an Actor trajectory successfully solves the task. Depending on the domain, M_{e} may be another LLM prompt, a heuristic rule, or an external execution environment such as code unit tests. Importantly, Reflexion only requires a coarse signal, such as success/failure, not a dense numeric reward. Self-Reflector ( M_{sr} ). When the Evaluator indicates failure, the Self-Reflection model converts this outcome into verbal feedback. Given the failed trajectory and its evaluation, M_{sr} produces natural-language reflections that diagnose what went wrong, explain why the reasoning failed, and suggest strategies to try the next attempt. These “verbal rewards” play the role of a lightweight optimization signal. Figure 1: The Reflexion Architecture. Actor performs an action on the environment. The evaluator gives feedback to reflector LLM, with feedback stored in short-term memory. A new action is then performed starting a new iteration. At a high level, each failed attempt produces a new reflection that is added

Read the complete text