HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning
Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin, University of Science, Technology of China, Alibaba Group, National University of Singapore · 2025
Read HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning by Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin, University of Science, Technology of China, Alibaba Group, National University of Singapore (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11393.txt.
Introduction
Large language models can score well on commonsense questions yet still stumble when a question is phrased in a slightly different way. Xiaoyuan Li and co-authors test whether that strength is real by building HellaSwag-Pro, a bilingual benchmark made of 11,200 cases with seven kinds of question variants. To build it, they first created a Chinese version of HellaSwag with 12,000 finely annotated items across 56 categories, then ran wide experiments on 41 representative models.
The results show the models are far from robust at commonsense reasoning, and their weakness depends on which language they are tested in. The benchmark is offered as a stricter tool for measuring real understanding rather than memorized patterns.
Catalog record
| CREATOR | Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin, University of Science, Technology of China, Alibaba Group, National University of Singapore |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, Junyang Lin, University of Science, Technology of China, Alibaba Group, National University of Singapore Abstract Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses. Do these models truly understand commonsense knowledge, or just memorize expression patterns? To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning. We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants. To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories. We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning. Furthermore, this robustness varies depending on the language in which the LLM is tested. This work establishes a high-quality evaluation benchmark, with extensive experiments offering valuable insights to the community in commonsense reasoning for LLMs. Variant Type Context Choices Initial data A lady walks to a barbell. She bends down and grabs the pole. The lady A. stands and lifts the weight over her head. B. swings and lands in her arms. C. pulls the barbell forward. D. pulls a rope attached to the barbell. Problem restatement A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The woman A. rises and hoists the barbell above her head. B. swings and lands in her arms. C. pulls the barbell forward. D. pulls a rope attached to the barbell. Reverse conversion The lady stands and lifts the weight over her head. Which could be the most possible context for this action? A. A lady walks to a barbell. She bends down and grabs the pole. B. A lady positions herself at the squat rack. She lowers her body before rising steadily. C. A lady approaches the kettlebell set. She swings the weight forcefully between her legs. D. A lady stands beside the bench press station. She lies down and lifts the barbell from her chest. Causal inference A lady walks to a barbell. She bends down and grabs the pole. The lady stands and lifts the weight over her head. Which could be the most possible reason for this action? A. She is performing a weightlifting exercise. B. She is using the barbell as a decoration for an event. C. She is moving the barbell to a different location in the gym. D. She is cleaning the barbell after a workout session. Sentence ordering 1. She bends down and grabs the pole. 2. A lady walks to a barbell. 3. The lady stands and lifts the weight over her head. Which is the correct order? A. 2-1-3 B. 3-1-2 C. 2-3-1 D. 1-3-2 Scenario refinement A lady walks to a barbell. She bends down and grabs the pole. The lady hesitates for a moment, then changes her mind. Instead of lifting the barbell, she A. swings and lands in her arms. B. stands and lifts the weight over her head. C. pulls the barbell forward. D. pulls a rope attached to the barbell. Negative transformation A woman approaches a weightlifting bar. She lowers her body and grasps the metal rod. The lady will not A. swings and lands in her arms. B. stands and lifts the weight over her head. C. bend her knees and lift the barbell. D. adjust her grip and lift the weight. Critical testing A lady walks to a barbell. She bends down and grabs the pole. The lady suddenly realizes she forgot her weightlifting gloves and decides to postpone her workout. The lady A. stands and lifts the weight over her head. B. swings and lands in her arms. C. pulls the barbell forward. D. pulls a rope attached to the barbell. E. None of the above four options are suitable. Table 1: Examples of the seven variants we adopt for an initial question, with the correct answer unchanged as (A). Modifications are highlighted in different colors for clarity. 1 Introduction Commonsense reasoning is a crucial part of intelligence, involving contextual understanding, implicit knowledge, and logical deduction ( Liu and Singh, 2004 ; Cambria et al., 2011 ; Davis and Marcus, 2015 ). Recent studies have focused on enhancing these capabilities in LLMs, achieving impressive performance ( Yang et al., 2024 ; OpenAI et al., 2024 ; Team et al., 2024 ). However, even slight changes to questions can lead to incorrect responses from the same models. For instance, in binary commonsense questions, human naturally recognizes both correct and incorrect options through a single inference process, while LLMs, though able to identify the correct answer, struggle to reason about why the alternative is wrong Balepur et al. (2024) . Therefore, we ask the question: Does this high-level performance stem from a genuine understanding of commonsense knowledge, or is it simply a result of memorizing specific expression patterns in pre-training data? Figure 1: Comparison of GPT-4o’s responses to an original question and its several meaning-preserving variants. GPT-4o successfully handles the original question but struggles with its variants on the same knowledge. To answer this question, an effective approach is to systematically evaluate the robustness of LLMs in answering commonsense reasoning questions. As illustrated in Figure 1 , we find that GPT-4o correctly answers an original question but fails on its variants, i.e., questions about the same commonsense knowledge but in different reasoning forms, such as reverse conversion. This indicates that GPT-4o has not fully grasped the commonsense knowledge behind the question; a genuine understanding of commonsense knowledge should be able to generalize to these question variants. However, existing benchmarks do not yet support a thorough evaluation of LLM robustness in commonsense reasoning. Most work evaluates LLMs on general benchmarks Zellers et al. (2019) ; Talmor et al. (2019) ; Mihaylov et al. (2018a) , or in specific domains of commonsense knowledge Zhou et al. (2019) ; Qin et al. (2021) ; Bisk et al. (2020) . Although some efforts have considered the robustness of commonsense reasoning, they either focus on whether models can learn genuine question-answer correlations under initial questions Jia and Liang (2017) ; Branco et al. (2021) , or examine only one type of simplistic question variant such as question paraphrasing Zhou et al. (2021) ; Ismayilzada et al. (2023) ; Balepur et al. (2024) , lacking investigation into robustness across diverse and complex variants. To address this gap, we present the first extensive evaluation on the robustness of commonsense reasoning for LLMs, starting with dataset construction. Firstly, recognizing that existing benchmarks are predominantly in English, which limits the assessment of non-English LLMs ( Davis, 2023 ), we develop a Chinese commonsense reasoning dataset based on the widely-used HellaSwag benchmark Zellers et al. (2019) , containing 12,000 questions. Specifically, we design 56 fine-grained categories, and propose a two-stage data annotation method including initial dataset generation and difficult sample replacement. Secondly, we design and compile seven variants from existing studies ( cf. Table 1 ), which