RESPONSE : Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

Aïssatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller · 2025

Read RESPONSE : Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation by Aïssatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11348.txt.

Description

Aissatou Diallo and co-authors present RESPONSE in 2025, a benchmark for language model reasoning in disaster situations. Commonsense reasoning about crises is a distinct challenge for these models. The data set is human-curated and partly validated by environmental engineers for realism.

It holds close to eighteen hundred instances with more than six thousand question sets. Items cover problem descriptions, missing resources, and time-sensitive solutions with justifications. Even a top model reached only about thirty-seven percent correctness on immediate response actions by human judgment. The authors see large room for improvement in crisis reasoning.

Catalog record

CREATORAïssatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11348

Browse the full library · About the Tower of Babel