AI reasoning effort mirrors human decision time on content moderation tasks

Thomas R. Davidson · 2025

Read AI reasoning effort mirrors human decision time on content moderation tasks by Thomas R. Davidson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20262.txt.

Introduction

Thomas Davidson compares how long people take to judge online posts with how much work chat models do before they answer. His 2025 study uses a content moderation task, the kind of check done on social media posts. The model side is gauged by the length of the steps it writes out before it rules.

Across three top models, longer written steps went with longer human choice times. Both sides also slowed down on the same hard cases, even when key facts were held fixed. The written steps named more of the facts around a post when the case was hard. Davidson says such traces may help explain model choices.

Catalog record

CREATORThomas R. Davidson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20262

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI reasoning effort mirrors human decision time on content moderation tasks Thomas R. Davidson Abstract Large language models can now generate intermediate reasoning steps before producing answers, improving performance on difficult problems. This study uses a paired conjoint experiment on a content moderation task to examine parallels between human decision times and model reasoning effort. Across three frontier models, reasoning effort consistently predicts human decision time. Both humans and models expended greater effort when important variables were held constant, suggesting similar sensitivity to task difficulty and patterns consistent with dual-process theories of cognition. These findings show that AI reasoning effort mirrors human processing time in subjective judgments and underscores the potential of reasoning traces for interpretability and decision-making. Chain-of-thought prompting, where large language models (LLMs) are instructed to think step-by-step before producing an answer, leads to performance improvements on diverse tasks wei_chain–thought_2022 . The latest frontier models are optimized to perform this intermediate reasoning before responding via reinforcement learning ( lightman_lets_2023 ; yao_tree_2023 ) . Computer scientists have drawn analogies to dual process theories of cognition: earlier LLMs responded immediately by generating the most likely token sequence (System 1), whereas large reasoning models (LRMs) “think” before answering (System 2) kojima_large_2022 ; yao_tree_2023 . Beyond the performance gains, the reasoning traces produced by LRMs, which represent the “thinking out loud” performed by the models, could enhance interpretability and aid AI safety efforts korbak_chain_nodate . Questions remain regarding the utility of reasoning traces and whether they mimic human cognitive processes. LRMs perform well on benchmarks that measure outcomes, but there is little evidence that the reasoning effort corresponds to human cognitive effort. Reasoning traces can also be unfaithful, obscuring information used to make decisions turpin_language_2023 ; chen_reasoning_2025 . This unfaithfulness is concerning for applications involving consequential decisions due to persistent biases in how language models process information ( hofmann_ai_2024 ; bai_explicitly_2025 ) . This study uses an experimental approach to compare LRMs and human subjects on a challenging, subjective task: comparing synthetic social media posts to determine which one is more likely to violate a content moderation policy. The study uses a conjoint experiment ( hainmueller_causal_2014 ; hainmueller_validating_2015 ) , but rather than quantifying the effects of post attributes, as in prior work on content moderation ( kozyreva_resolving_2023 ; solomon_illusory_2024 ) , it focuses on the reasoning effort expended by LRMs and the time taken by human subjects to make decisions. If LRMs mirror human effort, we should expect a positive association between the two. To identify parallels in reasoning across scenarios, we analyze how effort varies according to post attributes. We expect that it is easier to decide when important attributes are distinct (e.g., the posts use different types of offensive language) since people can use heuristics and satisfice to make quick decisions ( tversky_judgment_1974 ; krosnick_response_1991 ; hainmueller_validating_2015 ) . When important attributes are held constant, more careful consideration of second-order factors is necessary, engaging System 2 style thinking. To gain further insight into AI reasoning, we also descriptively analyze patterns across different stimuli and conduct qualitative analyses of the reasoning traces. Results Two examples of the posts used in the study are shown in Figure 1 . The results are obtained by comparing nearly 30k randomly sampled pairs of posts. We begin by evaluating the relationship between human decision time and AI reasoning effort. We compare three of the most advanced LRMs: OpenAI’s o3, Google’s Gemini 2.5 Pro, and xAI’s Grok 4. Each automatically performs reasoning and can handle text and image-based inputs. Figure 2 A shows a positive association between reasoning effort, measured by the number of reasoning tokens consumed, and the time spent by human subjects on each task. The effects are similar across all three models tested. A standard deviation increase in reasoning effort is associated with just over one second additional time for human subjects. For example, a 1SD increase in reasoning effort for Grok was associated with 1.24 additional seconds ( t=15.6,p<0.001 ). Figure 2 B shows how each outcome varies when both posts use the same slur. Human subjects take, on average, 4.5 seconds longer ( \hat{\beta}=4.47,t=18.5,p<0.001 ) to make a choice, corresponding to about 40% of the median decision time. Across all three models, identical pairs require over one standard deviation more reasoning tokens, ranging from 1.06 SD for o3 ( t=66.1,p<0.001 ) to 1.15 for Gemini ( t=71.2,p<0.001 ). While the standardized estimates are comparable, the token usage varies considerably, corresponding to approximately 60% of the median token usage for Gemini, 100% for o3, and 280% for Grok. Figure 1: Examples of the synthetic posts used in the study. Each conjoint task requires choosing which post is more likely to violate a policy from a randomly sampled pair. Figure 2: (A) There is a positive association between human decision time and the number of reasoning tokens consumed across all three models. (B) Human subjects take longer to click on conjoint tasks where both posts contain equally offensive terms, and all three reasoning models consume more tokens when making decisions. All error bars represent 95% confidence intervals. Figure 3: Each heatmap shows the median response for each pair of terms shown in the conjoint task. (A) shows represents the median time in seconds for a subject to click on an answer, and (B,C,D) the median number of reasoning tokens consumed for the specified models. All four plots show evidence of clustering along the diagonal, indicating that decisions require more effort when offensiveness is either constant or similar. The heatmaps are symmetric because the medians are calculated irrespective of the order in which the pair was shown. Figure 3 depicts the median decision time and reasoning effort for each pair of terms across all conjoint tasks. In all four heatmaps, the density is highest either on or immediately adjacent to the diagonal, meaning that decisions require the most effort when terms are either identical or similar in offensiveness. Human subjects take the most time to make decisions when presented with two posts without offensive terms or that use anti-White language. Their choices are fastest when shown incongruous pairs, such as homophobic or racist terms coupled with innocuous posts. Similar patterns are evident among the three LRMs, with some model-specific idiosyncrasies: o3 expends the most reasoning power when shown pairs of sexist or anti-White posts, or homophobia and a reclaimed slur; Gemini consumes the most tokens when comparing posts both featuring a generic insult; and Grok uses the most reasoning effort when evaluating pairs of anti-White posts. All three models tend to mirror human subjects when considering incongruous pairs. Finally, we conducted a qualitative analysis of the reasoning traces from Gemini 2.5 Pro, the only model to provide this data at the time of the study. When both posts are equally offensive, the traces typically state that both violate the content policy before turning to additional contextual factors. In some instances, the models focus on the language, often prioritizing personal attacks rather than statements about groups. In other cases, the traces emphasize factors including the user identity, discussion topic, and engagement metrics. Discussion The results reveal parallels in how humans and AI reason about content moderation decisions. The positive association between human decision time

Read the complete text