RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?
Yuyang Dai, Yan Lin, Zhuohan Xie, Yuxia Wang
Abstract
Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce RealFin , a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.
1
Introduction
A central requirement of reliable reasoning in high-stakes domains is not only answering correctly when a task is well-posed, but also recognizing when the available information does not justify a determinate conclusion.
In real-world financial problems, key premises needed for decision-making, such as assumptions, constraints, time horizons, or applicable regulatory and accounting standards, are often missing or unspecified in users’ initial requests
(
7
)
.
When this happens, the difficulty is not computation, but assessing whether any conditions are missing to forward a valid reasoning process.
This situation is common in complex financial practice and professional training, where problems are rarely well defined from beginning. They appear solvable but are logically underdetermined
(
5
)
. In such cases, committing to a specific assumption is unjustified, since multiple interpretations lead to different conclusions. A rational professional response is to withhold commitment until the missing conditions are clarified
(
7
)
.
Despite its practical importance, examination of such capabilities is largely absent in current financial evaluations.
Most financial benchmarks adopt an implicit closed-world assumption that every question is fully specified and admits a single correct answer
(
14
)
.
Consequently, benchmark design focuses on whether a model can select the correct option given complete premises, while excluding cases where missing conditions invalidate the act of answering, or no correct answers among the given options.
Systems such as FinEval, FinBen, FinMaster, and the Open FinLLM Leaderboard follow this paradigm, limiting their ability to assess whether models can identify under-specified questions or models confidently answer unsolvable questions
(
13
;
22
)
.
To address this gap, we introduce
RealFin
, a bilingual financial reasoning benchmark that explicitly evaluates models with incomplete problems.
Starting from professional, well-posed exam-style questions, we construct paired counterparts by removing logically necessary premises while preserving coherence and realism
(
8
)
. The resulting condition-missing questions remain linguistically plausible but underdetermined, so any concrete solution requires clarifying unstated assumptions.
Figure 1:
Original question and its revision, with GPT-5.1-mini predictions and confidence,
bold correct
option.
Overall, we design three task formulations:
(i) Original:
full-condition questions with a unique justified answer;
(ii) Revised:
condition-missing questions require further clarification;
and
(iii) None-of-the-above:
replacing the correct answer in multi-choice question answering with none-of-the-above (NOTA), which forces a model to genuinely infer correct answer, rather than pattern recognition.
Our contributions are as follows:
•
We propose to evaluate models’ financial reasoning with under-specified questions, assessing whether a model will over-commit and confidently answer an unsolvable question.
•
We manually curate a bilingual financial reasoning dataset in English and Chinese, with in total of 2,020 questions spanning full-condition and condition-missing variants.
•
We evaluated 10 LLMs including five commercial LLMs and five finance-specialized LLMs, revealing systematic performance degradation when conditions are missing or no options are correct for open-source models.
2
Related Work
Financial Benchmarks
Prior evaluations in the financial domain focused on information extraction and sentiment analysis
(
1
)
, as well as short-horizon prediction
(
26
)
, where evidence is typically assumed to be local and fully specified.
More recent work extends financial QA toward numerical and long-form reasoning over reports, as in FinQA
(
4
)
and TAT-QA
(
33
)
, and toward multi-task settings that integrate retrieval and multi-step reasoning, including PIXIU
(
24
)
, FinBen
(
23
)
, EMS
(
12
)
and FinChain
(
25
)
.
Beyond QA, simulation-based evaluations further explore strategic behavior in trading
(
8
)
, auctions
(
11
)
, and pricing games
(
2
)
.
Despite advances, these benchmarks assume that financial problems are fully specified and admit a unique correct answer, offering limited insight into
how models behave when critical conditions are missing
.
What still Challenges LLMs in Finance?
Recent studies show that current language models still struggle with many core financial tasks.
For problems that require precise numerical prediction or strict rule application (e.g., credit rating, risk assessment), traditional statistical and rule-based methods often outperform LLMs
(
6
)
.
Many models remain difficult to interpret and control in ways that satisfy regulatory expectations
(
20
)
.
9
shows that model performance drops when financial problems become uncertain or change over time.
LLMs are also sensitive to how questions are phrased. They change their behaviors when prompts are slightly modified, especially in strategic or incentive-driven settings
(
8
)
. Moreover, requests combining text with tables, charts, or time-series data remain challenging for most models
(
10
)
.
Research that treats LLMs as decision-making agents further finds that models can behave inconsistently or make choices that do not align with basic economic reasoning when incentives are involved
(
32
;
18
)
.
These hard cases guide the scope of our data collection: selecting challenging scenarios and samples.
Specialised Financial Language Models
The rise of financial AI has led to a proliferation of specialist models. Early pioneering work, such as BloombergGPT
(
21
)
and Pixiu
(
24
)
, established the effectiveness of large-scale domain-specific pre-training and multi-task instruction tailoring. Several open-source models, including FinGPT
(
26
)
, DISC-FinLLM
(
3
)
, and CFGPT
(
15
)
, as well as decision-orientated systems like InvestLM
(
27
)
, have since emerged.
Recently, the focus has switched to reasoning-enhanced architectures. New models, such as Fin-o1
(
19
)
, Fin-R1
(
17
)
, and Dianjin-R1
(
34
)
, use large-scale RL and Chain-of-Thought architectures to solve difficult financial logic. While these models represent the cutting edge of financial issue solver, they are still generally assessed on “closed-system” standards, which presume that a valid answer exists at all times. Leaderboards, such as the Open FinLLM Leaderboard
(
16
)
, test performance on fully stated tasks, but rarely evaluate on condition-missing problems.
Our work differs from previous studies by focusing on more practical scenarios. Instead of asking how many financial tasks a model can solve, we ask whether the model can tell when a question does not contain enough information to be answered correctly.
3
Dataset
We introduce a financial reasoning dataset designed to capture both the breadth and difficulty of real-world tasks.
3.1
Taxonomy
Question Types
We include six types of questions with incremental difficulty to gradually evaluate model capabilities, from understanding, application, to reasoning, judgment, and decision-making.
(i) Financial Term Explanation and Conceptual Understanding
questions require understanding the meaning of financial terminology, e.g., identifying what “rolldown return” or “cost basis” means in a bond or tax context.
(ii) Simple Calculation
questions typically require first understanding the problem, and then performing basic arithmetic. For example, computing interest income given a coupon rate and face value.
(iii) Complex Calculation
involves multiple steps computations using more complex formulas. For instance, compute a bond’s total return by jointly considering price change, coupon income, and yield movements over time.
(iv) Summary and Comprehensive Judgment
involve consolidating information from multiple sources and aspects, and then making a informed conclusion based on the information. For example, decide whether a firm’s financial position has improved based on changes in cash flow, leverage, and profitability.
(v) Knowledge Transfer and Application
questions require models to transfer understanding from theory to practical situations, e.g., applying depreciation rules learned in accounting theory to a specific asset purchase case.
(vi) Statistical and Econometric Methods
use statistical or econometric approaches to analyze data. A case like interpreting regression coefficients, hypothesis test results, or volatility estimates in a financial context.
Financial Topics
From the perspective of fine-grained financial topics, we classify English questions into nine sub-domains including equity investment, fixed income investment, quantitative analysis, derivatives investment, financial statement analysis, economics, corporate finance, portfolio management, and other investments. Chinese questions follow five institutionally grounded CPA categories: auditing, accounting, tax law, economic law, and wealth management.
3.2
Full-condition Data
Data Source and Collection
The full-condition dataset consists of 2020 financial reasoning questions in English and Chinese. The two raw data sources are released under permissive licenses that allow their use for academic research.
English questions are curated from publicly available CFA-style preparatory materials, excluding real examination content.
Chinese questions are collected from publicly accessible CPA-style instructional materials, also excluding real exam items.
During questions selection, we prioritize challenging cases identified in prior work (Section
2
).
Most selected questions typically require a combination of conceptual understanding, multi-step reasoning, information integration. We choose these question types because they expose model behaviors that purely factual questions cannot reveal, such as whether a model truly understands the problem structure, can reason across multiple constraints, or can recognize when the given information is insufficient to support a valid answer.
Afterwards, each question is manually reviewed to ensure quality, and tagged with both a sub-domain label (e.g.,
equity investment
) and a question type label indicating what capabilities are evaluated by this question (e.g.,
knowledge transfer and application
).
Quality Control
To ensure cross-lingual consistency and accuracy, we recruited two volunteer annotators who have education background in finance and AI research experience, with one female PhD and one male undergraduate student. Both are native Chinese speakers with study experience abroad and are fluent in English. Annotators were first trained and then asked to review and label using the guideline that defines the scope of each category.
3.3
Condition-Missing Data
Each condition-missing question is derived from a full-condition instance by manually removing one or more assumptions that are required to determine a unique answer. As a result, the revised question still appears well-formed, but no longer provides enough information to justify a single correct conclusion. In practice, we removed four types of conditions.
(i) Initial assumptions:
missing the high-level economic backdrop that the system takes for granted. (macro assumptions: interest-rate regime, inflation expectations, central bank forward guidance, and going-concern assumption).
(ii) Intermediate linking conditions:
missing the rules/models that turn the given inputs into the required output. (linking method: valuation model such as CAPM/DCF; definition of valuation multiples; hedging logic; depreciation/amortisation method)
(iii) Constraint-defining information:
missing safety guardrails and system boundaries within financial contracts. (red lines: financial covenants, seniority/recourse attributes, and related-party identifiers)
(iv) Standard-selection cues:
missing the governing rulebook that determines how to interpret and record the same transaction. (standards: IFRS vs GAAP; regulatory framework; revenue recognition timing; AML/CFT rating rules; taxable temporary differences)
To ensure that the revised questions remain valid and realistic, two annotators first revise the questions by removing some conditions independently. Then, they cross-validate the revised question by assessing
(i)
whether the questions still make sense within the financial context and
(ii)
whether they lead to situations where no unique answer can be derived. Discrepancies are resolved through discussions to maintain the integrity of the task design.
Reasoning Category
English
Chinese
Total
Ori.
Rev.
Ori.
Rev.
Financial Concepts
62
43
119
110
334
Simple Calculation
130
79
141
53
403
Complex Calculation
85
64
52
40
241
Summary & Judgment
161
136
127
99
523
Knowledge Transfer
96
88
87
73
344
Statistical / Econometric
70
47
48
10
175
Total
604
457
574
385
2020
Table 1:
Statistics of dataset across six question types in two languages and two task formulations: full- vs. missing- conditions (
Ori
ginal vs.
Rev
ised).
Statistics
Table
1
summarizes the distribution of questions across six categories for both languages and two task formulations. There are a total of 1,062 English examples (604 original and 457 revised) and 959 Chinese cases with 574 original and 385 revised.
Summary and Comprehensive Judgment
constitutes the largest category, reflecting our emphasis on context-dependent reasoning.
Simple Calculation
also accounts for a substantial portion, aiming to evaluate model’s math application in financial domain.
4
Experiments
We evaluate 10 language models including five general-purpose models and five finance-specific models, spanning both closed-source and open-source models.
Specifically, for general-purpose models, we use
GPT-5.1-mini
,
Gemini-2.5-Flash
,
Claude-Sonnet-3.5
,
DeepSeek-V3
,
Qwen3-Max
; for finance-specific models, we use
XuanYuan3-70B
,
Fin-R1-7B
,
CFGPT2-7B
,
DISC-FinLLM-13B
,
FinGPT-7B
. See more details in Table
6
.
1
1
1
Cells “–” in Tables
2
and
3
are explained in Appendix
C.1
.
4.1
Experimental Setups
All evaluations are performed in a zero-shot setting, without any in-context demonstration examples.
Prompts
For both full- and missing-condition questions, all models are instructed to answer using the same prompt: “
You are an expert in financial problem solving. You will be given a single- or multi-choice question. Please return a STRICT JSON object with three keys: reason, answer, and confidence.
” Figures
3
and
4
show the specific prompt for Chinese and English respectively.
For questions with a
None of the Above
option, models are triggered by specific instructions within JSON templates to identify instances of insufficient information: “
Briefly explain why these options are correct
”. Models are asked to choose it only when none of the other options can be justified by the question.
Decoding Configuration
All experiments are conducted with a fixed temperature of 0 (
do_sample=False
) to eliminate randomness. For all offline models, we set
max_new_tokens=1024
and use each model’s native chat template. More detailed in Appendix
C.2
and
C.5
to
C.9
.
4.2
Results
General Models
Financial Models
GPT-5.1-m
Gem-2.5
Cl-3.5
DS-V3
Qwen3
XY-70B
Fin-R1
CFGPT2
DISC
FinGPT
Question Type
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
English CFA
Conceptual
82.14
90.12
80.23
89.45
79.87
91.23
75.12
87.98
80.45
90.56
77.42
79.07
69.35
76.74
56.45
76.74
8.06
18.60
1.61
6.98
Simple Calc.
78.56
87.23
77.89
86.12
76.34
88.76
72.45
85.34
77.23
87.89
36.67
82.12
50.00
82.12
30.00
68.16
23.33
20.67
6.67
2.79
Complex Calc.
75.23
86.45
74.12
85.78
73.89
87.12
70.67
84.23
74.56
86.34
33.33
100.0
26.67
100.0
40.00
100.0
6.67
25.00
0.00
0.00
Comp. Judg.
81.45
89.78
79.67
88.23
78.23
90.12
74.56
86.89
79.12
89.45
62.38
88.37
57.43
84.88
60.40
80.23
22.77
22.09
1.98
4.65
Knowledge App.
79.87
88.45
78.12
87.12
77.56
89.34
73.23
85.67
78.45
88.78
40.70
75.00
38.37
75.00
31.40
75.00
10.47
12.50
0.00
0.00
Stats Methods
83.45
91.67
81.23
90.12
80.56
92.34
76.89
88.45
81.67
91.23
71.43
93.62
61.43
93.62
48.57
76.60
2.86
14.89
4.29
2.13
All
80.59
89.01
78.99
88.19
77.81
89.62
73.55
86.34
78.42
88.83
58.47
84.74
54.10
83.65
47.54
73.57
13.11
19.89
2.19
3.54
Chinese CPA
Conceptual
82.34
69.12
75.23
86.45
71.12
82.34
82.89
81.67
78.12
93.89
10.92
80.00
7.56
90.00
2.52
40.00
4.20
30.00
–
–
Simple Calc.
80.67
67.89
73.45
84.12
69.34
80.12
81.23
79.89
76.45
91.23
9.76
64.15
7.32
71.70
4.88
49.06
4.88
15.09
–
–
Complex Calc.
79.23
66.45
72.12
82.78
68.12
78.89
80.45
78.23
75.23
89.67
0.00
–
0.00
–
0.00
–
0.00
–
–
–
Comp. Judg.
81.45
68.56
74.23
85.12
70.23
81.23
81.89
80.56
77.12
92.45
10.34
71.72
5.75
72.73
2.30
49.49
5.75
20.20
–
–
Knowledge App.
80.12
67.23
73.12
83.45
69.12
79.67
81.12
79.12
76.23
90.89
0.00
66.67
14.29
66.67
0.00
0.00
14.29
0.00
–
–
Stats Methods
82.67
69.89
75.67
86.89
71.45
82.89
83.12
82.12
78.56
94.23
16.67
60.00
10.42
70.00
4.17
80.00
2.08
30.00
–
–
All
80.81
68.45
73.42
84.80
69.37
80.46
81.06
80.00
76.32
92.00
11.18
69.14
7.57
73.14
2.96
49.71
4.61
19.43
–
–
Table 2:
Accuracy (%) across six question types
for English CFA (top) and Chinese CPA (bottom).
O
= original;
R
= revised. For each row, best original is in
Best-O
,
Best-R
,
Worst-O
,
Worst-R
; best per row is
bolded
.
General Models
Financial Models
GPT-5.1-m
Gem-2.5
Cl-3.5
DS-V3
Qwen3
XY-70B
Fin-R1
CFGPT2
DISC
FinGPT
Topic
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
English CFA
Corp. Finance
82.50
90.00
91.25
100.0
81.88
90.00
82.19
90.00
83.75
91.00
60.00
90.00
50.00
90.00
60.00
90.00
30.00
40.00
0.00
0.00
Derivatives
80.00
88.00
87.20
96.00
83.68
92.00
80.08
88.00
82.80
90.00
53.33
92.00
53.33
88.00
31.11
76.00
20.00
28.00
6.67
4.00
Economics
81.82
90.00
81.82
90.00
81.82
90.00
81.82
90.00
81.82
89.09
53.33
80.00
53.33
60.00
53.33
60.00
13.33
0.00
0.00
0.00
Equity Inv.
79.49
88.46
79.49
88.46
79.49
88.46
76.28
84.62
78.85
87.18
57.63
86.54
62.71
86.54
59.32
78.85
20.34
13.46
1.69
3.85
FSA
79.66
88.89
79.66
88.89
79.66
88.89
79.66
88.89
79.66
89.63
65.12
88.89
39.53
81.48
53.49
66.67
11.63
22.22
2.33
3.70
Fixed Income
71.43
78.57
71.43
78.57
71.43
78.57
71.43
78.57
71.43
76.79
42.86
78.57
32.65
85.71
36.73
57.14
14.29
14.29
0.00
0.00
Other Inv.
79.75
88.89
77.78
86.42
79.75
88.89
76.54
85.19
78.02
86.42
62.77
86.42
57.45
87.65
50.00
77.78
10.64
23.46
3.19
3.70
Portfolio Mgt
90.91
100.0
90.91
100.0
90.91
100.0
90.91
100.0
90.91
100.0
75.00
100.0
75.00
100.0
75.00
100.0
0.00
33.33
0.00
33.33
Quant Analysis
80.00
88.89
80.00
88.89
82.09
91.11
78.21
86.67
78.61
87.78
65.96
86.67
72.34
88.89
42.55
73.33
0.00
6.67
0.00
2.22
All
80.59
89.01
78.99
88.19
77.81
89.62
73.55
86.34
78.42
88.83
58.47
84.74
54.10
83.65
47.54
73.57
13.11
19.89
2.19
3.54
Chinese CPA
Accounting
33.33
36.67
75.76
83.33
72.73
80.00
66.67
73.33
69.32
85.00
4.49
63.33
6.74
66.67
2.25
60.00
4.49
26.67
–
–
Auditing
84.85
93.33
90.91
100.0
87.88
96.67
84.85
93.33
88.64
97.50
25.76
96.67
13.64
83.33
7.58
60.00
4.55
26.67
–
–
Economic Law
66.28
72.97
78.57
86.49
76.19
83.78
81.08
89.19
73.81
90.63
6.67
78.38
6.67
91.89
1.67
45.95
6.67
18.92
–
–
Tax Law
67.11
73.81
75.76
83.33
69.23
76.19
80.10
88.10
71.43
85.94
15.79
66.67
7.02
71.43
1.75
40.48
5.26
14.29
–
–
Wealth Mgt.
45.45
50.00
58.06
63.89
60.61
66.67
50.51
55.56
54.55
67.50
0.00
44.44
0.00
52.78
0.00
47.22
0.00
13.89
–
–
All
80.81
68.45
73.42
84.80
69.37
80.46
81.06
80.00
76.32
92.00
11.18
69.14
7.57
73.14
2.96
49.71
4.61
19.43
–
–
Table 3:
Accuracy (%) across financial topics
for English CFA (top) and Chinese CPA (bottom).
O
= original;
R
= revised. For each row, best original is in
Best-O
,
Best-R
,
Worst-O
,
Worst-R
; best per row is
bolded
.
Who Wins?
The English CFA is led by Claude-Sonnet-3.5 under the missing-condition (89.62%), followed by GPT-5.1-mini (89.01%), Qwen3-Max (88.83%) and Gemini-2.5-Flash (88.19%). In the Chinese CPA, Qwen3-Max establishes a significant lead by 92.00%, whilst Gemini-2.5-Flash (84.80%), DeepSeek-V3 (81.06%), GPT-5.1-mini (80.81%), and Claude-Sonnet-3.5 (80.46%) show comparable performance. Overall, as shown in Tables
2
and
3
, general-purpose commercial models significantly outperform open-source financial models, with most green and blue cells presented at left, and red/yellow cells at right.
General Models:
analyzing by question types in Table
2
, GPT-5.1-mini sweeps all original categories in the CFA and DeepSeek-V3 in CPA. Revised tasks favor Claude-Sonnet-3.5 for CFA and Qwen3-Max for CPA.
Across topics in Table
3
, general models strikingly share identical high scores, with green/blue cells clustered instead of scattered. All general models gain 81.82% for original and 90.00% for revised in
Economics
, 90.91% (
O
) and 100.00% (
R
) in
Portfolio Management
, 79.66% (
O
) in
FSA
, and 71.43% (
O
) in
Fixed Income
. Such sameness may imply that
(i)
general pre-training corpora cover these topics and
(ii)
a common performance ceiling exists among state-of-the-art models.
What are Easy and What are Hard?
Across question types for both O and R in Table
2
, complex calculation is the hardest for all general models. Simple calculation and knowledge transfer & application are comparably challenging. Notably, GPT-5.1-mini exhibits a systemic weakness in CPA, particularly in revised complex calculations.
Regarding topic-specific performance in Table
3
, Fin-R1-7B achieves the highest scores in
fixed income
,
portfolio management
, and
economic law
. In particular, XuanYuan3-70B, Fin-R1-7B, and CFGPT2-7B achieve a perfect precision of 100.0% in
portfolio management
, showing its easiness.
Financial Models:
In the Chinese CPA, 4 times 0% in
knowledge application
and
complex calculation
represent systemic “dead zones” for financial models. Under the wealth management topic, the finance-specific models also stand out by 4 times 0%. These failures indicate that domain-specific supervised fine-tuning (SFT), whilst enhancing factual recall capabilities, has failed to cultivate the multi-step reasoning abilities that require for synthesising complex regulatory rules.
However, on the English CFA, XuanYuan3-70B, Fin-R1-7B, and CFGPT2-7B surprisingly attain their peak results in
complex calculation
by 100% and
statistical methods
for revised variants.
Overall, question difficulty is consistent across model architectures, which is determined by reasoning demands and information integration capabilities, rather than specific types or topics. Our correlation measurements also reflect the same finding: whether the questions is challenging is driven by information integration density (
r=0.78
) rather than surface features like question length (
\rho=0.43
) or numerical frequency (
\rho=0.52
), see measurement details in Appendix
B.1
,
B.2
,
B.3
.
Why Do Original and Revised Questions Behave Differently?
The performance gap between original (
O
) and revised (
R
) settings reveals how models respond to missing information. We examine accuracy changes across question types (Table
2
) and financial topics (Table
3
). While the general models maintain a near-total monopoly on Best-
O
results, a significant shift occurs in the revised setting, where approximately 1/5 of the Best-
R
are captured by specialized financial models.
Among the general models, they tend to speculate rather than detecting missing information. Changing from original to revised, English CFA accuracies improve by 8% to 12%. Specifically, the accuracy of GPT-5.1-mini increased from 80.59% to 89.01% (+8.42%), while DeepSeek-V3’s accuracy rose from 73.55% to 86.34% (+12.79%). However, the Chinese performance declines by 12% (the accuracy of GPT-5.1-mini decreased from 80.81% to 68.45%), and the Worst-
R
is 66.45% with GPT-5.1-mini in the Chinese CPA
complex calculation
type.
A natural question rises:
why these models can recognize the lack of information in English but not in Chinese?
This difference is likely driven by how questions are expressed in the two languages. In Chinese, many words are highly context-dependent and can take on different meanings depending on surrounding cues. For example, words like
走(
to walk
) can mean
walk
or
leave
in different contexts. Therefore, when a key condition is removed, the remaining words may still form a seemingly coherent sentence, but their meaning becomes ambiguous. Models are then easily misled by these remaining cues and tend to interpret the question in a plausible but unsupported way. In English, although context also matters, missing information more often triggers a different response. This can sometimes lead to higher accuracy, not because the model correctly recognizes the lack of information, but because the guessed assumption happens to align with the reference answer.
By contrast, reasoning-focused financial models such as Fin-R1-7B show much larger gains under missing conditions (e.g., +65.57% on Chinese CPA), while earlier financial models such as DISC-FinLLM-13B and FinGPT-7B remain stagnant or decline.
Specifically, zooming into English revised settings across question types in Table
2
, XuanYuan3-70B, Fin-R1-7B, and CFGPT2-7B achieve 100.0% accuracy in both
complex calculation
and
statistical methods
. These models similarly reach perfect scores in revised
portfolio management
in Table
3
, with Fin-R1-7B securing five Best-
R
rankings across diverse topics. The jump from near-zero scores in the original
complex calculations
to perfect performance in the revised setting shows that the issue is not math ability, but unclear concepts.
To further explore the two points above, we introduce an additional evaluation setting that tests whether models can choose which specific condition is missing. This setting enables us to assess whether models truly understand why a question cannot be answered, which is essential for dependable reasoning in complex financial decisions.
Why Do LLMs Make Errors?
The additional evaluation setting reveals that many LLM errors stem from an inability to reason about which specific condition is missing.
General-purpose models
such as GPT-5.1-mini and DeepSeek-V3 often respond with high confidence even when key information is missing.
When models are asked not only to choose an answer but to identify what information is absent, general models often fail to give a concrete explanation. Instead, they fall back on vague assumptions or restate parts of the question. This behavior indicates that the error is not simply a wrong final choice, but defaults to guessing based on learned patterns.
To elaborate, guessing happens because the model prioritizes producing a plausible completion over checking whether the question actually contains enough information to justify any option.
When the question leaves out a key condition, the model still tries to finish the task because it has been trained to treat prompts as solvable requests.
It then replaces the missing condition with a common scenario that often occurs with similar wording in its training data.
This makes the answer sound coherent even though it is no longer supported by the problem statement.
Financial models
such as DISC-FinLLM-13B and FinGPT-7B perform extremely poorly, with accuracy below 20.00%.
This outcome is counter-intuitive, as these models are explicitly fine-tuned for financial tasks and possess substantial domain knowledge.
A closer inspection suggests that this poor performance is not due to a lack of financial knowledge.
Instead, domain-specific terminology in the questions, such as "Processing Agreement" or "Consumption Tax," appears to trigger aggressive knowledge retrieval.
Once this retrieval process is activated, the models tend to commit to forced calculations or rule-based procedures, even when the information provided is insufficient to support any option. As a result, these models are more likely to select an answer rather than recognize that no option can be justified, which leads to particularly low accuracy in all settings.
In other words, early financial models such as DISC-FinLLM-13B and FinGPT-7B are trained on financial data, they mainly learn to recognize financial terms, and this helps them reduce errors caused by unfamiliar terminology and better identify the meaning of a question.
However, these models are not trained to recognize when the information provided is insufficient, hence, still lack the ability to detect when a question itself cannot be answered.
Reasoning-focused models such as Fin-R1-7B also face a new challenge.
Although Fin-R1-7B has learned to reject answers when information is insufficient, it sometimes struggles to decide where to stop. Its reasoning trajectory shows that after the model correctly recognizes the core principles, it still continues reasoning, such as considering unlikely financial cases that are not implied by the question.
Take-Away:
This behavior reflects a fundamental trade-off.
The ability to say
No
is essential for handling missing conditions, but without a clear stopping criterion, it can also trigger unnecessary complexity.
This mirrors a common challenge in human decision-making, where caution and thoroughness sometimes lead to overthinking.
Taken together, improving financial LLMs requires not only stronger reasoning, but also a clear sense of when to stop—balancing the ability to reject unsupported answers with the ability to accept well-specified ones.
4.3
How Do Models Behave When No Answer is Justified
Based on the revised subset, we replace the correct option with “none of the above” as the final choice, to assess whether models can truly identify missing information when no answer is justified.
Accuracy Decreases for Most Models When Models Are Forced to Choose NOTA
More apparent declines happen among general-purpose models, and financial-specific models witness majority of rises in Chinese questions in Table
4
.
General Models
Financial Models
Model
Chinese
English
Average
Model
Chinese
English
Average
Gemini 2.5 Flash
48.78
72.38
60.58
Fin-R1-7B
77.10
84.50
81.10
Claude 3.5 Sonnet
37.36
70.11
53.73
XuanYuan3-70B
61.70
84.70
73.20
Qwen3-Max
44.85
59.94
52.39
CFGPT2-7B
54.90
75.20
65.05
GPT-5.1-mini
30.64
68.23
49.44
DISC-FinLLM-13B
20.00
12.26
16.13
DeepSeek V3
36.21
56.87
46.54
FinGPT-7B
–
3.50
3.50
Avg.
39.57
65.51
52.54
Avg.
53.43
52.03
47.80
Table 4:
Cross-comparison of model accuracy (%) on None of The Above questions compared to Revised accuracy.
Green
indicates NOTA accuracy higher than Revised;
red
indicates lower.
Most Models Still Try to Answer Even When No Answer Is Justified
Both finance-specific models and general frontier models struggle when
None of the above
is the correct answer.
In Table
4
, models such as GPT-5.1-mini and DeepSeek V3 show large accuracy drops, especially on Chinese questions.
These errors do not stem from a lack of reasoning ability, but from a tendency to keep selecting an answer even when the question cannot be resolved with the given information.
Empirically, general-purpose models tend to over-commit by selecting an answer even when the question lacks a clear logical basis.
This behavior likely comes from training objectives that reward producing a response rather than recognizing when a problem cannot be solved.
As a result, these models fail not because they cannot reason, but because they do not know when to stop reasoning.
However, the strong performance of Fin-R1-7B shows that this limitation is not shared by all models.
Unlike other models, Fin-R1-7B does not treat the
None of the above
option as a fallback choice.
Instead, it consistently checks whether the information provided in the question is sufficient to justify any of the listed options. Its reasoning outputs indicate that the model actively looks for missing inputs before committing to an answer.
For example, in tax-related questions, Fin-R1-7B explicitly notes the absence of required quantities, such as a comparable market price, and refrains from selecting an option when such information is missing.
As Fin-R1-7B is designed to prioritize condition checking before answer selection, which directly counteracts the tendency to over-answer encouraged by RLHF.
This design also prevents the retrieval-heavy calculation mode triggered by financial terminology from turning into forced commitments, so the model can choose NOTA when the question is underspecified.
Why Accuracy Overestimates Reliability in NOTA
In NOTA settings, selecting the correct option alone is insufficient to indicate reliable decision-making.
When no answer is justified, accuracy can be artificially inflated by models that avoid making a clear choice without understanding why the problem cannot be resolved.
To address this, we examine whether a model’s reasoning supports the same conclusion as its final answer.
Reasoning-enhanced models such as Fin-R1-7B maintain high consistency between answers and explanations, identifying which required condition is missing before selecting NOTA.
By contrast, some financial domain models appear to perform reasonably well based on answer accuracy alone, but their explanations fail to support the chosen answer.
In these cases, the model selects NOTA simply because it is a safe option, not because it understands that the question cannot be answered.
We further observe a degenerate response pattern in some models, where uncertainty is expressed by selecting multiple options at once (
e.g., AB or ABCD
).
Although this behavior may seem cautious, it does not represent a valid financial decision because it offers no clear conclusion.
Taken together, financial applications require models to reject an answer for clear and traceable reasons, rather than by chance.
5
Conclusions
In this paper, we show that evaluating financial LLMs solely by answer accuracy overlooks a critical dimension of reliability: the ability to determine whether a question should be answered at all. General-purpose models tend to keep answering even when key information is missing. Most financial domain models avoid blind guessing, but they still fail to clearly identify which condition is missing. Reasoning-focused models such as Fin-R1 perform better by explicitly recognizing missing conditions, but sometimes overthink the problem.
Based on this, we argue that reliability in financial LLMs should be evaluated across three complementary behaviors: whether it can answer correctly when information is sufficient, whether it can identify which conditions are missing when information is incomplete, and whether it can reject answering when no option is supported by the question. Only models that behave consistently across these settings can be considered dependable for real-world financial use.
Limitations and Future Work
A main limitation of this work is that we do not modify or train any models. Our study can show where failures come from, but we do not directly demonstrate how much fine-tuning can fix them. Because our evaluation uses multiple-choice questions, the same behavior may show up in different ways when models respond free-form questions or interact with users.
These limits point to a clear next step. As our results suggests that models either guess, or they pick a safe option without being able to explain what is missing, or they reason too far. A practical solution is a unified training pipeline that separates what should be learned from what should be controlled. First, specialized SFT should teach professional knowledge and stable reasoning formats, so the model learns what variables and rules matter in common financial tasks. Then logic-driven RL should train the model’s behavior when conditions are missing. At this stage, the model should be rewarded for explicitly pointing out what information is missing, and penalized for guessing, making up assumptions, or reasoning too far beyond what the question supports.
We also plan to move beyond passive rejection. When a question lacks a key variable, the model should not only say ”
This question has no answer
”, but also state what information it needs and ask for it. This would build a interactive workflow for users. We will also broaden the dataset beyond accounting and tax law to cover risk management, securities regulation, and quantitative finance, so that the same evaluation logic can test whether models stay reliable across the full range of real financial work.
Ethical Statement
This work is intended to improve the safe and responsible use of large language models in financial applications. All datasets used in this study are constructed from publicly accessible materials or manually rewritten questions and do not include proprietary data, confidential information, or verbatim content from protected examinations. Our evaluation highlights a specific risk relevant to high-stakes domains: confident model outputs in situations where information is insufficient to justify a determinate answer. By explicitly identifying and measuring this behaviour, our work aims to reduce the likelihood of uncritical deployment of LLMs in financial decision-making, rather than to encourage automation of such decisions.
We emphasise that the proposed benchmark and any derivative artefacts, such as distilled small-parameter models, are intended for research and risk-assessment purposes only. Any deployment of language models in real financial, regulatory, or advisory contexts must prioritize the auditability of reasoning chains, involve appropriate human oversight, and comply with applicable legal and ethical standards. We aim to support the development of "logic-first" assistance tools rather than fully autonomous systems for high-stakes financial judgments.
Appendix A
Dataset Curation
A.1
Formal Definition of Condition-Missing Construction
We formalize the construction of condition-missing questions to clarify how missing information is introduced in a controlled manner.
Full-condition formulation.
Each full-condition financial reasoning question can be abstracted as a logical implication
(P_{1}\land P_{2}\land\dots\land P_{n})\Rightarrow C,
(1)
where
\{P_{i}\}
denotes the set of premises explicitly stated in the question, such as numerical assumptions, regulatory standards, time horizons, or accounting rules, and
C
denotes the unique justified conclusion.
Under full conditions, the premise set is sufficient to determine
C
without ambiguity.
Condition removal.
To construct a condition-missing question, we remove one or more logically necessary premises from the original set.
Let
\mathcal{P}^{\ast}\subset\{P_{1},\dots,P_{n}\}
denote the subset of premises identified as necessary for determining the conclusion.
The revised question is defined by the reduced premise set
\{P_{i}\}\setminus\mathcal{P}^{\ast}.
(2)
Coherence constraint.
After removal, the remaining premises must still admit at least one coherent interpretation.
Formally, we require
\exists\,\mathcal{M}\quad\text{s.t.}\quad\mathcal{M}\models\bigwedge_{P_{i}\in\{P\}\setminus\mathcal{P}^{\ast}}P_{i},
(3)
ensuring that the revised question remains grammatically and semantically well-formed, rather than logically inconsistent.
Non-uniqueness constraint.
At the same time, the remaining premises must no longer uniquely determine the conclusion.
Specifically, we require the existence of multiple internally consistent interpretations
\mathcal{M}_{j}\models\bigwedge_{P_{i}\in\{P\}\setminus\mathcal{P}^{\ast}}P_{i}\quad\text{and}\quad\mathcal{M}_{j}\models C_{j},
(4)
with
C_{j}\neq C_{j^{\prime}},
(5)
for some
j\neq j^{\prime}
.
This condition guarantees that no single answer can be justified given the available information.
Functional tagging.
Each removed premise
P_{k}\in\mathcal{P}^{\ast}
is annotated by its functional role
\tau(P_{k})\in\{\textit{numerical},\textit{regulatory},\textit{temporal},\textit{standard}\},
(6)
to ensure that only logically necessary information is removed, rather than incidental details.
Together, these constraints ensure that condition-missing questions differ from full-condition questions not in computational difficulty, but in epistemic status: the task shifts from computing a correct answer to recognizing that no uniquely justified answer can be derived.
A.2
Interpretation of Original vs. Revised Question Behavior
This appendix provides a formal interpretation of the behavioral differences between original (full-condition) and revised (condition-missing) questions discussed in Section 4.
A.3
Original Questions: Well-Posed Reasoning
Each original question is constructed to be well-posed.
Formally, an original financial reasoning task can be represented as
(P_{1}\land P_{2}\land\dots\land P_{n})\models C,
(7)
where
\{P_{i}\}
denotes the complete set of premises provided in the question, including numerical values, regulatory rules, and institutional assumptions, and
C
denotes the unique justified conclusion.
Under this setting, a correct solution requires the model to derive
C
from the full premise set.
Errors on original questions therefore arise from failures such as incorrect deduction, missing domain knowledge, or computational mistakes, but not from ambiguity in the task definition itself.
A.4
Revised Questions: Underdetermined Reasoning
Revised questions are constructed by removing one or more logically necessary premises.
Let
\mathcal{P}^{\ast}\subset\{P_{1},\dots,P_{n}\}
denote the set of removed conditions.
The remaining task is then defined by the reduced premise set
\{P_{i}\}\setminus\mathcal{P}^{\ast}.
(8)
In this case, the reduced premises no longer uniquely determine a conclusion.
Formally, there exist multiple internally consistent interpretations
\mathcal{M}_{j}\models\bigwedge_{P_{i}\in\{P\}\setminus\mathcal{P}^{\ast}}P_{i}\quad\text{and}\quad\mathcal{M}_{j}\models C_{j},
(9)
with
C_{j}\neq C_{j^{\prime}},
(10)
for some
j\neq j^{\prime}
.
As a result, no single conclusion can be justified solely from the available information.
A.5
Implications for Accuracy and Model Behavior
In original questions, accuracy directly reflects whether the model successfully derives the unique justified conclusion
C
.
In revised questions, however, accuracy depends on whether the model’s implicit assumptions happen to align with the reference answer retained from the original task.
Formally, a correct prediction on a revised question satisfies
\hat{C}=C_{\text{ref}},
(11)
even though
\{P_{i}\}\setminus\mathcal{P}^{\ast}\;\not\models\;C_{\text{ref}}.
(12)
This explains why some models improve after condition removal while others degrade.
Performance changes reflect differences in fallback assumptions and inductive biases, rather than differences in reasoning ability.
Consequently, accuracy ceases to be a reliable indicator of reasoning quality once questions are no longer well-posed.
A.6
Formalization of Abstention Under Missing Correct Answers
This appendix provides a formal explanation of the experiment in Section 4.3, where the correct option in condition-missing questions is replaced by
“None of the above”
(English) or
“以上都不是”(Chinese).
Problem Setup
A full-condition financial reasoning question can be abstracted as a set of premises and a target conclusion,
(P_{1}\land P_{2}\land\dots\land P_{n})\;\models\;C,
(13)
where the premises jointly support a unique, justified conclusion.
In the condition-missing setting, one or more logically necessary premises are removed.
The resulting task is characterized by
(P_{1}\land\dots\land P_{k-1}\land P_{k+1}\land\dots\land P_{n})\;\not\models\;C,
(14)
meaning that the remaining information is insufficient to justify any single concrete answer.
Implicit Assumption Completion
Despite this underdetermination, models often still select a concrete option.
This behavior can be understood as implicitly completing the missing information with a default assumption.
Formally, the model selects an internal completion
\mathcal{M}_{j}
such that
\mathcal{M}_{j}\models\bigwedge_{i\neq k}P_{i}\quad\text{and}\quad\mathcal{M}_{j}\models C_{j},
(15)
where
C_{j}
corresponds to one of the provided answer options.
If
C_{j}
happens to match the reference answer retained from the original full-condition question, the prediction is counted as correct, even though the conclusion is not logically supported by the observed premises alone.
Abstention as the Rational Strategy
When the correct option is explicitly set to
“None of the above”
, this shortcut is no longer available.
In this setting, a rational response requires recognizing that none of the concrete options is justified.
Formally, the correct decision rule is
\forall j,\;\;\{P_{i}\}\not\models C_{j}\quad\Rightarrow\quad\text{select NOTA}.
(16)
This formulation makes explicit that the task is no longer to identify the best answer, but to assess whether any answer can be supported at all.
Empirically, however, models frequently violate this condition by selecting a specific option even when no option is entailed.
Interpretation
The sharp accuracy drop observed in the
None-of-the-above
setting therefore indicates that many correct answers in the original revised setting were achieved through implicit assumption alignment rather than principled reasoning.
This experiment isolates a structural limitation of current LLMs: they lack a robust mechanism for recognising when available information is insufficient and reasoning should be suspended rather than completed.
Dataset
Type
Content
Original
(Chinese)
Topic
Auditing
Q
下列关于管理层编制财务报告的说法中,正确的有()。
Opts
A. 根据财务报告编制基础编制,不能运用自身判断 | B. 根据相关法律法规的规定确定适用的财务报告编制基础 | C. 根据适用的财务报告编制基础编制财务报表 | D. 在财务报表中对使用的财务报告编制基础作出恰当的说明
Gold
BCD
Rsn
管理层在编制财务报表时应严格按照财务报告编制基础,但是也存在需要根据使用的财务报告编制基础运用判断做出合理会计估计的情况,不是完全不运用自身判断。
Missing
(Chinese)
Topic
Accounting
Q
若要计算委托加工业务中受托方应代收代缴的消费税额,应补充下列哪项关键信息()。
Opts
A. 该应税物资在受托方是否存在同类消费品的销售价格 | B. 该批原材料是否属于自产产品 | C. 加工费是否以现金方式结算 | D. 委托方是否具有一般纳税人资格
Gold
A
Rsn
需要知道是否有同类消费品销售价格来确定计税依据。
Original
(English)
Topic
Financial Statement Analysis
Q
Which of the following is most likely to have been included in Sea Ltd’s SEC registration statement?
Opts
A. Underwriters’ fairness opinion | B. Assessment of risk factors | C. Projected cash flows and earnings
Gold
B
Rsn
SEC registration statements must include risk factor assessments.
Missing
(English)
Topic
Derivatives Investment
Q
For partnerships to achieve pass-through taxation, which structural characteristic must be present?
Opts
A. Entity level taxation | B. Individual partner level only | C. Both entity and individual | D. No taxation
Gold
B
Rsn
Pass-through taxation means income is taxed only at individual partner level.
Table 5:
Question examples from Original and Missing datasets. Gold answers and reasons are highlighted in yellow and displayed in
bold italic
. “Q” = Question, “Opts” = Options, “Gold” = Gold Answer, “Rsn” = Reason.
Figure 2:
Question Type Distribution Across Datasets
Appendix B
Statistical Analysis Methodology
This appendix details the computational procedures and formulas used for all statistical analyses reported in Section X.
B.1
Difficulty Ranking Consistency (Spearman’s
\rho
)
Purpose.
To measure whether different models exhibit consistent difficulty rankings across question classes.
Data Preparation.
For each model and each question class, we compute the average accuracy as the percentage of correctly answered questions.
For model
m
on question class
c
:
\text{Accuracy}(m,c)=\frac{\text{Number of correct answers}}{\text{Total questions in class }c}\times 100\%
(17)
Ranking Assignment.
For each model
m
, we rank the question classes from hardest (rank 1) to easiest (rank
C
) based on accuracy:
R_{m}(c)=\text{rank of class }c\text{ for model }m
(18)
where lower accuracy yields lower (harder) rank.
Pairwise Spearman Correlation.
For each pair of models
(m_{1},m_{2})
, we compute Spearman’s rank correlation coefficient:
\rho_{m_{1},m_{2}}=1-\frac{6\sum_{c=1}^{C}d_{c}^{2}}{C(C^{2}-1)}
(19)
where
d_{c}=R_{m_{1}}(c)-R_{m_{2}}(c)
is the difference in ranks for class
c
.
Average Correlation.
The reported average Spearman’s
\rho=0.91
is computed as:
\bar{\rho}=\frac{2}{M(M-1)}\sum_{m_{1}=1}^{M-1}\sum_{m_{2}=m_{1}+1}^{M}\rho_{m_{1},m_{2}}
(20)
where
M=10
models.
Statistical Significance.
For each pairwise correlation, we compute the test statistic:
t=\rho\sqrt{\frac{C-2}{1-\rho^{2}}}
(21)
which follows a
t
-distribution with
C-2
degrees of freedom under the null hypothesis of no correlation. With
C=7
question classes, all pairwise correlations yielded
p<0.001
.
B.2
Surface Feature Correlations
Question Length Analysis.
For each question class
c
, we compute the average token count:
L_{c}=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\text{tokens}(q_{i})
(22)
where
\text{tokens}(q_{i})
is the number of tokens in question
i
.
Independent Samples
t
-test.
To test whether token length differs between the hardest (Knowledge Transfer,
L_{\text{KT}}=127
) and easiest (Term Explanation,
L_{\text{TE}}=134
) categories:
t=\frac{\bar{L}_{\text{KT}}-\bar{L}_{\text{TE}}}{\sqrt{\frac{s_{\text{KT}}^{2}}{n_{\text{KT}}}+\frac{s_{\text{TE}}^{2}}{n_{\text{TE}}}}}
(23)
where
s_{c}^{2}
is the sample variance of token counts in class
c
, and
n_{c}
is the sample size.
Numerical Density Correlation.
We compute numerical density as:
D_{c}=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\frac{\text{count of numbers in }q_{i}}{\text{tokens}(q_{i})}\times 100
(24)
Pearson correlation between
D_{c}
and error rate
E_{c}=100-\text{Acc}_{\text{avg}}(c)
:
r_{D}=\frac{\sum_{c=1}^{C}(D_{c}-\bar{D})(E_{c}-\bar{E})}{\sqrt{\sum_{c=1}^{C}(D_{c}-\bar{D})^{2}}\sqrt{\sum_{c=1}^{C}(E_{c}-\bar{E})^{2}}}
(25)
B.3
Information Integration Metrics
Feature Annotation.
Each question was manually annotated by three expert annotators (Fleiss’
\kappa=0.81
, substantial agreement) for:
•
Interacting Premises
(
P_{i}
): Number of distinct information pieces that must be combined
•
Cross-domain Dependencies
(
C_{i}
): Rated 1–5, whether reasoning spans multiple financial subdomains
•
Constraint Reconciliation
(
R_{i}
): Rated 1–5, complexity of reconciling conflicting requirements
Aggregation.
For each question class
c
:
\displaystyle\bar{P}_{c}
\displaystyle=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}P_{i}
(26)
\displaystyle\bar{C}_{c}
\displaystyle=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}C_{i}
(27)
\displaystyle\bar{R}_{c}
\displaystyle=\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}R_{i}
(28)
Pearson Correlation with Error Rate.
For feature
X\in\{P,C,R\}
:
r_{X}=\frac{\sum_{c=1}^{C}(\bar{X}_{c}-\bar{\bar{X}})(E_{c}-\bar{E})}{\sqrt{\sum_{c=1}^{C}(\bar{X}_{c}-\bar{\bar{X}})^{2}}\sqrt{\sum_{c=1}^{C}(E_{c}-\bar{E})^{2}}}
(29)
Appendix C
LLMs Implementation Details
C.1
LLMs Overview and Selection
We set up a 5v5 comparison framework to make sure that the cross-paradigm examination was thorough and fair. This includes five cutting-edge general-purpose models that can be accessed through standard APIs: GPT-5.1-mini, Gemini-2.5-Flash, Claude-3.5-Sonnet, DeepSeek-V3, and Qwen3-Max. These models were compared to five representative open-weight financial models that have been successfully used in our local inference environment: XuanYuan3-70B
(
30
)
, Fin-R1-7B
(
17
)
, CFGPT2-7B
(
15
)
, DISC-FinLLM-13B
(
3
)
, and FinGPT-7B
(
26
)
.
We started with a poll of possible candidates and then narrowed it down to this curated list based on strong standards of operational viability and technical transparency. Some intriguing models were left out because of administrative or proprietary issues. For example, InvestLM (HKUST)
(
27
)
was still not available since the licensing approval process did not respond to multiple academic enquiries. Industrial systems like BloombergGPT
(
21
)
and Baichuan4-Finance
(
28
)
were also left out because their weights are strictly closed-source. Zhi Xiaobao, on the other hand, is part of a consumer-facing app that doesn’t have a research-grade API. Also, ICBC and Hundsun’s
2
2
2
https://www.hs.net/lightgpt/
internal infrastructure models can only be used in private banking settings, which makes it impossible to verify them independently.
Commercial API limits and architectural restrictions also required certain exclusions. HithinkGPT (Tonghuashun)
3
3
3
https://aimiai.com/
has a conversational interface, but its rigorous token-level quota of only 1,000 tokens with no institutional top-up option made it unsuitable for high-throughput reasoning audits. FinGPT-7B
(
26
)
was successfully deployed, but its tokeniser, which came from Llama-2 and Falcon, doesn’t work well with Chinese characters; hence, Chinese tasks produce garbled outputs.
To protect the integrity of the data and make sure the comparison was fair, FinGPT-7B’s evaluation was limited to the English sub-benchmarks. Furthermore, across all question types in the Chinese CPA, all financial-specific models face collapse when encountering complex calculation problems, whether original or revised questions.
Table
6
summarises representative commercial and financial LLMs. Looking ahead, we intend to broaden this research by including a broader spectrum of emerging Financial LLMs, such as Fin-o1-8B
4
4
4
https://huggingface.co/TheFinAI/Fin-o1-8B
(
19
)
, DianJin-R1-32B
5
5
5
https://huggingface.co/DianJin/DianJin-R1-32B
(
34
)
, Finance-Llama
6
6
6
https://huggingface.co/tarun7r/Finance-Llama-8B
, Finance-Qwen
7
7
7
https://huggingface.co/WiroAI/WiroAI-Finance-Qwen-1.5B
;
https://huggingface.co/WiroAI/WiroAI-Finance-Qwen-7B
, Fin-PRM
(
31
)
. Future editions of this study will look into additional specialised open-source financial models as they become available in order to better benchmark the growing environment of financial AI.
Beyond static model evaluation, our future work will pivot towards autonomous financial intelligence. We aim to explore and develop frameworks similar to FinChain
(
25
)
, focusing on structured financial reasoning chains, and FinAgent
(
29
)
, which emphasizes multi-agent collaboration for complex task automation. These directions will allow us to transition from benchmarking individual models to evaluating integrated financial AI systems in real-world operational environments.
Category
Organization
Model
Size
Key Features / Notes
Source / Reference
General
Purpose
Closed-source
GPT-5.1-mini
∗
–
Frontier general-purpose performance
openai/gpt-5.1-mini
Claude-3.5-Sonnet
∗
–
Strong reasoning and stability
anthropic/claude-3.5
Gemini-2.5-Flash
∗
–
High-efficiency reasoning model
google/gemini-2.5
Open-source
DeepSeek-V3
∗
16B/236B
Advanced MoE reasoning focus
deepseek/deepseek-v3
Qwen3-Max
∗
–
Leading multilingual performance
qwen/qwen3-max
Academic
Financial
FinGPT-7B
∗
7B
Multi-source data curriculum
AI4Finance/fingpt-7b
InvestLM
65B
Expert-level CFA-style reasoning
HKUST/investlm
DISC-FinLLM-13B
∗
13B
Multi-turn advisory optimization
Fudan/disc-finllm-13b
CFGPT2-7B
∗
7B
Knowledge graph integration
Tongji/cfgpt2-7b
Fin-R1-7B
∗
7B
GRPO reinforcement learning
SUFE/fin-r1-7b
FinMA-7B
7B
Multi-task instruction tuning
Pixiu/finma-7b
Industrial
Financial
BloombergGPT
50B
Proprietary domain pre-training
bloomberg/bloomberggpt
XuanYuan3-70B
∗
70B
Long-report synthesis focus
duxiaoman/xuanyuan3-70b
Baichuan4-Fin
–
Professional logic alignment
baichuan/baichuan4-fin
HithinkGPT
–
Real-time API integration
hithink/hithinkgpt
Zhi Xiaobao
–
Consumer wealth management
antgroup/zhixiaobao
Infra-
structure
Financial
ICBC Zhiyong
–
Risk control and compliance
icbc/zhiyong
Light GPT
–
Securities trading backend
lightgpt/lightgpt
Table 6:
Comprehensive Taxonomy of Large Language Models in the Financial Domain. Models marked with (
∗
) are those specifically evaluated in our empirical study.
C.2
Hardware Configuration
The hardware specifications for each model were established based on their parameter counts and memory requirements. XuanYuan3-70B, the largest model in our assessment with 70 billion parameters, demands around 140GB of VRAM at FP16 precision, thus requiring two NVIDIA H20 GPUs (96GB each) with automatic tensor parallelism. For the 13B-parameter DISC-FinLLM derived from Baichuan-13B, we utilised a virtual GPU setup featuring 32GB of VRAM. The remaining 7B-scale models, CFGPT2, FinGPT, and Fin-R1—were all operable on a single NVIDIA RTX 4090 (24GB) utilising FP16 precision.
In contrast, the five frontier general-purpose models are GPT-5.1-mini, Gemini-2.5-Flash, Claude-3.5-Sonnet, DeepSeek-V3, and Qwen3-Max. They were evaluated via high-performance API calls. These cloud-based evaluations eliminated the necessity for local GPU infrastructure while ensuring access to the latest model iterations.
All experiments used a unified software stack: PyTorch 2.1.0, Python 3.10 (Ubuntu 22.04), CUDA 12.1, and Transformers 4.38.2. Models were run with greedy decoding (
do_sample=False
) and
max_new_tokens=1024
to ensure deterministic and reproducible outputs.
C.3
Prompts and Structured Output Protocol
All models are assessed under a unified protocol tailored to their specific deployment: locally implemented financial models utilize their native chat templates via
tokenizer.apply_chat_template()
to maintain strict alignment with original training paradigms, whereas general-purpose commercial models are accessed directly via standard API endpoints.
All experiments, which including the
Original
,
Revised
, and
NOTA
settings—employ bilingual prompts mandating a structured JSON output with three fields:
reason
,
answer
, and
confidence
. To ensure robust data extraction and minimize evaluation bias, we develop a cascaded four-stage parsing strategy: the system first attempts to parse the entire output string using standard library decoders; if this fails, it proceeds to extract content within
json
or markdown tags; if the output remains unparsable, regular expressions are applied to identify the outermost balanced curly braces
{
…} to isolate the JSON object from potential conversational filler; finally, in cases of severe syntax corruption, we utilize field-level regex fallback to recover the
reason
,
answer
, and
confidence
fields independently, thereby maximizing the recovery of partial but valid reasoning. For multiple-choice questions, extracted answers are normalized to uppercase and sorted alphabetically.
One major thing we saw when we looked at the five open-weight financial LLMs is the rise of a "formatting tax". This means that using strict JSON syntax usually lowers accuracy by 3% to 4% compared to free-form output. We hypothesize that the cognitive overhead of constraint satisfaction competes with the neural resources required for complex multi-step financial reasoning. However, we prioritize the structured protocol to ensure Epistemic Awareness can be explicitly audited through the reason field, decoupling logical deduction from linguistic fluency.
Chinese Prompt Template (CPA)
System:
您是一位金融问题解决专家。您将收到一道单选或多选题。请返回一个严格的 JSON 数据,包含三个键:
"reason"
、
"answer"
和
"confidence"
。
User:
问题:
{question}
请严格按照以下格式输出 JSON 数据。请勿在 JSON 对象之外包含任何文本。
{
"reason": "...",
"answer": "A/B/C/D or ACD",
"confidence": 0--100
}
Figure 3:
The Chinese prompt template for CPA-style evaluation. This template is consistently used across all test variants.
English Prompt Template (CFA)
System:
You are an expert in financial problem solving. You will be given a single- or multi-choice question. Please return a STRICT JSON object with three keys:
reason
,
answer
, and
confidence
.
User:
Question:
{question}
A.
{option_A}
B.
{option_B}
C.
{option_C}
D.
{option_D}
Output Format:
{ "reason": "...", "answer": "...", "confidence": 0-100 }
Figure 4:
The English prompt template for CFA-style evaluation. The strict JSON requirement ensures logical decoupling.
Figure 5:
The evaluation pipeline
C.4
Gated-Auditing Architecture
To mitigate the "formatting tax" and curb excessive speculative reasoning, we incorporated a Gated Attention mechanism within the final three layers of the model in the next step. This gating layer acts as a "logical firewall," dynamically filtering out redundant cognitive paths. During the RL phase, we utilized a Dual-Auditing Reward (DAR) that incentivizes the explicit identification of missing variables while penalizing over-extended reasoning chains that deviate from the core financial context.
C.5
XuanYuan3-70B Configuration
Model Source.
Duxiaoman-DI/Llama3-XuanYuan3-70B-Chat
8
8
8
https://huggingface.co/Duxiaoman-DI/Llama3-XuanYuan3-70B
(
30
)
XuanYuan3-70B is a state-of-the-art Chinese large language financial model developed by Du Xiaoman. Building upon the Llama-3-70B architecture, it extends the foundational capabilities established in its predecessor, XuanYuan 2.0, through advanced post-training and financial-domain-specific instruction tuning. It is designed to handle sophisticated financial analysis, regulatory compliance, and cross-lingual financial reasoning with high parameter efficiency and domain expertise. We deploy XuanYuan3-70B using 2
\times
NVIDIA H20 GPUs (96GB each) with automatic tensor parallelism, FP16 precision, greedy decoding, and
repetition_penalty=1.05
.
Chat Template.
XuanYuan3-70B follows the native Llama-3 chat template, in which
system
,
user
, and
assistant
roles are explicitly encoded at the tokenizer level. We apply the template via
tokenizer.apply_chat_template()
with
add_generation_prompt=True
(Figure
6
).
<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>
{system}<|eot_id|>
<|start_header_id|>user<|end_header_id|>
{user}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Figure 6:
Llama-3 chat template used by XuanYuan3-70B.
Detailed Results.
Table
7
demonstrates that the new strategy performs considerably better in all areas of finance. The model achieves nearly perfect results in Portfolio Management (100%) and Derivatives Investment (92.00%) on the English CFA track. The model shows a significant improvement in the Chinese CPA track, which is more crucial. Accuracy increased by 70.91% in Auditing and 71.71% in Economic Law. This shows that the optimization was effective in assisting the model in dealing with complex laws and regulations unique to distinct places.
Question Topic (CFA)
Orig.
Rev.
\Delta
n
Question Topic (CPA)
Orig.
Rev.
\Delta
n
Corporate Finance
60.00
90.00
+30.00
10
Accounting
4.49
63.33
+58.84
89
Derivatives Investment
53.33
92.00
+38.67
45
Auditing
25.76
96.67
+70.91
66
Economics
53.33
80.00
+26.67
15
Economic Law
6.67
78.38
+71.71
60
Equity Investment
57.63
86.54
+28.91
59
Tax Law
15.79
66.67
+50.88
57
Financial Statement Analysis
65.12
88.89
+23.77
43
Wealth Management
0.00
44.44
+44.44
32
Fixed Income Investment
42.86
78.57
+35.71
49
Other Investments
62.77
86.42
+23.65
94
Portfolio Management
75.00
100.00
+25.00
4
Quantitative Analysis
65.96
86.67
+20.71
47
Overall
58.47
84.74
+26.27
366
Overall
11.18
69.14
+57.96
304
Table 7:
Accuracy (%) of XuanYuan3-70B by question topic. Left: English CFA (9 subjects). Right: Chinese CPA (5 subjects).
n
denotes the number of original questions.
Table
2
illustrates the proficiency gains across specific task categories. We observe a significant advancement in mathematical reasoning; within the English CFA, accuracy in Complex Calculation reaches 100.0% under revised settings. Similarly, in the Chinese CPA, performance in Simple Calculation increases by more than 50.0%. Most notably, XuanYuan3-70B effectively addresses the zero-shot performance collapse in CPA Knowledge Application, elevating accuracy from 0.00% to 66.67%. These results indicate that the model maintains professional-level proficiency across both conceptual frameworks and quantitative analysis.
English CFA
Chinese CPA
Question Type
Orig.
Rev.
\Delta
n
Orig.
Rev.
\Delta
n
Conceptual Understanding
77.42
79.07
+1.65
62
10.92
80.00
+69.08
119
Simple Calculation
36.67
82.12
+45.46
30
9.76
64.15
+54.39
41
Complex Calculation
33.33
100.00
+66.67
15
0.00
–
–
2
Comprehensive Judgment
62.38
88.37
+26.00
101
10.34
71.72
+61.37
87
Knowledge Application
40.70
75.00
+34.30
86
0.00
66.67
+66.67
7
Statistical Methods
71.43
93.62
+22.19
70
16.67
60.00
+43.33
48
Overall
58.47
84.74
+26.27
364
11.18
69.14
+57.96
304
Table 8:
Accuracy (%) of XuanYuan3-70B by question type across English CFA and Chinese CPA examinations.
Robustness Analysis under Adversarial Masking.
The performance of XuanYuan3-70B within the NOTA-masked adversarial environment reveals a significant logical calibration challenge inherent to large-scale parameter models (Table
9
). The model exhibits a pronounced decoupling between reasoning trajectories and final label selection. Specifically, qualitative analysis shows that while the internal reasoning often converges on a specific numerical option, the model ultimately selects the “None of the Above” (NOTA) label due to surface-level pattern matching of the adversarial prompt structure. Furthermore, the model maintains exceptionally high confidence levels (
\mu>90.0
) despite logical validity falling below 10%, indicating a severe overconfidence bias. This suggests that while the model successfully identifies adversarial markers with high frequency in the English CFA (exceeding 80%), it fails to ground its selection in a valid deductive process.
Benchmark Subset
Total
Answer Acc.
Reasoning Acc.
Mean Conf.
Revised English CFA (Masked)
367
84.7%
8.4%
92.9
Revised Chinese CPA (Masked)
175
61.7%
5.1%
87.9
Table 9:
XuanYuan3-70B performance on NOTA masked questions: High accuracy vs. reasoning collapse.
In summary, whilst XuanYuan3-70B utilizes a parameter count ten times that of Fin-R1-7B, it exhibits significantly lower logical consistency. This disparity suggests that the structural reasoning constraints established through reinforcement learning, specifically GRPO, represent a fundamental capability that cannot be replaced by the brute-force aesthetics of parameter expansion. These findings underscore that specialized reasoning paths are not emergent properties of scale alone but require targeted algorithmic constraints to ensure deductive integrity.
C.6
Fin-R1-7B Configuration
Model Source.
SUFE-AIFLM-Lab/Fin-R1
9
9
9
https://huggingface.co/SUFE-AIFLM-Lab/Fin-R1
(
17
)
Fin-R1 is developed by Shanghai University of Finance and Economics (SUFE), based on Qwen2.5-7B. It achieves state-of-the-art reasoning performance via GRPO-style reinforcement learning for complex financial problems. As of December 2025, it nearly matched the results of Xuanyuan3-70B. We use a single NVIDIA RTX 4090 (24GB) with FP16 precision, greedy decoding, and
repetition_penalty=1.05
.
Chat Template.
Fin-R1 inherits the Qwen2.5 chat template, accessible via
tokenizer.apply_chat_template()
. We use the standard message format with system and user roles:
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
Figure 7:
Fin-R1 inference using official Qwen2.5 chat template.
Note: The official Fin-R1 repository recommends deployment via vLLM with an OpenAI-compatible API and a specific system prompt using
......
format for chain-of-thought reasoning. For consistency with our unified evaluation framework, we use direct inference with our standard JSON-output prompt instead.
Detailed Results.
The evaluation results for Fin-R1-7B (Table
10
and
11
) present several significant insights. Despite possessing merely 7 billion parameters, Fin-R1-7B attains an overall accuracy of
83.65%
in English CFA and
73.14%
in Chinese CPA, closely approaching the performance of the significantly larger XuanYuan3-70B. This illustrates that GRPO-style reinforcement learning markedly improves the parameter efficacy of the base model, allowing it to internalise complex financial reasoning processes that generally necessitate higher-capacity architectures.
Question Topic (CFA)
Orig.
Rev.
\Delta
n
Question Topic (CPA)
Orig.
Rev.
\Delta
n
Corporate Finance
50.00
90.00
+40.00
10
Accounting
6.74
66.67
+59.93
89
Derivatives Investment
53.33
88.00
+34.67
45
Auditing
13.64
83.33
+69.70
66
Economics
53.33
60.00
+6.67
15
Economic Law
6.67
91.89
+85.22
60
Equity Investment
62.71
86.54
+23.83
59
Tax Law
7.02
71.43
+64.41
57
Financial Statement Analysis
39.53
81.48
+41.95
43
Wealth Management
0.00
52.78
+52.78
32
Fixed Income Investment
32.65
85.71
+53.06
49
Other Investments
57.45
87.65
+30.21
94
Portfolio Management
75.00
100.00
+25.00
4
Quantitative Analysis
72.34
88.89
+16.55
47
Overall
54.10
83.65
+29.55
366
Overall
7.57
73.14
+65.57
304
Table 10:
Accuracy (%) of Fin-R1-7B by question topic. Left: English CFA (9 subjects). Right: Chinese CPA (5 subjects).
n
denotes the number of original questions.
Fin-R1-7B exhibits a transformative accuracy surge within the Chinese CPA benchmark, rising from a near-random baseline of 7.57% to 73.14% (
\Delta
+65.57%). Most notably, in Economic Law, the model achieves a peak accuracy of 91.89%, surpassing several significantly larger architectures. This trajectory suggests that reasoning-oriented fine-tuning generalizes effectively across linguistic boundaries, even when navigating localized regulatory and legal frameworks.
As detailed in Table
2
, Fin-R1-7B demonstrates substantial advancements in quantitative reasoning. Performance in Complex Calculation jumped from 26.67% to 100.0%, while Simple Calculation witnessed a gain exceeding 32%. This improvement represents a hallmark of the reasoning model (R1) lineage, where the integration of structured chain-of-thought (CoT) processes—even when constrained to a JSON output format—significantly mitigates arithmetic and logical hallucinations.
The model maintains balanced proficiency across diverse cognitive tasks, specifically within Conceptual Understanding (90.00% in CPA) and Statistical Methods (93.62% in CFA). The consistent improvement (
\Delta
) across all task categories indicates that the reinforcement learning process does not merely optimize for specific keyword triggers but instead fosters a systematic enhancement in professional judgment and deductive integrity.
English CFA
Chinese CPA
Question Type
Orig.
Rev.
\Delta
n
Orig.
Rev.
\Delta
n
Conceptual Understanding
69.35
76.74
+7.39
62
7.56
90.00
+82.44
119
Simple Calculation
50.00
82.12
+32.12
30
7.32
71.70
+64.38
41
Complex Calculation
26.67
100.00
+73.33
15
0.00
–
–
2
Comprehensive Judgment
57.43
84.88
+27.45
101
5.75
72.73
+66.98
87
Knowledge Application
38.37
75.00
+36.63
86
14.29
66.67
+52.38
7
Statistical Methods
61.43
93.62
+32.19
70
10.42
70.00
+59.58
48
Overall
54.10
83.65
+29.55
364
7.57
73.14
+65.57
304
Table 11:
Accuracy (%) of Fin-R1-7B by question type across English CFA and Chinese CPA examinations.
Robustness Analysis under Adversarial Masking.
Fin-R1-7B demonstrates significant resilience in adversarial settings where the correct response is “None of the Above” (NOTA), achieving accuracies of 84.5% in the CFA and 77.7% in the CPA. Unlike general-purpose models that often exhibit a nearest-neighbor bias—the tendency to select the numerical option closest to an erroneous internal calculation—Fin-R1-7B maintains high reasoning consistency by actively rejecting plausible distractors through self-verification. The strong alignment between its high mean confidence (
\mu>90.0
) and actual accuracy indicates a well-calibrated professional judgment likely fostered by GRPO reinforcement learning. This training process appears to effectively penalize logical shortcuts, enabling the model to identify specific flaws in distractor logic, such as the failure to account for tax-exempt income deductions. These findings highlight the parameter efficiency of specialized reasoning models, as the 7B architecture develops robust, logic-driven heuristics that rival the performance of much larger general architectures.
Benchmark Subset
Total
Answer Acc.
Reasoning Acc.
Mean Conf.
Revised English CFA (Masked)
367
84.5%
42.5%
93.8
Revised Chinese CPA (Masked)
175
77.7%
41.1%
90.3
Table 12:
Fin-R1-7B performance on NOTA masked questions: Answer vs. Reasoning consistency.
C.7
CFGPT2-7B Configuration
Model Source.
TongjiFinLab/CFGPT2-7B
10
10
10
https://huggingface.co/TongjiFinLab/CFGPT2-7B
(
15
)
CFGPT2 is a Chinese financial assistant developed by Tongji University, built upon the InternLM-7B base. It is pre-trained and fine-tuned on a massive corpus of Chinese financial data. We use a single NVIDIA RTX 4090 (24GB) with BF16 precision.
Chat Teamplate.
CFGPT2 uses the InternLM-style
model.chat()
method. The official example uses an empty
meta_instruction
, but the interface supports passing
system-level instructions:
response, history = model.chat(
tokenizer=tokenizer,
query=user_prompt,
history=[],
max_new_tokens=1024,
do_sample=False,
repetition_penalty=1.1,
meta_instruction=system_prompt
)
Figure 8:
CFGPT2-7B inference code (InternLM-style).
Detailed Results.
The evaluation of CFGPT2-7B (Table
13
and
14
) highlights the significant impact of [Optimization Method] on a domain-specific 7B model.
The original model (
Orig.
) did almost nothing in the Chinese CPA track (2.96%), which shows that it was very out of line with professional Chinese financial norms. But the amended version (
Rev.
) jumped to 49.71%, showing that fine-tuning can quickly bring out hidden financial information.
Like bigger reasoning models, CFGPT2-7B got a 100% accuracy in English
Complex Calculation
after revision. The increases in
Simple Calculation
were more than 38% and 44% across both benchmarks.
Question Topic (CFA)
Orig.
Rev.
\Delta
n
Question Topic (CPA)
Orig.
Rev.
\Delta
n
Corporate Finance
60.00
90.00
+30.00
10
Accounting
2.25
60.00
+57.75
89
Derivatives Investment
31.11
76.00
+44.89
45
Auditing
7.58
60.00
+52.42
66
Economics
53.33
60.00
+6.67
15
Economic Law
1.67
45.95
+44.28
60
Equity Investment
59.32
78.85
+19.53
59
Tax Law
1.75
40.48
+38.73
57
Financial Statement Analysis
53.49
66.67
+13.18
43
Wealth Management
0.00
47.22
+47.22
32
Fixed Income Investment
36.73
57.14
+20.41
49
Other Investments
50.00
77.78
+27.78
94
Portfolio Management
75.00
100.00
+25.00
4
Quantitative Analysis
42.55
73.33
+30.78
47
Overall
47.54
73.57
+26.03
366
Overall
2.96
49.71
+46.75
304
Table 13:
Accuracy (%) of CFGPT2-7B by question topic. Left: English CFA (9 subjects). Right: Chinese CPA (5 subjects).
n
denotes the number of original questions.
The model is quite good in CFA’s
Corporate Finance
(
90.00%
) and
Portfolio Management
(100%). In CPA, it scored 80.00% in
Statistical Methods
, making it a great candidate for quantitative financial jobs even though it has fewer parameters.
English CFA
Chinese CPA
Question Type
Orig.
Rev.
\Delta
n
Orig.
Rev.
\Delta
n
Conceptual Understanding
56.45
76.74
+20.29
62
2.52
40.00
+37.48
119
Simple Calculation
30.00
68.16
+38.16
30
4.88
49.06
+44.18
41
Complex Calculation
40.00
100.00
+60.00
15
0.00
–
–
2
Comprehensive Judgment
60.40
80.23
+19.83
101
2.30
49.49
+47.19
87
Knowledge Application
31.40
75.00
+43.60
86
0.00
0.00
0.00
7
Statistical Methods
48.57
76.60
+28.03
70
4.17
80.00
+75.83
48
Overall
47.54
73.57
+26.03
364
2.96
49.71
+46.75
304
Table 14:
Accuracy (%) of CFGPT2-7B by question type across English CFA and Chinese CPA examinations.
Robustness Analysis under Adversarial Masking.
CFGPT2-7B exhibits a critical
reasoning-selection gap
, where its moderate
Answer Accuracy
(75.2% in CFA; 54.9% in CPA) is starkly contradicted by its near-complete collapse in
Reasoning Accuracy
(
\sim
10%). This suggests that the model’s correct outputs are largely driven by stochastic output heuristics rather than deterministic derivation. Unlike Fin-R1-7B, CFGPT2-7B lacks a functional verification closed-loop (i.e., the logical gate: “Calculated Result
\neq
Candidates
\rightarrow
Select NOTA”), often leading to a correct label despite a fallacious reasoning path.
The performance of CFGPT2-7B is characterized by unstable hallucination-driven hits. Qualitative traces indicate that even when the model correctly identifies the NOTA mask, its CoT frequently contains factual distortions or logical non-sequiturs. This implies that financial domain knowledge has not been internalized into a robust reasoning faculty but remains at the level of probabilistic surface matching.
Despite its logical invalidity, CFGPT2-7B maintains an average confidence score of
\mu>81.0
. This severe miscalibration represents a significant risk in financial applications; the model exhibits a “blind certainty,” providing authoritative yet erroneous justifications.
Benchmark Subset
Total
Answer Acc.
Reasoning Acc.
Mean Conf.
Revised English CFA (Masked)
367
75.2%
8.5%
82.6
Revised Chinese CPA (Masked)
175
54.9%
11.4%
81.0
Table 15:
CFGPT2-7B performance on NOTA masked questions: Answer vs. Reasoning gap.
C.8
DISC-FinLLM-13B Configuration
Model Source.
Go4miii/DISC-FinLLM
11
11
11
https://huggingface.co/Go4miii/DISC-FinLLM
(
3
)
DISC-FinLLM is a specialised financial large language model fine-tuned on the Baichuan-13B base. It is worth noting that DISC-FinLLM is primarily engineered for multi-turn financial consulting dialogues rather than single-turn MCQ tasks. We use a virtual GPU (vGPU-32GB) with FP16 precision and greedy decoding.
Chat Template.
DISC-FinLLM uses the Baichuan-style
model.chat()
method, which does not support a separate system role. To incorporate our evaluation instructions, we concatenate the system prompt with the user query into a single user message:
full_content = f"{system_prompt}\n\n
{user_content}"
messages = [{"role": "user", "content":
full_content}]
response = model.chat(tokenizer, messages)
Figure 9:
DISC-FinLLM inference with merged system prompt.
Detailed Results.
The evaluation results for DISC-FinLLM (Table
16
and
17
) provide insights into the cross-task generalisation of dialogue-specialised financial models
As a model primarily optimised for multi-turn financial consulting, DISC-FinLLM exhibited relatively low baseline accuracy in both the English CFA (13.11%) and Chinese CPA (4.61%) tracks. This suggests that the instruction-tuning for conversational equity does not naturally translate into the rigorous logical deduction required for professional MCQs.
Although the revised version (
Rev.
) showed improvement—climbing to 19.89% in CFA and 19.43% in CPA—the gains are significantly more modest compared to reasoning-dense models like Fin-R1. Notably, in several CFA subjects such as
Economics
and
Equity Investment
, the model even experienced performance regression (
\Delta<0
), indicating a potential instruction interference where revised prompts might conflict with its conversational priors.
The question-type breakdown reveals that DISC-FinLLM-13B struggles with Statistical Methods and Quantitative Analysis, often scoring near or below the expected value of random guessing. While it showed a minor breakthrough in CPA
Conceptual Understanding
(rising from 4.20% to 30.00%), its overall performance trajectory confirms that without specific reinforcement learning for reasoning paths, dialogue-centric models remain insufficient for autonomous high-stakes financial examination tasks.
Question Topic (CFA)
Orig.
Rev.
\Delta
n
Question Topic (CPA)
Orig.
Rev.
\Delta
n
Corporate Finance
30.00
40.00
+10.00
10
Accounting
4.49
26.67
+22.18
89
Derivatives Investment
20.00
28.00
+8.00
45
Auditing
4.55
26.67
+22.12
66
Economics
13.33
0.00
–13.33
15
Economic Law
6.67
18.92
+12.25
60
Equity Investment
20.34
13.46
–6.88
59
Tax Law
5.26
14.29
+9.03
57
Financial Statement Analysis
11.63
22.22
+10.59
43
Wealth Management
0.00
13.89
+13.89
32
Fixed Income Investment
14.29
14.29
0.00
49
Other Investments
10.64
23.46
+12.82
94
Portfolio Management
0.00
33.33
+33.33
4
Quantitative Analysis
0.00
6.67
+6.67
47
Overall
13.11
19.89
+6.78
366
Overall
4.61
19.43
+14.82
304
Table 16:
Accuracy (%) of DISC-FinLLM-13B by question topic. Left: English CFA (9 subjects). Right: Chinese CPA (5 subjects).
n
denotes the number of original questions.
English CFA
Chinese CPA
Question Type
Orig.
Rev.
\Delta
n
Orig.
Rev.
\Delta
n
Conceptual Understanding
8.06
18.60
+10.54
62
4.20
30.00
+25.80
119
Simple Calculation
23.33
20.67
–2.66
30
4.88
15.09
+10.21
41
Complex Calculation
6.67
25.00
+18.33
15
0.00
–
–
2
Comprehensive Judgment
22.77
22.09
–0.68
101
5.75
20.20
+14.45
87
Knowledge Application
10.47
12.50
+2.03
86
14.29
0.00
–14.29
7
Statistical Methods
2.86
14.89
+12.03
70
2.08
30.00
+27.92
48
Overall
13.11
19.89
+6.78
364
4.61
19.43
+14.82
304
Table 17:
Accuracy (%) of DISC-FinLLM-13B by question type across English CFA and Chinese CPA examinations.
Robustness Analysis under Adversarial Masking.
The erratic performance and diminished confidence ratings, particularly in English CFA, suggest that DISC-FinLLM-13B is acutely susceptible to hostile prompt alterations, lacking the ability to generalise outside its conversational tuning distribution. The deterioration of reasoning accuracy indicates that the model lacks the structural logic necessary to recognise absent information, instead depending on language templates that are readily undermined in NOTA situations.
Benchmark Subset
Total
Answer Acc.
Reasoning Acc.
Mean Conf.
Revised English CFA (Masked)
367
12.26%
2.18%
24.20
Revised Chinese CPA (Masked)
175
20.00%
18.86%
65.19
Table 18:
DISC-FinLLM-13B performance on NOTA masked questions: Answer vs. Reasoning gap.
C.9
FinGPT Configuration
Model Source.
AI4Finance-Foundation/FinGPT
(
26
)
.
12
12
12
https://huggingface.co/FinGPT
Due to significant technical constraints in cross-lingual adaptation, we utilised the
Falcon-7B-based variant
of FinGPT to evaluate the English CFA tracks (Original and Revised). Our preliminary assessments of other variants revealed critical limitations: (1) Tokenization Issues in early Llama-2/Falcon versions caused severe character corruption in Chinese; (2) Reasoning Dissociation in the Qwen-based variant, where LoRA adapters appeared to interfere with the base model’s generative consistency in zero-shot professional domains.
Chat Template.
FinGPT utilizes an explicit
Instruction/Input/Answer
format. To maintain consistency with its training distribution, we mapped evaluation system prompts to the
Instruction
field (Figure
10
).
# FinGPT-Falcon-7B inference logic
prompt = f"Instruction: {system_prompt}\n
Input: {user_prompt}\n
Answer: "
Figure 10:
FinGPT-Falcon-7B inference template for English CFA tasks.
Benchmark Subset
Total
Answer Acc.
Reasoning Acc.
Mean Conf.
Revised English CFA (Masked)
367
3.5%
0.0%
50.3
Table 19:
FinGPT-7B performance on NOTA masked questions.
Detailed Results.
As shown in Tables
20
and
20
, FinGPT-7B achieved an overall accuracy of only
2.19%
(Original) and 3.54% (Revised), significantly below random guessing. Qualitative inspection revealed that the model frequently outputs sentiment markers (e.g., “positive/negative”) or jumbled characters instead of logical steps. This confirms that its fine-tuning distribution—highly specialized for sentiment classification—severely compromises its capacity for multi-step financial deduction.
(a)
Accuracy (%) by Topic
Topic
Orig.
Rev.
\Delta
n
Corp. Fin.
0.00
0.00
0.00
10
Derivatives
6.67
4.00
–2.67
45
Economics
0.00
0.00
0.00
15
Equity
1.69
3.85
+2.16
59
FSA
2.33
3.70
+1.37
43
Fixed Inc.
0.00
0.00
0.00
49
Other Inv.
3.19
3.70
+0.51
94
Portfolio
0.00
33.33
+33.33
4
Quant.
0.00
2.22
+2.22
47
Overall
2.19
3.54
+1.35
366
(b)
Accuracy (%) by Question Type
Question Type
Orig.
Rev.
\Delta
n
Conceptual
1.61
6.98
+5.37
62
Simple Calc.
6.67
2.79
–3.88
30
Complex Calc.
0.00
0.00
0.00
15
Comp. Judgment
1.98
4.65
+2.67
101
Know. App.
0.00
0.00
0.00
86
Stat. Methods
4.29
2.13
–2.16
70
Overall
2.19
3.54
+1.35
364
Table 20:
Performance of FinGPT-7B on English CFA Benchmark.
Robustness Analysis under Adversarial Masking.
FinGPT-7B undergoes a total functional collapse in NOTA-masked scenarios (Table
19
). The near-zero reasoning accuracy and stagnant confidence score (
\sim
50.0) reveal an absolute lack of
logical plasticity
. Rather than identifying missing information, the model reverts to fixed templates, proving that sentiment-optimized models remain inherently fragile in adversarial professional environments.
General Models
Financial Models
GPT-5.1-m
Gem-2.5
Cl-3.5
DS-V3
Qwen3
XY-70B
Fin-R1
DJ-32B
FL-8B
FQ-7B
CFGPT2
DISC
Question Type
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
English CFA
Conceptual
82.14
90.12
80.23
89.45
79.87
91.23
75.12
87.98
80.45
90.56
77.42
79.07
69.35
76.74
80.65
86.05
40.32
60.47
51.61
67.44
56.45
76.74
8.06
18.60
Simple Calc.
78.56
87.23
77.89
86.12
76.34
88.76
72.45
85.34
77.23
87.89
36.67
82.12
50.00
82.12
66.67
87.15
23.33
53.63
36.67
62.57
30.00
68.16
23.33
20.67
Complex Calc.
75.23
86.45
74.12
85.78
73.89
87.12
70.67
84.23
74.56
86.34
33.33
100.00
26.67
100.00
60.00
100.00
13.33
75.00
26.67
100.00
40.00
100.00
6.67
25.00
Comp. Judg.
81.45
89.78
79.67
88.23
78.23
90.12
74.56
86.89
79.12
89.45
62.38
88.37
57.43
84.88
84.16
89.53
41.58
62.79
43.56
61.63
60.40
80.23
22.77
22.09
Knowledge App.
79.87
88.45
78.12
87.12
77.56
89.34
73.23
85.67
78.45
88.78
40.70
75.00
38.37
75.00
44.19
87.50
24.42
62.50
19.77
50.00
31.40
75.00
10.47
12.50
Stats Methods
83.45
91.67
81.23
90.12
80.56
92.34
76.89
88.45
81.67
91.23
71.43
93.62
61.43
93.62
90.00
91.49
35.71
57.45
44.29
80.85
48.57
76.60
2.86
14.89
All
80.59
89.01
78.99
88.19
77.81
89.62
73.55
86.34
78.42
88.83
58.47
84.74
54.10
83.65
72.95
88.28
33.61
57.49
38.52
65.40
47.54
73.57
13.11
19.89
Chinese CPA
Conceptual
82.34
69.12
75.23
86.45
71.12
82.34
82.89
81.67
78.12
93.89
10.92
80.00
7.56
90.00
15.13
90.00
9.24
40.00
8.40
30.00
2.52
40.00
4.20
30.00
Simple Calc.
80.67
67.89
73.45
84.12
69.34
80.12
81.23
79.89
76.45
91.23
9.76
64.15
7.32
71.70
4.88
62.26
4.88
30.19
9.76
18.87
4.88
49.06
4.88
15.09
Complex Calc.
79.23
66.45
72.12
82.78
68.12
78.89
80.45
78.23
75.23
89.67
0.00
–
0.00
–
0.00
–
0.00
–
0.00
–
0.00
–
0.00
–
Comp. Judg.
81.45
68.56
74.23
85.12
70.23
81.23
81.89
80.56
77.12
92.45
10.34
71.72
5.75
72.73
13.79
84.85
8.05
25.25
4.60
34.34
2.30
49.49
5.75
20.20
Knowledge App.
80.12
67.23
73.12
83.45
69.12
79.67
81.12
79.12
76.23
90.89
0.00
66.67
14.29
66.67
0.00
100.00
0.00
33.33
14.29
33.33
0.00
0.00
14.29
0.00
Stats Methods
82.67
69.89
75.67
86.89
71.45
82.89
83.12
82.12
78.56
94.23
16.67
60.00
10.42
70.00
20.83
70.00
8.33
30.00
6.25
20.00
4.17
80.00
2.08
30.00
All
80.81
68.45
73.42
84.80
69.37
80.46
81.06
80.00
76.32
92.00
11.18
69.14
7.57
73.14
13.82
77.71
7.89
28.00
7.24
28.57
2.96
49.71
4.61
19.43
Table 21:
Accuracy (%) across six question types
for English CFA (top) and Chinese CPA (bottom).
O
= original;
R
= revised. For each row:
Best-O
,
Best-R
,
Worst-O
,
Worst-R
; best per row is
bolded
. DJ-32B = DianJin-R1-32B; FL-8B = Finance-Llama-8B; FQ-7B = Finance-Qwen-7B.
General Models
Financial Models
GPT-5.1-m
Gem-2.5
Cl-3.5
DS-V3
Qwen3
XY-70B
Fin-R1
DJ-32B
FL-8B
FQ-7B
CFGPT2
DISC
Topic
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
O
R
English CFA
Corp. Finance
82.50
90.00
91.25
100.00
81.88
90.00
82.19
90.00
83.75
91.00
60.00
90.00
50.00
90.00
80.00
90.00
60.00
90.00
40.00
70.00
60.00
90.00
30.00
40.00
Derivatives
80.00
88.00
87.20
96.00
83.68
92.00
80.08
88.00
82.80
90.00
53.33
92.00
53.33
88.00
62.22
96.00
28.89
64.00
28.89
68.00
31.11
76.00
20.00
28.00
Economics
81.82
90.00
81.82
90.00
81.82
90.00
81.82
90.00
81.82
89.09
53.33
80.00
53.33
60.00
60.00
90.00
13.33
40.00
46.67
80.00
53.33
60.00
13.33
0.00
Equity Inv.
79.49
88.46
79.49
88.46
79.49
88.46
76.28
84.62
78.85
87.18
57.63
86.54
62.71
86.54
76.27
88.46
37.29
53.85
40.68
67.31
59.32
78.85
20.34
13.46
FSA
79.66
88.89
79.66
88.89
79.66
88.89
79.66
88.89
79.66
89.63
65.12
88.89
39.53
81.48
76.74
88.89
34.88
48.15
41.86
48.15
53.49
66.67
11.63
22.22
Fixed Income
71.43
78.57
71.43
78.57
71.43
78.57
71.43
78.57
71.43
76.79
42.86
78.57
32.65
85.71
46.94
78.57
18.37
50.00
24.49
57.14
36.73
57.14
14.29
14.29
Other Inv.
79.75
88.89
77.78
86.42
79.75
88.89
76.54
85.19
78.02
86.42
62.77
86.42
57.45
87.65
79.79
88.89
41.49
69.14
36.17
71.60
50.00
77.78
10.64
23.46
Portfolio Mgt
90.91
100.00
90.91
100.00
90.91
100.00
90.91
100.00
90.91
100.00
75.00
100.00
75.00
100.00
100.00
100.00
50.00
100.00
50.00
100.00
75.00
100.00
0.00
33.33
Quant Analysis
80.00
88.89
80.00
88.89
82.09
91.11
78.21
86.67
78.61
87.78
65.96
86.67
72.34
88.89
89.36
88.89
31.91
64.44
57.45
82.22
42.55
73.33
0.00
6.67
All
80.59
89.01
78.99
88.19
77.81
89.62
73.55
86.34
78.42
88.83
58.47
84.74
54.10
83.65
72.95
88.28
33.61
57.49
38.52
65.40
47.54
73.57
13.11
19.89
Chinese CPA
Accounting
33.33
36.67
75.76
83.33
72.73
80.00
66.67
73.33
69.32
85.00
4.49
63.33
6.74
66.67
7.87
60.00
7.87
33.33
7.87
20.00
2.25
60.00
4.49
26.67
Auditing
84.85
93.33
90.91
100.00
87.88
96.67
84.85
93.33
88.64
97.50
25.76
96.67
13.64
83.33
36.36
93.33
4.55
33.33
4.55
60.00
7.58
60.00
4.55
26.67
Economic Law
66.28
72.97
78.57
86.49
76.19
83.78
81.08
89.19
73.81
90.63
6.67
78.38
6.67
91.89
8.33
89.19
11.67
21.62
8.33
13.51
1.67
45.95
6.67
18.92
Tax Law
67.11
73.81
75.76
83.33
69.23
76.19
80.10
88.10
71.43
85.94
15.79
66.67
7.02
71.43
8.77
88.10
10.53
14.29
8.77
23.81
1.75
40.48
5.26
14.29
Wealth Mgt.
45.45
50.00
58.06
63.89
60.61
66.67
50.51
55.56
54.55
67.50
0.00
44.44
0.00
52.78
3.12
55.56
3.12
41.67
6.25
30.56
0.00
47.22
0.00
13.89
All
80.81
68.45
73.42
84.80
69.37
80.46
81.06
80.00
76.32
92.00
11.18
69.14
7.57
73.14
13.82
77.71
7.89
28.00
7.24
28.57
2.96
49.71
4.61
19.43
Table 22:
Accuracy (%) across financial topics
for English CFA (top) and Chinese CPA (bottom).
O
= original;
R
= revised. For each row:
Best-O
,
Best-R
,
Worst-O
,
Worst-R
; best per row is
bolded
. DJ-32B = DianJin-R1-32B; FL-8B = Finance-Llama-8B; FQ-7B = Finance-Qwen-7B.
General Models
Financial Models
Model
Chinese
English
Average
Model
Chinese
English
Average
Gemini 2.5 Flash
48.78
72.38
60.58
Fin-R1-7B
77.10
84.50
81.10
Claude 3.5 Sonnet
37.36
70.11
53.73
XuanYuan3-70B
61.70
84.70
73.20
Qwen3-Max
44.85
59.94
52.39
DianJin-R1-32B
77.71
88.28
83.00
GPT-5.1-mini
30.64
68.23
49.44
FL-8B
28.57
56.95
42.76
DeepSeek V3
36.21
56.87
46.54
FQ-7B
48.00
62.67
55.34
CFGPT2-7B
54.90
75.20
65.05
DISC-FinLLM-13B
20.00
12.26
16.13
Avg.
39.57
65.51
52.54
Avg.
52.57
66.37
59.51
Table 23:
Cross-comparison of model accuracy (%) on None of The Above questions compared to Revised accuracy.
Green
indicates NOTA accuracy higher than Revised;
red
indicates lower;
gray
indicates NOTA equals Revised. FL-8B = Finance-Llama-8B; FQ-7B = WiroAI Finance-Qwen-7B.