Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis
Md Talha Mohsin · 2025
Read Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis by Md Talha Mohsin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22936.txt.
Introduction
Md Talha Mohsin grades, in a 2025 study, five large language models on questions about company financial reports. Banks and investors increasingly lean on such models to read long filings, yet their steadiness in high-stakes work is poorly mapped. The test used the Business sections of US 10-K filings, the yearly reports firms must file.
Three lenses were used: human judges, automated text-match scores, and behaviour checks across prompts. Humans rated the models on relevance, completeness, clarity, brevity, and accuracy, though the judges agreed only modestly with each other. Scores and stability varied by model, and no model led on every lens. Mohsin reads the gaps as tendencies under set conditions, not proof of general reliability, and urges caution in deployment.
Catalog record
| CREATOR | Md Talha Mohsin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis Md Talha Mohsin Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide variety of Financial Natural Language Processing (FinNLP) tasks. However, systematic comparisons among widely used LLMs remain underexplored. Given the rapid advancement and growing influence of LLMs in financial analysis, this study conducts a thorough comparative evaluation of five leading LLMs, GPT, Claude, Perplexity, Gemini and DeepSeek, using 10-K filings from the ”Magnificent Seven” technology companies. We create a set of domain-specific prompts and then use three methodologies to evaluate model performance: human annotation, automated lexical-semantic metrics (ROUGE, Cosine Similarity, Jaccard), and model behavior diagnostics (prompt-level variance and across-model similarity). The results show that GPT gives the most coherent, semantically aligned, and contextually relevant answers; followed by Claude and Perplexity. Gemini and DeepSeek, on the other hand, have more variability and less agreement. Also, the similarity and stability of outputs change from company to company and over time, showing that they are sensitive to how prompts are written and what source material is used. keywords Large Language Models (LLMs), Natural Language Processing (NLP), Financial Text Analysis, 10-K, AI in Finance, Generative AI, Explainable AI (XAI) 1 Introduction Artificial Intelligence (AI) has rapidly transformed how information is processed, analyzed, and interpreted across a multitude of industries. Among its most groundbreaking developments is the emergence of Large Language Models (LLMs)—a class of AI systems trained on vast corpora of text to generate human-like responses. Recently, LLMs have garnered significant attention in both academia and business ( Bommasani et al. 2022a , Zhao et al. 2023a , Wei et al. 2022a ). These models are being used more and more in professional fields for their effectiveness, accuracy and rapidness. As the need for scalable, low-cost tools for strategy evaluation grows, LLMs are being used to understand financial disclosures, study market sentiment, and put together unstructured data ( Daimi & Iqbal 2024a ). For example, ( Kim et al. 2025b ) states how LLMs can work wonders because they can predict changes in company earnings better than human analysts can, even when there aren’t any clear narratives or industry-specific signals. They can also show how people feel about a company and any possible biases that might affect how investors act and how prices change ( Nakagawa et al. 2024a ). Additionally, LLMs can create financial digests by condensing large amounts of text into clear, actionable insights ( Lazarev & Sedov 2024a ); this leads to better trading methods with higher Sharpe ratios ( Kim et al. 2025b ). In these situations, it is very important to know how to use domain-specific language and make products that can be understood. Because of this, LLMs are now judged not only on their fluency or coherence, but also on their organized, expert-level analysis in complex Financial Natural Language Processing (FinNLP) tasks. A lot of natural language processing tasks, like summarizing, answering questions, and semantic document analysis, have been done very well by LLMs. But not much is known about how different LLMs act in high-stakes, domain-specific situations like company financial disclosures. This is especially worrying in the sense that in the financial world, regulators, institutional investors, and analysts depend on complicated textual disclosures heaivily to make decisions. Transformer-based LLMs, which are pre-trained on huge datasets and fine-tuned to spot complex language patterns ( Kim et al. 2025c ), have sped up recent progress in natural language processing (NLP). In finance, NLP methods have mostly been used for classifying emotions or recognizing entities. However, new research is looking into how well they can be used for semantic reasoning and understanding stories ( Wu et al. 2024a ). Since financial reports are mostly text, it makes sense to use AI-powered tools to analyze them ( Abdaljalil & Bouamor 2021a ). The Securities and Exchange Commission (SEC) requires public companies to file structured textual disclosures, like the 10-K annual report. Important soft data that quantitative predictors miss can be found in these filings ( Lombardo et al. 2024a ). Crucial insights into the performance of a company can be gained through the examination of extensive records, particularly 10-K filings. Due to the fact that these filings encompass qualitative aspects such as strategy, risk exposure, and competitive positioning, they are excellent candidates for advanced natural language processing-based analysis. Financial statements, risk disclosures, and strategy overviews are all the components that are included in 10-K filings, which offer a structured but uncurated perspective of the health of the corporation. They are, nevertheless, notoriously difficult to understand. Both human and machine analysts face difficulties when dealing with them because of their unstructured format and technical jargon. The process of text mining provides a powerful method for extracting value from these disclosures by revealing patterns, sentiment, and relational data across companies and industries ( Kim et al. 2023a ). This study address that gap by suggesting a multi-dimensional evaluation approach to see how five cutting-edge LLMs— ChatGPT-4, Perplexity, Claude 4 Opus, Gemini, and DeepSeek—look at the Business section (Item 1) of 10-K filings from the ”Magnificent 7” tech companies over the last three years. We introduce a Chain-of-Thought (CoT) prompting approach that tells models to behave like financial analysts in order to simulate real-life analytical workflows. Their responses are evaluated using three distinct lenses: human-centric judgment, automated metric-based scoring, and prompt-sensitivity based behavioral analysis. Our work makes the following three contributions: (i) We test how sensitive LLM is to the design and amount of information in prompts, showing patterns of behavior across model structures. (ii) We create a benchmark that can be replicated to test LLMs’ understanding of financial matters by focusing on detailed information rather than numbers, and (iii) We give academics and professionals a direction on how to use or evaluate LLMs in financial situations, mainly for strategic analysis, and information extraction The remainder of this study is structured as follows: Section 2 discusses background, Section 3 provides contextual insights into how LLMs work, Section 4 outlines the data and methodology, Section 5 presents the findings, section 6 offers discussion, and Section 7 concludes the study. 2 Related Work 2.1 Financial Text Analysis Using NLP The systematic, and consistent quality of content analysis has made it a fundamental component of qualitative research for a long time. This is especially true in the field of finance, where content or text analysis has undergone substantial development since the introduction of Natural Language Processing (NLP). By mining massive amounts of financial text, modern natural language processing systems are now able to assist educated investment strategies, track macroeconomic signals, and improve decision-making processes within institutions ( 6 6 ). When new natural language processing (NLP) and information retrieval technologies come into existence, the meaning of ”new” information and the accessibility of it evolved, providing early adopters with a temporary advantage ( Araci 2019a ). These techniques integrate linguistic, statistical, and deep learning methodologies in order to extract value from complicated documents ( Wang et al. 2024a ). Automating the extraction of structured insights from unstructured financial writings is one of the strengths of natural language processing (NLP). According to ( Oyewole et al. 2024a ), these methods