AI and the Law: Evaluating ChatGPT’s Performance in Legal Classification Journal: To be determined
Paweł Weichbroth* · 2025
Read AI and the Law: Evaluating ChatGPT’s Performance in Legal Classification Journal: To be determined by Paweł Weichbroth* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12193.txt.
Introduction
Pawel Weichbroth tests whether ChatGPT can sort legal cases under the Polish Penal Code. Much talk has covered the tool in law, but little work had tried it on Polish text, so he set up cases for the tool to class as one type or another.
The tool sorted all test cases to the right side in the simple two way task, and review found its cited rules and text fit each case well. He concludes the tool could help people with less legal skill review proof and find the right rules. The study is small and early.
Catalog record
| CREATOR | Paweł Weichbroth* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12193
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI and the Law: Evaluating ChatGPT’s Performance in Legal Classification Journal: To be determined Paweł Weichbroth* Abstract The use of ChatGPT to analyze and classify evidence in criminal proceedings has been a topic of ongoing discussion. However, to the best of our knowledge, this issue has not been studied in the context of the Polish language. This study addresses this research gap by evaluating the effectiveness of ChatGPT in classifying legal cases under the Polish Penal Code. The results show excellent binary classification accuracy, with all positive and negative cases correctly categorized. In addition, a qualitative evaluation confirms that the legal basis provided for each case, along with the relevant legal content, was appropriate. The results obtained suggest that ChatGPT can effectively analyze and classify evidence while applying the appropriate legal rules. In conclusion, ChatGPT has the potential to assist interested parties in the analysis of evidence and serve as a valuable legal resource for individuals with less experience or knowledge in this area. 1 Introduction The law is a fundamental part of our daily lives [ 1 ] , and it is almost inevitable that everyone will interact with the justice system at some point. Legal professionals, such as attorneys, prosecutors, and judges, apply the law according to their specific roles, including advocating for clients, investigating criminal activity, and adjudicating cases. Meanwhile, the other parties involved, especially those without prior expertise and relevant knowledge, may find themselves confused in legal proceedings as defendants or plaintiffs, whether they have been accused of violating the law or their rights have been violated or their rights have been violated by others. Although the principle of ignorantia iuris nocet (“ignorance of the law is harmful” [ 2 ] ) does not always apply here, interpreting and analyzing legal regulations remains a daunting task for most, or even in some cases barely feasible. The advent of artificial intelligence (AI) has been a game changer in many fields [ 3 , 4 , 5 ] , including law [ 6 ] . A recent report, published by Thomson Reuters in 2024 [ 7 ] , highlights how AI is reshaping the legal profession by automating routine tasks and increasing lawyer productivity. AI-powered tools streamline document review, legal research, and contract analysis, allowing lawyers to focus on higher-value work. Note that that AI could also save lawyers four hours per week while generating an additional $100,000 in billable time per lawyer per year. On the other hand, AI-based tools can help people with little or no legal education understand legal language by simplifying complex legal terminology into plain, easy-to-understand text [ 8 ] . Such tools can break down statutes [ 9 ] , contracts [ 10 ] , and court decisions [ 11 ] into concise summaries so that key implications can be grasped. In addition, AI can provide contextual explanations and real-world examples to illustrate how legal principles apply to specific situations [ 12 ] . By making legal information more accessible, AI empowers individuals to make informed decisions before seeking professional legal assistance [ 13 ] . One such AI-based tool is ChatGPT . Developed by OpenAI [ 14 ] , an AI research and deployment company [ 15 ] , it is a state-of-the-art conversational agent built upon the Generative Pre-trained Transformer (GPT) architecture [ 16 ] . Its development represents a significant milestone in the field of AI [ 17 ] , particularly in natural language processing (NLP) [ 18 ] . The evolution of ChatGPT can be traced back to earlier iterations of GPT models [ 19 ] , with each version introducing advancements in language understanding and generation [ 20 ] . The release of ChatGPT has garnered considerable attention [ 21 ] , leading to discussions about its applications, limitations , and potential future developments [ 22 , 23 , 24 ] . With these opportunities and challenges in mind, it is important to carefully evaluate the role of ChatGPT in the legal field. Understanding its strengths and limitations can help determine its potential use in a specific context. That being said, this study aims to explore the potential and limitations of ChatGPT for the Polish language by analyzing its effectiveness in classifying legal cases in terms of the Penal Code. By conducting a qualitative and quantitative analysis of ChatGPT’s performance in legal tasks, this study seeks to provide insights into the extent to which AI is able to correctly infer, classify and explain evidence written in the form of short text notes. The rest of the paper is structured as follows. Section 2 discusses related work. Section 3 describes the research methodology. Section 4 presents the results evaluation. Section 5 outlines the further discussion on the research, in particular idicating its contribution and limitations. Section 6 concludes the paper. 2 Related Work There is no doubt that the emergence of advanced tools such as ChatGPT has brought new opportunities to enhance law enforcement [ 25 ] , risk management [ 26 ] , improve efficiency [ 27 ] , and expand operational capacity [ 28 ] . The embedded AI-driven capabilities can be leveraged across multiple tasks that have been explored in different scenarios and tailored settings. Perlman [ 29 ] demonstrated the use cases based on prompting ChatGPT with questions to test its capabilities in document generation, legal analysis, information and research. The author stated that the obtained responses “were imperfect and at times problematic", while the generated documents were incomplete. Nevertheless, attorneys may be willing to prepare initial drafts of complex legal instruments, easily adopting the firm’s style and incorporating its substantive knowledge. Interestingly, one of the author’s other closing observations was that “AI will not eliminate the need for lawyers, but it does portend the end of lawyering as we know it". We also agree that the AI-driven revolution in legal practice is undeniable, and resistance seems futile. Armstrong [ 30 ] found that ChatGPT-4 was able, although sometimes imperfectly, to engage in legal reasoning about law and factual data, answer questions about legal opinions or documents, and even analyze, draft and summarize legal cases, write motions, draft patents, and write reports. However, in some cases ChatGPT performed particularly poorly on this set of prompts, occasionally producing unsatisfactory results with some serious errors. To conclude, since ChatGPT presented fiction as fact (and with confidence), legal professionals should not rely on ChatGPT responses, and thus it is not a substitute for careful reading and case analysis. At the University of Minnesota Law School, Choi et al. [ 31 ] tested ChatGPT to generate answers to four real exams, including Constitutional Law: Federalism and Separation of Powers, Employee Benefits, Taxation, and Torts. In all four courses, over 95 multiple-choice questions and 12 essay questions, its performance was on average at the level of a C+ student, achieving a low but passing grade. In general, ChatGPT performed better on the essay sections of the exams than on the multiple choice questions. Note that if this performance were consistent throughout law school, its grades would be sufficient for a student to graduate. Jan and others [ 32 ] asked a stirring question: ChatGPT as an artificial lawyer? In