Assessing the Capability of LLMs in Solving POSCOMP Questions

Cayo Viegas 3 , Rohit Gheyi 3 , Márcio Ribeiro 2 · 2025

Read Assessing the Capability of LLMs in Solving POSCOMP Questions by Cayo Viegas 3 , Rohit Gheyi 3 , Márcio Ribeiro 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20338.txt.

Introduction

Cayo Viegas, Rohit Gheyi, and Márcio Ribeiro test chat models on a tough exam for computer science hopefuls in Brazil, in a study from 2025. The exam, POSCOMP, gates entry to top graduate courses. It mixes hard text questions with image tasks, which makes it a stern test of broad skill.

Four models took the 2022 and 2023 exams first. The best of them, ChatGPT-4, got 57 of 69 right in 2022 and beat all human takers in 2023. All four did far better on text than on images. The team then ran four newer models on three years of exams. The new set beat both the mean and the top human scores in each year. The work shows fast growth, with image sense still the weak flank.

Catalog record

CREATORCayo Viegas 3 , Rohit Gheyi 3 , Márcio Ribeiro 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing the Capability of LLMs in Solving POSCOMP Questions Cayo Viegas 3 , Rohit Gheyi 3 , Márcio Ribeiro 2 Abstract Recent advancements in Large Language Models (LLMs) have significantly expanded the capabilities of artificial intelligence in natural language processing tasks. Despite this progress, their performance in specialized domains such as computer science remains relatively unexplored. Understanding the proficiency of LLMs in these domains is critical for evaluating their practical utility and guiding future developments. The POSCOMP, a prestigious Brazilian examination used for graduate admissions in computer science promoted by the Brazlian Computer Society (SBC), provides a challenging benchmark. This study investigates whether LLMs can match or surpass human performance on the POSCOMP exam. Four LLMs – ChatGPT-4, Gemini 1.0 Advanced, Claude 3 Sonnet, and Le Chat Mistral Large – were initially evaluated on the 2022 and 2023 POSCOMP exams. The assessments measured the models’ proficiency in handling complex questions typical of the exam. LLM performance was notably better on text-based questions than on image interpretation tasks. In the 2022 exam, ChatGPT-4 led with 57 correct answers out of 69 questions, followed by Gemini 1.0 Advanced (49), Le Chat Mistral (48), and Claude 3 Sonnet (44). Similar trends were observed in the 2023 exam. ChatGPT-4 achieved the highest performance, surpassing all students who took the POSCOMP 2023 exam. LLMs, particularly ChatGPT-4, show promise in text-based tasks on the POSCOMP exam, although image interpretation remains a challenge. Given the rapid evolution of LLMs, we expanded our analysis to include more recent models – o1, Gemini 2.5 Pro, Claude 3.7 Sonnet, and o3-mini-high – evaluated on the 2022–2024 POSCOMP exams. These newer models demonstrate further improvements and consistently surpass both the average and top-performing human participants across all three years. I Introduction Context. The POSCOMP [ 1 ] is a prestigious assessment designed to test the knowledge of prospective computer science graduate students, promoted by the Brazilian Computer Society (SBC). It serves as an entry criterion for many graduate programs across Brazil. Using this exam as a benchmark for evaluating Large Language Models (LLMs) allows for a direct comparison between AI capabilities and human standards, offering valuable insights into the strengths and limitations of current AI models. Recent advancements in LLMs [ 2 , 3 ] have significantly expanded the capabilities of Artificial Intelligence (AI), particularly in natural language processing tasks. These developments have prompted increasing interest in understanding how well LLMs can solve complex problems in specialized domains, such as computer science. Recent studies have examined the performance of LLMs on Brazilian educational assessments. Mendonça [ 4 ] evaluated ChatGPT-4 Vision on Brazil’s undergraduate Computer Science ENADE exam, showing that the model outperformed average human participants, despite struggling with certain aspects of visual reasoning. Similarly, Pires et al. [ 5 ] assessed ChatGPT-4 Vision and ChatGPT-4 Text on the national ENEM exams, finding that text-only prompts with image captions outperformed image-only ones when using few-shot prompting with Chain-of-Thought [ 6 ] . Nunes et al. [ 7 ] conducted a comprehensive analysis of ChatGPT-3.5 and ChatGPT-4 on ENEM questions from 2009–2017 and 2022, revealing that ChatGPT-4 with Chain-of-Thought achieved an impressive 87% accuracy on the 2022 exam. While these studies demonstrate promising results for LLMs in high school and undergraduate-level assessments, little is known about their performance on graduate-level assessments such as POSCOMP. This gap motivates a deeper investigation into how well state-of-the-art LLMs perform in this more advanced context. Study Objectives. In this article, we investigate the ability of state-of-the-art LLMs to solve questions from the POSCOMP exam. We evaluate the performance of four models – ChatGPT-4 [ 8 ] , Gemini 1.0 Advanced [ 9 ] , Claude 3 Sonnet [ 10 ] , and Le Chat Mistral Large [ 11 ] – on the 2022 and 2023 editions of the exam, aiming to determine whether these models can match or surpass human performance in a rigorous academic setting. In light of the rapid evolution of LLMs, we extend this analysis to include more recent frontier models, such as o1, Gemini 2.5 Pro Experimental, Claude 3.7 Sonnet, and o3-mini-high. This expanded evaluation includes the 2024 edition of the POSCOMP and allows us to assess the progress of cutting-edge LLMs across three consecutive years. Main Contributions. Our findings show that ChatGPT-4 consistently outperformed the other models, achieving the highest scores in 2023 and demonstrating particularly strong performance in Mathematics and Computer Science Fundamentals. While Gemini 1.0 and Le Chat Mistral performed reasonably well across various subjects, only ChatGPT-4 exhibited consistent excellence. Claude 3 Sonnet showed strength in explanation and reasoning tasks but struggled in quantitative areas. We also observed significant performance gains in more recent models. Gemini 2.5 Pro Experimental showed substantial improvement over Gemini 1.0, and, along with o3-mini-high, achieved or exceeded 90% accuracy across several topics – surpassing both average and top human participants. In addition to performance benchmarking, our study highlights the practical utility of LLMs in educational assessment design. By examining model responses, problem setters can identify ambiguities, clarify question wording, and refine answer choices – ultimately improving the quality and fairness of exam content. The remainder of the paper is organized as follows. Section II relates our work to previous ones. Section III provides details about the POSCOMP exam. Section IV presents evaluations of LLM performance in answering POSCOMP questions. Section V outlines the threats to the validity of the study. Finally, Section VI concludes our work. II Related Work Pires et al. [ 5 ] evaluated ChatGPT-4 Vision and ChatGPT-4 Text in two Brazilian ENEM exams. Text-only prompts with captions describing images outperformed the image-only version. They used few-shot prompting with Chain-of-Thought [ 6 ] . The LLM struggled most in the Mathematics section. In our work, we evaluated four LLMs on two POSCOMP editions using zero-shot prompts. Additionally, ChatGPT-4 performed well on math questions. Zhang et al. [ 12 ] assessed ChatGPT-3.5, ChatGPT-4, and ERNIE-Bot (including its Turbo version) with questions from the 2010-2022 Chinese University Entrance Exam (GAOKAO), excluding those containing images. Zero-shot prompting and human evaluation were used. Results showed that the models performed well on knowledge-based questions but struggled with specific logical reasoning, math problems, and reading comprehension of longer Chinese texts. In our work, we evaluated four LLMs on two POSCOMP editions using zero-shot prompts, where ChatGPT-4 excelled in math. Guillen-Grima et al. [ 13 ] examined ChatGPT-3.5 and ChatGPT-4’s performance on the 2022 Spanish Medical Residency Entrance Exam, in both the native language and English translations, finding a slightly better performance in English. In our study, we assessed four LLMs on two POSCOMP editions. After using image prompts in Portuguese, we translated the questions into English and saw improved performance across all models. Bommarito and Katz [ 14 ] evaluated ChatGPT-3.5 on the Multistate Bar Examination portion of the National Conference of Bar Examiners’ Bar Exam, a prerequisite for legal practice in the U.S. ChatGPT-3.5 achieved a passing rate and scores comparable to human examinees. In our evaluation, two of the models tested in POSCOMP 2023 scored higher than human examinees. Bommarito et al. [ 15 ] tested ChatGPT-3.5 on two assessments based on the American Institute of Certified Public Accountants’ Uniform CPA Exam Blueprints, a U.S. accounting

Read the complete text