Making Bielik LLM Reason (Better): A Field Report

Adam Trybus, Bartosz Bartnicki, Remigiusz Kinas · 2026

Read Making Bielik LLM Reason (Better): A Field Report by Adam Trybus, Bartosz Bartnicki, Remigiusz Kinas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10640.txt.

Introduction

Adam Trybus, Bartosz Bartnicki, and Remigiusz Kinas report a research program on the reasoning of Bielik, the Polish large language model. The field report walks through stages of work: first benchmarking the model and building the evaluation method, then comparing results with other models. The authors are candid about the limits of the analyses so far. They close with future prospects, framed by a fast-moving, competitive field in which a smaller national model must keep pace.

Catalog record

CREATORAdam Trybus, Bartosz Bartnicki, Remigiusz Kinas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25131
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making Bielik LLM Reason (Better): A Field Report Adam Trybus, Bartosz Bartnicki, Remigiusz Kinas Abstract This paper presents a research program dedicated to evaluating and advancing the reasoning capabilities of Bielik, a Polish large language model. The study describes a number of stages of work: initial benchmarking and creation of evaluation methodology, analyzing of comparative results with other LLMs and outlining of future prospects that take into account the limitations of the analyses conducted so far and aims to keep Bielik in the race give the ever-changing — and competitive — AI landscape. Keywords Bielik LLM \cdot reasoning models \cdot benchmarking 1 Introduction Frontier artificial intelligence models have begun to outperform humans in programming and mathematical competitions, while systems such as AlphaFold have become indispensable tools in the life sciences. The trajectory of scientific research is increasingly shaped by collaboration between human researchers and AI agents. Significant advances have already been achieved in several areas of physics, highlighting the potential of AI-driven approaches. The overall direction of development points toward a transition from predictive AI systems, through partially assistive tools, to fully autonomous and integrated research platforms orchestrated by AI agents. Some of the major figures in the field, argue that Poland has largely missed the last seven years of global AI progress and remains at the bottom of the EU rankings in AI adoption, just ahead of Romania. The country lacks visibility on the global map of major AI research and technology hubs. While Poland’s AI ecosystem is arguably underdeveloped, there is significant progress to be noted as well. One of the most promising projects is the development of Bielik — a large language model that can be competitive on the local market with the major international platforms. The model is under continous development ( 2 , 3 , 1 , 4 , 5 ) and has already been incorporated in at least one commercial solution on a big scale. The reported work is aimed at measuring, comparing and developing the reasoning capabilities of Bielik in order to further increase its competetiveness on the ever-changing AI landscape. We discuss initial research, current efforts, and future prospects. We note that the most recent trend is to go beyond the use of a single model but rather a creation of orchestrated multi-component systems, or pipelines, where LLMs play the role of intermediary among other elements. Those other elements provide functionality unavailable to LLMs, such as strict formal analysis, explainability or repeatability. Our long-horizon goals reflect this trend, also in the context of domain-specific systems e.g. mathematical or legal reasoners. 2 Initial Efforts The work began with the goal of assessing the current reasoning capabilities of Bielik 2.3. The first analyses of reasoning were conducted manually. Its capabilities were compared with the results of state-of-an-art reasoning models. Bielik 2.3 exhibited very limited reasoning abilities — even relatively simple tasks were challenging. Nevertheless Einstein’s Riddles drawn from printed sources proved surprisingly effective: they challenged strong models such as DeepSeek-R1 (March 2025), scaled naturally in difficulty through additional parameters and misleading clues, and demanded iterative reconciliation of premises and intermediate conclusions. The principal risk was the "lost-in-the-middle" effect, arising from the considerable length of such tasks. The three-person research team recognized that manual analysis was inefficient, and thus decided to automate the entire process. As a result, the first automated solution was developed, allowing the addition of tasks and instructions for a metamodel responsible for evaluating the responses. In the first version, ChatGPT-4o served as the “judge,” evaluating both the reasoning process and the final answer. Evaluation was performed on a 1–5 scale, taking into account the correctness and clarity of the response. For the purposes of the project, we outlined a tentative taxonomy of reasoning, and divided the tasks broadly into deterministic (a single well-specified answer) and non-deterministic (requiring interpretation, argumentation analysis, or judgment). In addition, a dozen or so reasoning taxonomies were defined, covering both formal and informal logic, each supplemented with several example tasks. Inductive reasoning tasks —deducing rules from examples—were indispensable. Sequence continuation (e.g., 2, 6, 18, 54, 162, …) or fictional language translation tasks required pattern extraction and rule inference. Induction was a sine qua non condition for general reasoning competence. However, many compelling inductive puzzles were visual rather than textual. Text-only variants often failed to challenge the model meaningfully, although further research might have uncovered stronger candidates. Classic riddles and word puzzles were deterministic and concise, but limited in number and insufficiently demanding in terms of multi-step reasoning. They lacked multi-turn scalability and advanced inferential depth. Non-deterministic reasoning categories introduced additional complexity. Paradox analysis —such as the Barber paradox or the Ship of Theseus—tested conceptual consistency. However, the set of canonical paradoxes was finite, multi-turn potential was limited, and evaluation of solutions was partially subjective. Constructing valid arguments introduced further non-determinism. While multi-turn argument expansion was possible, evaluating the sufficiency and hierarchical structure of premises inevitably required arbitrariness. Attempts to justify justifications risked infinite regress. Critiquing argument structures posed similar challenges. Although models typically performed well in identifying weaknesses in arguments about video games and violence or flawed historical generalizations (e.g., misrepresentations of Einstein’s education), such tasks bordered on writing skill assessment rather than pure reasoning evaluation. Deterministic scoring was difficult. Several additional reasoning categories remained under development: syllogistic reasoning, logical fallacy identification, hypothesis formation, and analogical problem solving . These areas required more systematic task design to ensure objective evaluation criteria. After much trial-and-error and deliberation, the team settled on question types and desirable answer formats. The final dataset included questions from various sub-categories. Most notably, we tested formal reasoning capabilities with simple tasks in propositional and predicate logic. For example: 1 1 1 All the questions were originally phrased in Polish. Example 1: What follows logically from the statements \lnot p\vee\lnot q , p , and q\vee r ? Example 2: Is the statement \forall x,\phi\to\exists x,\phi a logical truth? We also designed questions testing various other modes of reasoning, such as abduction or induction. For example: Example 3: Earlier in the morning, you saw several coworkers go into the office kitchen to brew coffee. In the meantime, you had to leave to take care of some errands for your boss. An hour later, while walking down the hallway at work, you pass someone holding a steaming mug with the strong smell of freshly brewed coffee rising from it. You pause for a moment and ask yourself: where did this coffee come from? What conclusion suggests itself? How certain is that conclusion? Example 4: Assume the following observations are true. Do not make any additional assumptions about the relationships between the listed phenomena. Observation 1: Fever (A), cough (B), headache (C), and fatigue (D) occur together with inflammation, dehydration, an immune response, and a viral infection. Observation 2: Cough (B), headache (C), and fatigue (D) occur together with inflammation, dehydration, and an immune response. According to these assumptions, which of the listed phenomena (A–D) is a cause, part of a cause, or an effect of a viral infection? Why? Moreover, a range of other types of puzzles were incorporated, including queue problems, clock problems, and some trick-answer questions. For example: Example 5: Adam, Betty, and Celine

Read the complete text