Comparing Open-Source and Commercial LLMs for Domain-Specific Analysis and Reporting: Software Engineering Challenges and Design Trade-offs Journal: Information and Software Technology
Theo Koraag, Niklas Wagner, Felix Dobslaw, Lucas Gren · 2025
Read Comparing Open-Source and Commercial LLMs for Domain-Specific Analysis and Reporting: Software Engineering Challenges and Design Trade-offs Journal: Information and Software Technology by Theo Koraag, Niklas Wagner, Felix Dobslaw, Lucas Gren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24344.txt.
Description
Firms weigh open-source models against commercial ones for sensitive analysis work, and the trade-offs are rarely clean. Theo Koraag and 3 co-authors ran an exploratory case study on financial report analysis and commentary. They built two systems in rounds: one with local open models in a multi-agent workflow, and one on a commercial cloud model, then had experts judge both on real reporting tasks.
Both systems showed strong potential, but integration was the hard part. Prompt design, context limits, and reliability all bit in practice. The cloud model was more fluent and usable but raised privacy and dependency worries. Local models gave control and compliance at a far higher engineering cost.
Catalog record
| CREATOR | Theo Koraag, Niklas Wagner, Felix Dobslaw, Lucas Gren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24344