Grafite : Generative Regression Analysis Framework for Issue Tracking and Evaluation

Ja Young Lee, Mírian Silva, Mohamed Nasr · 2026

Read Grafite : Generative Regression Analysis Framework for Issue Tracking and Evaluation by Ja Young Lee, Mírian Silva, Mohamed Nasr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18173.txt.

Introduction

Lee and co-authors present GRAFITE, a platform for tracking and testing issues in large language models. Benchmark exposure during training can inflate scores over time, they note. GRAFITE builds a growing repository of model problems from user feedback. A pipeline tests models against these issues with LLM-as-a-judge quality checks. Side-by-side views across releases help teams spot regressions as models change.

Catalog record

CREATORJa Young Lee, Mírian Silva, Mohamed Nasr
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29621
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18173

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grafite : Generative Regression Analysis Framework for Issue Tracking and Evaluation Ja Young Lee, Mírian Silva, Mohamed Nasr Abstract Large language models (LLMs) are largely motivated by their performance on popular topics and benchmarks at the time of their release. However, over time, contamination occurs due to significant exposure of benchmark data during training. This poses a risk of model performance inflation if testing is not carefully executed. To address this challenge, we present Grafite , a continuous LLM evaluation platform through a comprehensive system for maintaining and evaluating model issues. Our approach enables building a repository of model problems based on user feedback over time and offers a pipeline for assessing LLMs against these issues through quality assurance (QA) tests using LLM-as-a-judge. The platform enables side-by-side comparison of multiple models, facilitating regression detection across different releases. The platform is available at https://github.com/IBM/grafite . The demo video is available at www.youtube.com/watch?v=XFZyoleN56k . Figure 1: The pipeline of Grafite and stakeholders of each step 1 Introduction Large language models (LLMs) have demonstrated remarkable performance through rapid development as showcased by popular benchmarks. However, there have been growing concerns about the risk of inflated model performance metrics due to benchmark data contamination during training if testing procedures are not meticulously designed and executed 6 ; 17 ; 18 . There have been various efforts to mitigate this risk by constantly updating the benchmark with newly released datasets 16 ; 9 , leveraging LLMs 19 or human intervention 4 . Regardless, challenges remain in tracking a model’s regression performance or improvement in a specific domain over time. Constantly updating the benchmark dataset is not well-suited for quantifying absolute performance improvements over time and leveraging LLM or human intervention can introduce subjectivity or bias as the evaluation quality relies on the evaluator 15 . Continuous model evaluation in a specific domain presents additional challenges in user experience. Frameworks exist that offer streamlining of LLM evaluation, such as LLMeBench 5 , LM Evaluation Harness 7 , HELM 11 , and OpenAI Evals 1 1 1 https://github.com/openai/evals . While they offer high flexibility with the dataset and evaluators, they focus on one-time performance benchmarking and require technical knowledge to be utilized effectively, as they are command-line interface tools. Tracking model performance using one’s own setup remains the responsibility of the user. In addition, while most benchmarks aim to evaluate models’ general and broad-use capabilities, they often overlook domain-specific scenarios for specific use cases 3 ; 12 . Such scenarios require a nuanced evaluation approach, which standard benchmarks typically do not address. Issue Title : Math - Geometry User input : A triangle has a base of 6.5 meters and a height of 4.3 meters. Calculate its area. Model Output : <to be generated by the selected model> . Sample Reference Answer (ground truth) : To calculate the area of a triangle, you can use the formula \text{Area}=\frac{1}{2}\times\text{base}\times\text{height} . Since the given base is 6.5 meters and the given height is 4.3 meters, the area of the triangle is \frac{1}{2}\times 6.5\times 4.3 . Thus, \text{Area}=\frac{1}{2}\times 6.5\times 4.3=3.25\times 4.3=13.975 square meters. Therefore, the area of the triangle with a base of 6.5 meters and a height of 4.3 meters is 13.975 square meters. Evaluation Criteria Judge Type : "Prompt template 1: input + output + ground truth + judge guideline" Judge Guideline: 1. Use the correct geometric equation to solve. 2. In math, the answer must match the reference exactly to be correct. 3. The reference and predicted answer should be in the simplest form. 4. The input should be a geometry math question. Table 1: A sample test from an issue. To address these challenges, we present a web-based platform, Grafite , for continuous LLM benchmarking through issue creation and management with model evaluation using an ensemble of LLM judges 14 and human-in-the-loop annotation. The Grafite pipeline is shown in Figure 1 : When an end user asks a question and is unsatisfied, they provide a thumbs down as feedback. This triggers expert triaging by subject matter experts who use Grafite to assess whether an issue and test should be created. All issues can then be evaluated by LLM judge validation to generate a report as feedback to model developers. This analysis can also be used to improve and adjust the issues and tests as needed. An example of an issue and test is shown in Table 1 . This design enables end users to contribute their feedback and provides model developers and researchers with detailed performance reports while maintaining the integrity of the data. Our contributions are: • Grafite issue tracking: A novel framework to track domain-specific model problems across model versions. Our platform serves as a repository of model issues, associated with human-contributed tests per issue, to validate model performance on each issue. • Grafite evaluation: By leveraging human-contributed test sets, researchers and model developers can assess model performance over time across versions or against other models, gaining insights into model improvements and regressions within targeted scenarios. By evaluating models on human-created test sets that target specific use cases not visible to model developers, Grafite supports more robust model development by mitigating the risk of inflated performance that can result from focusing on popular benchmarks. • Grafite analytics: An analytics visualization tool that facilitates understanding of model performance. This feature streamlines benchmarking models across domains, versions, and comparisons between different models through a user-friendly graphical user interface (GUI). • We accompany Grafite with a use case study on sample data with a publicly available model to fully illustrate its capabilities and a user study to assess the platform’s efficacy and usability. We open-source our platform to the community, along with the sample data, at https://github.com/IBM/grafite to promote collaborative research advancement. 2 Grafite Platform Grafite is a web application developed using Next.js 2 2 2 https://nextjs.org/ , Ag-grid 3 3 3 https://www.ag-grid.com/ , Carbon Design System 4 4 4 https://carbondesignsystem.com/ , RabbitMQ 5 5 5 https://www.rabbitmq.com/ , FastAPI 6 6 6 https://fastapi.tiangolo.com/ , and MongoDB 7 7 7 https://www.mongodb.com/ . It is compatible with Ollama 8 8 8 https://ollama.com/ and WatsonX.ai 9 9 9 https://www.ibm.com/products/watsonx-ai for model inference. Users can easily execute tasks using the GUI. All data, including evaluation results, is stored in MongoDB for issue management and long-term performance tracking. The system is designed to enable model developers and researchers to track and manage selected model problems and evaluate LLMs over time. As shown in the shaded box in Figure 1 , the app aggregates model problems into a set of issues and tests. These issues are assessed using the LLM-as-a-judge technique 19 , which translates qualitative evaluations into quantitative metrics using LLMs as a panel of evaluators 14 . This LLM-based evaluation is combined with human evaluation to streamline the analysis of model performance and generate reports. 2.1 Issues and Tests The issue is the bedrock of the Grafite system. An issue is the erroneous behavior of a model during testing. It is synthesized from multiple sources of domain-specific feedback. Each issue includes metadata concerning: (a) a detailed description of the identified error, (b) user feedback from which the issue originated for reference, (c) resolution statuses, (d) domain tags, and (e) a set

Read the complete text