Evaluating psychiatric conference posters: Benchmarking a custom generative pre-trained transformer against human inter-rater variability using a structured assessment framework

Anusa Arunachalam Mohandoss, Rooban Thavarajah, Raj Kiran Donthu, Swapnajeet Sahoo, Sai Krishna Tikka, Shahul Ameen · 2026

Read Evaluating psychiatric conference posters: Benchmarking a custom generative pre-trained transformer against human inter-rater variability using a structured assessment framework by Anusa Arunachalam Mohandoss, Rooban Thavarajah, Raj Kiran Donthu, Swapnajeet Sahoo, Sai Krishna Tikka, Shahul Ameen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-13623258.txt.

Description

Mohandoss and co-authors benchmark an AI poster judge against human raters in a 2026 study. Conference posters in psychiatry lack a shared, neutral scoring rubric. Human grades vary by rater, which makes prizes and feedback feel arbitrary.

Their framework, called PA²IRS, was refined through AI-assisted rounds modeled on Delphi consensus. Sixty posters across case reports, original research, and reviews were graded by the system. Three mental health professionals graded the same set on their own. Agreement between the machine and humans is set against human-to-human agreement. A rubric that grades like a fair panel could clean up poster halls everywhere.

Catalog record

CREATORAnusa Arunachalam Mohandoss, Rooban Thavarajah, Raj Kiran Donthu, Swapnajeet Sahoo, Sai Krishna Tikka, Shahul Ameen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC13623258

Browse the full library · About the Tower of Babel