Application of a Large Language Model In Radiation Oncology Safety Event Reports
Abstract
Purpose
To auto-classify radiation oncology safety events reported in narrative descriptions using a large language model (LLM) and propose a framework for future AI-assisted reporting systems.
Methods
Safety event data were extracted from our institutional reporting system, de-identified, and formatted using Python scripts. GPT-5 (OpenAI) was accessed via Application Programming Interface (API) and refined through iterative prompt engineering to classify incidents across multiple categories such as failure mode, severity, treatment type, discoverer’s role, exclusion criteria, and workflow stage for occurrence and discovery. The model performance was assessed against expert review using a random subset of cases. The complete dataset was classified using the validated model and the output was analyzed for quality improvement analysis.
Results
The optimized LLM demonstrated high classification accuracy across most dimensions, achieving overall agreement of 89.7%. Severity scoring exhibited high variability (76.6%), indicating a category that can be improved. The analysis of the failure modes showed communication as the most prominent contributor to safety events, accounting for approximately 25% of all failure modes, treatment planning 15%, and documentation/consent 14%. In the proposed AI-assisted reporting system, LLM was applied to classify the narrative safety event for review and correction, if necessary, before reporting, thus improving the report efficiency and accuracy.
Conclusion
LLM can accurately and efficiently classify safety events, providing a practical solution to a long-standing challenge of analyzing narrative safety reports. The resulting classifications reveal meaningful patterns in failure modes, severity, and workflow vulnerabilities that can inform targeted quality improvement initiatives. Integrating LLMs with human-in-the-loop verification, modern reporting platforms have the potential to redefine incident learning, enhance patient safety, and support a more resilient radiation oncology workflow.