Large Language Model-Assisted Analysis of Planning Peer Review Data to Drive Quality Improvement In MR-Guided Online Adaptive Radiotherapy
Abstract
Purpose
Magnetic resonance imaging-guided online adaptive radiotherapy (MRgRT) requires a complex planning process, during which an upstream peer review prior to physician approval plays an essential role in early error detection and supporting safe and efficient online adaptation. Analyzing the peer review data gives critical insights for process improvement. However, extracting actionable items from the large volume of peer review data from different reviewers can be resource-intensive and difficult to scale. Leveraging a unique real-world structured dataset of physics-led peer review, this study evaluates the use of large language model (LLM)–assisted analysis to support the evolution of peer review processes toward safer and more efficient MRgRT workflows.
Methods
For the past year, an upstream peer review process was implemented prior to physician approval for MRgRT plans, with findings documented using a structured checklist with free-text comments. Over one year, 599 cases underwent review, generating a large pool of clinically annotated data. Using a manually-reviewed subset (99 test cases, 50 validation cases), a LLM prompt-engineering workflow was developed to identify issue-bearing comments, normalize reviewer variability, and classify findings into clinically-relevant categories. Peer review duration and issue distributions were analyzed to assess workflow efficiency and practical applicability.
Results
LLM-assisted analysis identified 798 issues, most frequently detected in Structures (28.2%), IMRT Constraints (19.3%), Physician Intent (13.9%), and Plan Quality (10.4%). The distribution and prioritization of identified issues were consistent with expectations from experienced reviewers. Over the implementation period, the mean review duration reduced from 37.6 to 18.5 min, reflecting improved efficiency and more focused use of physics resources within the MRgRT workflow.
Conclusion
LLM can be effectively leveraged to analyze clinical MRgRT peer review data to inform targeted quality improvement measures. This approach facilitates seamless MRgRT adaptation and provides a pragmatic pathway toward sustainable peer review in the resource-intensive adaptive radiotherapy environments.