Using Incident Learning Metrics to Evaluate the Effectiveness of APEx Accreditation
Abstract
Purpose
Radiation oncology accreditation programs aim to promote safety and quality in radiotherapy using nationally recognized standards. This study evaluates the impact of the ASTRO Accreditation Program for Excellence (APEx) at a single institution across three survey cycles.
Methods
The institution first received full accreditation in 5/2017 and again in 10/2022, both after first attempts. APEx survey reports and 2,940 incident learning system (ILS) events from 2015–2025 were analyzed. ILS data were grouped into three periods: pre‑accreditation (2015–2016; C0), first accreditation cycle (6/2017–2022; C1), and re‑accreditation cycle (10/2022–12/2025; C2). ILS-events were categorized by 55 APEx evidence indicators (EIs), with fewer events indicating improvement. The number of ILS-events were compared across all 3 cycles and between C1 and C2.
Results
Scores for medical records review, physics interview, and team interview ranged from 90–100% in both surveys, demonstrating consistently high performance. ILS-events mapped to 22 EIs, seven of which showed notable changes. Three EIs—informed consent (EI 14.1), simulation directive (EI 2.1), and patient timeout (EI 3.2)—showed reductions during C1 and C2 compared with C0, with average decreases of 24.3±5.5%, 14.6±7.3%, and 6.1±1.1%. Four EIs—treatment planning directive (EI 2.2), radiation prescription (EI 2.3), treatment plan (EI 2.4), and patient-specific treatment plan checks (EI 3.4)—showed average increases of 2.6±1.0%, 2.0±2.3%, 3.8±2.5%, and 1.6±2.2% for C1 and C2 compared with C0. Overall event changes across the 22 EIs from C1 to C2 were statistically insignificant. However, five EIs—EI 2.3, EI 2.4, EI 3.2, EI 3.3 (standard operating procedures), and EI 3.4—showed an average reduction of 3.8±2.3%, indicating continued improvement with ongoing accreditation.
Conclusion
We present the first quantitative method to assess the impact of APEx accreditation on clinical operations and a safety and quality program at one academic institution. Quantifying APEX accreditation outcomes improves understanding of how accreditation influences clinical practice and supports ongoing improvement.