Improving Portability of Knowledge-Based Planning Using an LLM-Driven Plan Refinement Framework
Abstract
Purpose
Knowledge-based planning (KBP) improves plan quality and efficiency. However, training institution-specific models requires substantial clinical data and expertise, and publicly available models may not align with local clinical objectives. This study evaluates whether the planning co-pilot, an LLM-driven plan refinement framework, can enable clinics to use off-the-shelf KBP models without local retraining.
Methods
We developed a multi-agent LLM system that automatically refines KBP-generated plans through direct interaction with the Eclipse treatment planning system. First, a Planner agent analyzes current DVH metrics and selects optimization actions from a discrete action space. Second, an Evaluator agent evaluates each constraint as Met or Not Met through numerical comparison and reports to the Supervisor agent, which audits the evaluation for numerical and context accuracy. Using 62 retrospective NSCLC patients, we applied the planning co-pilot system across three different KBP models without any model-specific prompt tuning: a publicly available model (UCSD: Lung-Mediastinum-ORBIT-RT), and two institutional models with different training configurations (Institutional T1: in-house trained; Institutional T2: in-house refined). Clinical goal achievement rates were compared between initial KBP plans and Co-Pilot optimized plans. PTV coverage was normalized to 95% for all cases.
Results
The clinical goal achievement rates increased from 79% to 98% (UCSD), 74% to 97% (Institutional T1), and 69% to 98% (Institutional T2), demonstrating the system's ability to achieve similar plan quality across different KBP models. With the co-pilot, plans with unmet constraints were reduced from 13 to 1, 16 to 2, and 19 to 1, respectively. The most frequently resolved violations were Lung Dmean and Lung V20.
Conclusion
The study demonstrated that an LLM-based planning co-pilot can effectively adapt plans generated from heterogeneous KBP models to meet local clinical requirements without fine-tuning the underlying models, providing a practical pathway for broader KBP adoption and clinical use of off-the-shelf KBP models.