One Agent, Many Models: An AI-Agent Approach to Universal Tumor Segmentation
Abstract
Purpose
This work develops an agentic AI framework that bridges the gap between state-of-the-art tumor segmentation models and clinical deployment, where model discovery, data preprocessing, and output QA remain time-intensive and require computational expertise. This framework enables clinicians to access a vast repository of high-performing segmentation models without specialized computing skills, facilitating scalable deployment powered by the fast-growing global research community.
Methods
The proposed system employed a sequential reasoning framework, decomposing the tumor segmentation task into four stages: model selection, data preparation, model execution, and output verification. Each stage was handled by a dedicated AI agent, while a Python executor performed model inference using agent-prepared inputs. Agents interacted with the data and execution environment through Python functions registered as FastMCP tools. Tumor segmentation models were deployed on a local server from publicly available sources. Agents were prompted with structured role descriptions and rules to interpret image modality and tumor site, retrieve model documentation for model selection reasoning, organize raw image data according to model-specific requirements, and verify segmentation outputs following inference.
Results
Eight tumor segmentation models were evaluated across six scenarios; a case was counted successful if the system selected a compatible model, completed inference, and performed automated checks. Across 48 total cases, the system achieved an end-to-end task success rate of 77.1% (F1 = 0.85), ranging from 87.5% in best-case scenarios to 62.5% under the most constrained conditions. Model selection accuracy reached 83.3% (F1 = 0.91). Performance remained stable under data complication and model constraint scenarios, both achieving 87.5% success rates.
Conclusion
The proposed agentic AI framework enables robust, transparent, and scalable tumor segmentation by translating diverse segmentation models into real-world clinical workflows. It demonstrates resilience to heterogeneity with high sub-agent performance. Future work will extend this framework to provide interactive end-to-end agentic assistance for clinicians with tumor treatment.