Watch Once, Work Forever: One-Shot Computer-Use Agents for Clinical UI Automation
Poster
CUA_AAPM2026.pdf Hosted by AAPM · 0.5 MBAbstract
Purpose
Radiotherapy planning requires frequent interaction with complex clinical software, creating inefficiency and opportunities for error. Current automation pipelines depend on scripting, large datasets, or vendor-specific APIs. Here, we assess whether a one-shot, pixel-to-action computer-use agent can learn a workflow from a single demonstration and generalize across interface states.
Methods
A computer use agent (CUA) was developed to perform common tasks within a sandbox environment, including plan navigation, structure review, DVH extraction, and report generation in the Eclipse TPS. The agent leverages a multimodal perception-action model combining visual interface understanding, natural-language task representation, and one-shot learning from a single un-annotated demonstration. After observation, the agent was evaluated on unseen tasks and workflow variations without additional fine-tuning. Performance was assessed by task completion rate, execution time, and agreement between agent-generated and human-generated ground truth outputs. Failure behaviours were also analyzed to assess clinical feasibility.
Results
Candidate vision–language backbones were benchmarked on a 100-prompt UI localization suite (53 coordinate click tasks, 47 boolean UI-state questions; temperature=0). Qwen3-VL-8B led spatial grounding (79.2% clicks within 50 px; 12.5 px median error; 25.23 s/prompt) while maintaining 100% boolean accuracy. Qwen3-VL-4B offered the best speed–accuracy tradeoff (75.5% within 50 px; 16.0 px median error; 15.29 s/prompt; 97.9% boolean). UI-TARS-7B-DPO was similar in localization (73.6% within 50 px; 15.3 px median error) with 100% boolean accuracy but higher latency (31.09 s/prompt). In contrast, FARA variants retained strong semantic performance (97.9–100% boolean) yet failed at spatial grounding (0% within 50 px; ~700 px median error), revealing a semantic–spatial split that limits reliable autonomous clicking in complex clinical GUIs.
Conclusion
One-shot learning computer use agents demonstrate promising capability to autonomously execute complex dosimetric workflow tasks after minimal supervision. These findings support further investigation into agent-assisted treatment planning as a human-in-the-loop clinical tool.