Topology-Aware Learning for Robust Tubular Structure Segmentation In 3D Optical Microscopy
Abstract
Purpose
To address the persistent loss of connectivity and thickness instability in 3D microscopy segmentation of tubular structures, we propose a topology-aware learning framework that explicitly supervises tubular geometry beyond voxel-wise overlap.
Methods
We introduce a topology-aware adaptive radius loss that supervises a dense radius field derived from the medial axis of ground-truth tubular structures. The radius field encodes local thickness and implicitly captures global connectivity without relying on computationally expensive soft-skeletonization. The proposed loss is coupled with a dual-head 3D network that jointly predicts segmentation masks and radius fields, with adaptive loss balancing based on exponential moving averages to stabilize multi-task optimization. A Hybrid Kernel Network is further designed to efficiently combine local boundary detail with long-range structural continuity. The method is evaluated on three public 3D microscopy datasets spanning light-sheet, two-photon, and multicolor fluorescence imaging of vascular and neuronal tubular networks.
Results
Across all datasets, the proposed method consistently outperforms strong 3D baselines, including nnU-Net V2, SwinUNETR, UMamba, and SAM-Med3D. In addition to improved voxel-level accuracy, notable gains are achieved in topology-sensitive metrics, with up to 2–3% improvement in clDice and NSD, indicating superior preservation of connectivity and thickness. Qualitative results demonstrate fewer broken branches, reduced radius collapse at junctions, and more continuous tubular structures in 3D.
Conclusion
Explicit supervision of tubular geometry through an adaptive radius field provides an effective and computationally efficient mechanism for preserving topology in 3D microscopy segmentation. By prioritizing structurally vulnerable regions such as thin branches and junctions, the proposed approach improves true 3D connectivity beyond what is achievable with voxel-wise objectives alone. This framework generalizes across imaging modalities and anatomical targets, offering a practical route to topology-preserving segmentation in large-scale microscopy applications.