Modality-Aware Cross-Modal Transformer for Joint Fusion of PET/CT and Clinical Data for Relapse-Free Survival Prediction In Multi-Centric HPV-Associated Oropharyngeal Cancer
Abstract
Purpose
To develop and validate a modality-aware cross-modal transformer for joint fusion in relapse-free survival prediction in HPV-associated oropharyngeal cancer (OPSCC), integrating PET/CT imaging and clinical data for risk stratification.
Methods
We used the HECKTOR 2025 multi-center dataset (N = 678; 542 training, 136 test). The proposed Cross-Modal Transformer architecture employs:(1) 3D DenseNet imaging encoders for PET/CT and FTTransformer encoder for clinical variables (i.e., demographics, stage, HPV status, etc.), and radiomic features derived from nnUNet-autosegmented primary/nodal tumour volumes. (2) linear tokenizers projecting modalities to shared latent space, (3) learnable positional encodings for modality identity, (4) shallow 3-layer transformer encoder with 8-head self-attention enabling cross-modal interactions, and (5) mask-aware pooling for missing data robustness. We compared against unimodal baselines (clinical-only, imaging-only, radiomic-only) and late fusion variants (attention, weighted, stacking). Models were trained with DeepHit and Contrastive losses using 5-fold cross-validation. Discrimination was assessed via concordance index (C-index) with 95% 1000-resample bootstrap CIs; calibration via Integrated Brier Score (IBS), and risk stratification used training-derived risk cutoffs on HPV+ test patients, with hazard ratios (HR) comparing high vs. low-risk groups.
Results
The Cross-Attention model achieved a holdout C-index of 0.675 (95% CI: 0.563-0.770) with the best calibration (IBS=0.160). It successfully stratified HPV+ patients (HR=4.46; 95%CI: 1.38-14.40; p=0.012) between high- and low-risk groups. Late Fusion with attention achieved the highest discrimination (C-index=0.701, HR=7.59, p=0.009), with wider confidence intervals, indicating less precise stratification. All unimodal models demonstrated reasonable cross-validation discrimination (C-index=0.659-0.714) yet failed to produce clinically meaningful risk stratification.
Conclusion
Multimodal transformer joint fusion of PET/CT imaging and clinical data improves clinically actionable risk stratification beyond unimodal and late-fusion approaches in HPV-associated OPSCC. Modelling cross-modal interactions reveals biologically distinct risk groups within HPV-associated disease, achieving a balanced combination of discrimination, calibration, and stratification precision that may guide treatment intensification and future de-escalation strategies.