Self-Supervised Bayesian Multimodal Learning for Uncertainty-Aware Prediction of Radiation Pneumonitis
Abstract
Purpose
Accurate prediction of radiation-induced toxicity is crucial for optimizing radiotherapy outcomes. However, most existing predictive models rely on uni-modal data and deterministic models that are vulnerable to label noise and uncertainty. This study aims to develop an uncertainty-aware probabilistic framework that integrates multi-modal data to enable interpretable toxicity prediction, supporting improved clinical decision-making.
Methods
A multi-modal variational autoencoder (VAE) was pre-trained in a self-supervised manner to learn robust features representations from CT images and spatial dose distributions using two convolutional neural networks (CNNs)-based or Vision Transformers (ViTs)-based encoders, reducing susceptibility to label noise. Modality-specific features were fused in the latent space via a product-of-expert for generating a unified multi-modal representation. A downstream Bayesian classifier was trained on the fused latent features. During inference, Monte Carlo (MC) dropout was implemented to approximate Bayesian inference by sampling 100 or 200 estimates, yielding predictive distributions. The mean of each predictive distributions was used for label prediction, while the standard deviation quantified predictive epistemic uncertainty. The framework was evaluated on 416 patients from RTOG 0617, with radiation pneumonitis treated as noisy binary outcomes using a grade ≥ 2 threshold.
Results
The CNN-based model achieved AUCs of 0.8389 and 0.8393 when using 100 and 200 MC samples, respectively, with corresponding uncertainty ranges of 0.02-0.08 and 0.02-0.07. The ViT-based model yielded AUCs of 0.8056 and 0.8076 under the same sampling settings, with uncertainty ranges of 0.04-0.09 and 0.05-0.09. The CNN-based model slightly outperformed the ViT-based model, and both exhibited reasonable and well-controlled uncertainty, indicating stable predictions for most patients.
Conclusion
This study demonstrates the feasibility of Bayesian multimodal learning using CT and dose information for radiation pneumonitis prediction and uncertainty estimation. By providing risk predictions and quantitative measures of epistemic uncertainty, the framework offers clinicians a confidence metric, facilitating safer and proactive toxicity management.