TRIPOD level-4 validation for a larynx cancer survival model using distributed learning

Abstract

Purpose or Objective Prediction models are needed to support clinical decision making; however, models need to be robustly validated in diverse cohorts to demonstrate generalisability to the clinical community. Healthcare providers also have a competing responsibility to protect sensitive patient data. The current study used distributed learning to validate a larynx cancer survival model in an international multi-centre setting without patient data leaving their own host institute. Materials and Methods Patients receiving radiotherapy for larynx cancer from 2005-2018 at three international centres were identified to validate the overall survival (OS) model of Egelmeer et al. (Radiother. Oncol. 2011). This model utilises the parameters: time corrected EQD2 tumour dose, haemoglobin at treatment start, sex, age, site (glottic vs non-glottic), tumour and nodal stage. Data imputation for a maximum of one missing variable was allowed. An institution-stratified Cox regression model was developed utilising an open-source privacy-by-design distributed learning network. The validation aimed to test whether the hazard predicted from the original model would benefit from multiplication by a recalibration (RCA) factor. The study is a TRIPOD level 4 validation, where the model is fully supported if RCA=1. During the entire model optimisation, no patient data left their own hospital. Results 1930 patients were identified, with 1278 suitable for use in the evaluation. The RCA factor determined across the centres was 0.76 [95%CI 0.62-0.91], i.e. showing the original model would benefit from recalibration. The three centres' Harrell C indices were 0.68±0.06, 0.74±0.02 and 0.70±0.04 (95%CI), indicating a generally acceptable model performance. The distributed learning system produced centre-specific calibration plots and comparisons between observed and predicted Kaplan-Meier curves split by risk group. Following RCA, the data in the calibration plot is close to the identity line, indicating the model's general applicability. The Kaplan-Meier plots (fig 2) show that the gain from the RCA factor is centre-dependent. Due to the stratified approach, baseline risks can also be calculated per centre. Differences between centres are observed, indicating that differences in OS cannot be fully accounted for based only on the included model parameters. Additional parameters are needed to improve the original model’s centre-specific performance to account for this. Conclusion A TRIPOD type 4 evaluation has been performed of the Egelmeer et al. model using distributed learning to protect patient privacy. It is shown that the model needed RCA to increase the predictive accuracy. However, the improvement in prediction power was institution-dependent, indicating that differences within the cohorts exist beyond those accounted for by the original model parameters. This indicates the need to evaluate the regression value for the included model parameter or include additional parameters, e.g. smoking status and tumour volume.