TRIPOD level-4 validation for a larynx cancer survival model using distributed learning
Hansen, C. R. Field, M. Price, Gareth J Sarup, N. Zukauskaite, R. Johansen, J. Eriksen, J. G. Aly, F. McPartlin, Andrew J Holloway, L.
Hansen, C. R.
Field, M.
Price, Gareth J
Sarup, N.
Zukauskaite, R.
Johansen, J.
Eriksen, J. G.
Aly, F.
McPartlin, Andrew J
Holloway, L.
Citations
Altmetric:
Abstract
Purpose or Objective
Prediction models are needed to support clinical decision making; however, models need to be robustly validated in diverse
cohorts to demonstrate generalisability to the clinical community. Healthcare providers also have a competing responsibility
to protect sensitive patient data. The current study used distributed learning to validate a larynx cancer survival model in
an international multi-centre setting without patient data leaving their own host institute.
Materials and Methods
Patients receiving radiotherapy for larynx cancer from 2005-2018 at three international centres were identified to validate
the overall survival (OS) model of Egelmeer et al. (Radiother. Oncol. 2011). This model utilises the parameters: time
corrected EQD2 tumour dose, haemoglobin at treatment start, sex, age, site (glottic vs non-glottic), tumour and nodal
stage. Data imputation for a maximum of one missing variable was allowed. An institution-stratified Cox regression model
was developed utilising an open-source privacy-by-design distributed learning network. The validation aimed to test
whether the hazard predicted from the original model would benefit from multiplication by a recalibration (RCA) factor.
The study is a TRIPOD level 4 validation, where the model is fully supported if RCA=1. During the entire model optimisation,
no patient data left their own hospital.
Results
1930 patients were identified, with 1278 suitable for use in the evaluation. The RCA factor determined across the centres
was 0.76 [95%CI 0.62-0.91], i.e. showing the original model would benefit from recalibration. The three centres' Harrell C indices were 0.68±0.06, 0.74±0.02 and 0.70±0.04 (95%CI), indicating a generally acceptable model performance. The
distributed learning system produced centre-specific calibration plots and comparisons between observed and predicted
Kaplan-Meier curves split by risk group. Following RCA, the data in the calibration plot is close to the identity line, indicating
the model's general applicability. The Kaplan-Meier plots (fig 2) show that the gain from the RCA factor is centre-dependent. Due to the stratified approach,
baseline risks can also be calculated per centre. Differences between centres are observed, indicating that differences in
OS cannot be fully accounted for based only on the included model parameters. Additional parameters are needed to
improve the original model’s centre-specific performance to account for this. Conclusion
A TRIPOD type 4 evaluation has been performed of the Egelmeer et al. model using distributed learning to protect patient
privacy. It is shown that the model needed RCA to increase the predictive accuracy. However, the improvement in
prediction power was institution-dependent, indicating that differences within the cohorts exist beyond those accounted
for by the original model parameters. This indicates the need to evaluate the regression value for the included model
parameter or include additional parameters, e.g. smoking status and tumour volume.
Description
Date
2022
Publisher
Collections
Keywords
Type
Meetings and Proceedings
Citation
Hansen CR, Field M, Price G, Sarup N, Zukauskaite R, Johansen J, et al. TRIPOD level-4 validation for a larynx cancer survival model using distributed learning. Radiotherapy and Oncology. 2022 May;170:S667-S8. PubMed PMID: WOS:000806764200303.