---

About

Challenge Performance

Date	Challenge	Phase	Rank
Sept. 7, 2023	PI-CAI	Closed Testing Phase - Testing (Final Ranking)	5
June 13, 2024	PI-CAI	Closed Testing Phase - Tuning	4

Model Facts

Summary

This algorithm represents the submission from the HeviAI team (A. Karagöz, et al.; Turkey) to the PI-CAI challenge [1]. We independently retrained this algorithm using the PI-CAI Private and Public Training Dataset (9107 cases from 8028 patients, including a sequestered dataset of 7607 cases and the public dataset of 1500 cases). This algorithm will perform two tasks: localize and classify each lesion with clinically significant prostate cancer (if any) using a 0–100% likelihood score and classify the overall case using a 0–100% likelihood score for clinically significant prostate cancer diagnosis.

To this end, this model uses biparametric MRI data and several clinical variables associated with the examination. Specifically, this algorithm uses the axial T2-weighted MRI scan, the axial apparent diffusion coefficient map, and the calculated or acquired axial high b-value scan. This algorithm uses the prostate-specific antigen level normalized by the total volume of the gland as measured by the prostate segmentation method to inform their predictions.

1. A. Saha, J. S. Bosma, J. J. Twilt, B. van Ginneken, A. Bjartell, A. R. Padhani, D. Bonekamp, G. Villeirs, G. Salomon, G. Giannarini, J. Kalpathy-Cramer, J. Barentsz, K. H. Maier-Hein, M. Rusu, O. Rouvière, R. van den Bergh, V. Panebianco, V. Kasivisvanathan, N. A. Obuchowski, D. Yakar, M. Elschot, J. Veltman, J. J. Fütterer, M. de Rooij, H. Huisman, and the PI-CAI consortium. “Artificial Intelligence and Radiologists at Prostate Cancer Detection on MRI (PI-CAI): An International, Paired, Non-Inferiority, Confirmatory Study”. The Lancet Oncology, 2024; 25(7): 879-887.

Mechanism

Team: Hevi AI

Ahmet Karagöz (1,#), Mustafa Ege Şeker (2,#), Mert Yergin (3,#), Mustafa Said Kartal (4,#), Ercan Karaarslan (2,#), Deniz Alis (2,#), Ilkay Oksuz (1,#)

(1) Computer Engineering, Istanbul Technical University, Istanbul, Turkey

(2) Acibadem Mehmet Ali Aydinlar University School of Medicine, Istanbul, Turkey

(3) Department of Software Engineering and Applied Sciences, Bahcesehir University, Istanbul, Turkey

(4) Faculty of Medicine, Sivas Cumhuriyet University, Sivas, Turkey

(#) These authors contributed equally to this works

Contact: ahmet.karagoz@hevi.ai.

Code availability: github.com/ahmetkrgztr/HeviAI_picai.

Trained model availability: grand-challenge.org/algorithms/pi-cai-pubpriv-heviai/

Abstract: We present a workflow which predicts csPCA on biparametric prostate MRI PI-CAI 2022 Challenge with over 10,000 carefully-curated prostate MRI exams. We propose to segment the prostate gland first to the central gland (transition + central zone) and the peripheral gland. Then we utilize these predictions in combination with T2, ADC and DWI images to train an ensemble nnU-Net model. Finally, we utilize clinical indices PSA and ADC intensity distributions of lesion regions to reduce the false positives. Our method achieves top results on open-validation stage with a AUROC of 0.888 and AP of 0.732.

Data preparation: All images were converted from MHA (.mha) to nifti format (.nii.gz) as a preprocessing step for the lesion segmentation / detection models because nnU-Net [1] requires nifti images within its own structure. To create images of the same size and resolution, each patient’s ADC and DWI images were resampled, cropped, or padded. T2 was used as the primary modality within each patient, and the DWI and ADC images were adjusted to match T2 in terms of size and resolution. ADC images were normalized with complete z-score normalization with respect to the entire dataset while T2 and DWI images were normalized with instancewise z-score normalization since ADC images are more robust than T2 and DWI images. By starting at the voxel with the highest degree of confidence and encompassing all related voxels (in 3D) with at least 40% of the peak’s degree of confidence, we specifically produced a lesion candidate. The candidate lesion is then eliminated from the softmax (confidence map), and the procedure is repeated up to the extraction of 5 lesions, if there are still remaining candidates. Small candidates with 10 voxels or fewer (0.009 cm³) are eliminated [2].

Training setup: To begin with, a nnU-Net model was trained using T2, ADC, and DWI images (Fig. A3). We employed the nnU-Net framework, but instead of the default combination of dice and cross entropy loss, Focal Loss and cross entropy loss were employed, as in [2].

Model parameters:

• Total number of parameters of nnU-Net for prostate segmentation: 44,808,793 x5

• Total number of parameters of nnU-Net for lesion detection: 44,800,800 x5

Inference setup: During inference, first we generate probabilistic zonal prostate gland segmentation. Then, we run our nnU-Net lesion segmentation setup. Finally, PSA thresholding was performed to analyze whether the model performance could be improved by combining the clinical data with the best-performing model obtained as a result of the experiments with DL methods. Although a threshold of 0.1 is considered appropriate in the literature for the differentiation of malignant and benign, in this experiment, 0.04 was chosen as the threshold value to prevent possible false negative predictions, therefore patients below this value were directly considered non-csPCA. PSA thresholding did not significantly improve performance, but this was due to the threshold that was kept low to prevent possible false negatives.

Acknowledgements: This work has been produced benefiting from the 1001 Science and Technology Grant Program National Program of TUBITAK (Project No: 122E022). The work has been produced benefiting from the 2232 International Fellowship for Outstanding Researchers Program of TUBITAK (Project No: 118C353). However, the entire responsibility of the publication/paper belongs to the owner of the paper. The financial support received from TUBITAK does not mean that the content of the publication is approved in a scientific sense by TUBITAK.

Figure: Overview of the solution of HeviAI. Prostate lesion segmentation model with prostate zone masks(A). The prostate gland segmentation model was trained on 203 patients of the ProstateX [3] dataset. The prostate gland segmentation model’s output, Peripheral and Central gland predictions (probabilistic), were given to the model as separate modalities in addition to T2, ADC, and DWI . By doing this, the effects of prostate glands on lesion segmentation performance were examined. Models were trained for five folds using T2, ADC, and DWI images.

References:

1. F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,” Nature methods, vol. 18, no. 2, pp. 203–211, 2021.

2. J. S. Bosma, A. Saha, M. Hosseinzadeh, I. Slootweg, M. de Rooij, and H. Huisman, “Semisupervised learning with report-guided pseudo labels for deep learning-based prostate cancer detection using biparametric mri,” Radiology: Artificial Intelligence, p. e230031, 2023.

3. Armato SG 3rd, Huisman H, Drukker K, Hadjiiski L, Kirby JS, Petrick N, Redmond G, Giger ML, Cha K, Mamonov A, Kalpathy-Cramer J, Farahani K. PROSTATEx Challenges for computerized classification of prostate lesions from multiparametric magnetic resonance images. J Med Imaging (Bellingham). 2018 Oct;5(4):044501. doi: 10.1117/1.JMI.5.4.044501.

Validation and Performance

This algorithm was evaluated on the PI-CAI Testing Cohort. This hidden testing cohort included prostate MRI examinations from 1000 patients across four centers, including 197 cases from an external unseen center. Histopathology and a follow-up period of at least 3 years were used to establish the reference standard. See the PI-CAI paper for more information [1].

Patient-level diagnosis performance is evaluated using the Area Under Receiver Operating Characteristic (AUROC) metric. Lesion-level detection performance is evaluated using the Average Precision (AP) metric. Overall score used to rank each AI algorithm is the average of both task-specific metrics: Overall Ranking Score = (AP + AUROC) / 2.

This algorithm achieved an AUROC of 0.892, an AP of 0.616, and an Overal Ranking Score of 0.754.

Free-Response Receiver Operating Characteristic (FROC) curve is used for secondary analysis of AI detections (as recommended in Penzkofer et al., 2022). We highlight the performance on the FROC curve using the SensX metric. SensX refers to the sensitivity of a given AI system at detecting clinically significant prostate cancer (i.e., Gleason grade group ≥ 2 lesions) on MRI, given that it generates the same number of false positives per examination as the PI-RADS ≥ X operating point of radiologists. Here, by radiologists, we refer to the radiology readings that were historically made for these cases during multidisciplinary routine practice. Across the PI-CAI testing leaderboards (Open Development Phase - Testing Leaderboard, Closed Testing Phase - Testing Leaderboard), SensX is computed at thresholds that are specific to the testing cohort (i.e., depending on the radiology readings and set of cases).

This algorithm achieved a Sens3 of 0.714, a Sens4 of 0.676, and a Sens5 of 0.497.

Figure. Diagnostic performance of the top five AI algorithms (N. Debs et al. [Guerbet Research, France], Y. Yuan et al. [University of Sydney, Australia], H. Kan et al. [University of Science and Technology, China], C. Li et al. [Stanford University, United States] and , A. Karagöz et al. [Istanbul Technical University, Turkey]), and the AI system ensembled from all five methods, across the 400 cases used in the reader study (left column) and the full hidden testing cohort of 1000 cases (right column). Case-level diagnosis performance was evaluated using receiver operating characteristic curves and the AUROC metric (top row), while lesion-level detection performance was evaluated using precision-recall curves and the AP metric (middle row). Secondary analysis of lesion-level detection performance was conducted using FROC curves (bottom row).

1. A. Saha, J. S. Bosma, J. J. Twilt, B. van Ginneken, A. Bjartell, A. R. Padhani, D. Bonekamp, G. Villeirs, G. Salomon, G. Giannarini, J. Kalpathy-Cramer, J. Barentsz, K. H. Maier-Hein, M. Rusu, O. Rouvière, R. van den Bergh, V. Panebianco, V. Kasivisvanathan, N. A. Obuchowski, D. Yakar, M. Elschot, J. Veltman, J. J. Fütterer, M. de Rooij, H. Huisman, and the PI-CAI consortium. “Artificial Intelligence and Radiologists at Prostate Cancer Detection on MRI (PI-CAI): An International, Paired, Non-Inferiority, Confirmatory Study”. The Lancet Oncology, 2024; 25(7): 879-887.

Uses and Directions

• For research use only. This algorithm is intended to be used only on biparametric prostate MRI examinations of patients with raised PSA levels or clinical suspicion of prostate cancer. This algorithm should not be used in different patient demographics.

• Benefits: AI-based risk stratification for clinically significant prostate cancer using prostate MRI can potentially aid the diagnostic pathway of prostate cancer, reducing over-treatment and unnecessary biopsies.

• Target population: This algorithm was trained on patients with raised PSA levels or clinical suspicion of prostate cancer, without prior treatment (e.g. radiotherapy, transurethral resection of the prostate (TURP), transurethral ultrasound ablation (TULSA), cryoablation, etc.), without prior positive biopsies, without artifacts, and with reasonably-well aligned sequences.

• MRI scanner: This algorithm was trained and evaluated exclusively on prostate biparametric MRI scans acquired with various commercial 1.5 Tesla or 3 Tesla scanners using surface coils from Siemens Healthineers, Erlangen, Germany or Philips Medical Systems, Eindhoven, Netherland. It does not account for vendor-neutral properties or domain adaptation, and in turn, the compatibility with scans derived using any other MRI scanner or those using endorectal coils is unkown.

• Sequence alignment and position of the prostate: While the input images (T2W, HBV, ADC) can be of different spatial resolutions, the algorithm assumes that they are co-registered or aligned reasonably well.

• General use: This model is intended to be used by radiologists for predicting clinically significant prostate cancer in biparametric MRI examinations. The model is not a diagnostic for cancer and is not meant to guide or drive clinical care. This model is intended to complement other pieces of patient information in order to determine the appropriate follow-up recommendation.

• Appropriate decision support: The model identifies lesion X as at a high risk of being malignant. The referring radiologist reviews the prediction along with other clinical information and decides the appropriate follow-up recommendation for the patient.

• Before using this model: Test the model retrospectively and prospectively on a diagnostic cohort that reflects the target population that the model will be used upon to confirm the validity of the model within a local setting.

• Safety and efficacy evaluation: To be determined in a clinical validation study.

Warnings

• Risks: Even if used appropriately, clinicians using this model can misdiagnose cancer. Delays in cancer diagnosis can lead to metastasis and mortality. Patients who are incorrectly treated for cancer can be exposed to risks associated with unnecessary interventions and treatment costs related to follow-ups.

• Inappropriate Settings: This model was not trained on MRI examinations of patients with prior treatment (e.g. radiotherapy, transurethral resection of the prostate (TURP), transurethral ultrasound ablation (TULSA), cryoablation, etc.), prior positive biopsies, artifacts or misalignment between sequences. Hence it is susceptible to faulty predictions and unintended behaviour when presented with such cases. Do not use the model in the clinic without further evaluation.

• Clinical rationale: The model is not interpretable and does not provide a rationale for high risk scores. Clinical end users are expected to place the model output in context with other clinical information to make the final determination of diagnosis.

• Inappropriate decision support: This model may not be accurate outside of the target population. This model is not designed to guide clinical diagnosis and treatment for prostate cancer.

• Generalizability: This model was developed with prostate MRI examinations from Radboud University Medical Center, Ziekenhuisgroep Twente, and Prostaat Centrum Noord-Nederland. Do not use this model in an external setting without further evaluation.

• Discontinue use if: Clinical staff raise concerns about the utility of the model for the intended use case or large, systematic changes occur at the data level that necessitate re-training of the model.

Common Error Messages

Information on this algorithm has been provided by the Algorithm Editors, following the Model Facts labels guidelines from Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020). 10.1038/s41746-020-0253-3