Decreasing false-positive detection of intracranial hemorrhage (ICH) using RAPID ICH 3

Introduction

Intracranial hemorrhage (ICH) affects approximately 23 to 43 per 100,000 people in the USA yearly, 1 , 2 steadily rising in recent years 2 and associated with significant morbidity and mortality. 3 – 5 Prompt detection of ICH remains among the most critical factors driving outcomes. 3 , 6 Delays in recognition of ICH have previously been shown to predict poor outcome, 7 , 8 a problem further compounded in over-burdened healthcare systems since the advent of the COVID-19 pandemic. 8

The use of automated software 9 – 11 designed to detect hemorrhage on non-contrast head CT (NCCT), the most commonly acquired initial image in the acute setting, 12 can be an effective method to address potential delays. RAPID ICH (iSchemaView, Menlo Park, CA) is a deep learning convolutional neuronal network (CNN) based software that was specifically designed for the acute identification of ICH for the purposes of clinical triage and management. 10 , 13 The RAPID software platform already includes tools for identification of large-vessel occlusion strokes, 14 , 15 which is available to use in multiple centers worldwide.

RAPID ICH has previously been shown as effective in detection of ICH as well as for estimation of volumes of both intraparenchymal and intraventricular components. 10 However, automated detection software can be limited due to the requirement for physicians to quickly separate true positive from false-positive notifications. Specifically, previous versions of RAPID ICH software 10 have exhibited false positive rates of approximately 5% that, while low, still resulted in 1 out of every 20 scans generating spuriously positive ICH detection. Particularly in high acuity settings or in those without sub-specialist expertise, the ability to reduce false-positive notifications while still maintaining sensitivity is paramount to improved outcomes. Our aim was to test the latest version of RAPID ICH (version 3.0) in its ability to accurately detect ICH and assess enhancements aimed at reduction of potential false-positive ICH detection.

Methods

Image Acquisition

This retrospective cohort study complied with the Health Insurance Portability and Accountability Act. Institutional review board approval was obtained at each site, and the need for informed consent was waived. NCCTs were obtained from 15 hospital centers (all either primary or comprehensive stroke centers) or from seven different clinical trials across international sites from June 2006 to February 2013. Scans were excluded for ¹ severe CT artifacts, ² prior neurosurgical procedures, ³ recent intravenous contrast, or ⁴ ICH <0.4 ml (the specified detection limit of the software). Images were not manipulated or corrected for tilting prior to detection analysis. The presence or absence of ICH was confirmed separately by at least two neuroradiology experts with a third expert used as arbiter in cases for which there was not a consensus in detection.

NCCT studies from multiple vendors were acquired in the axial plane with section thickness that ranging from 1 to 5 mm. Radiation doses varied by vendor and location in this retrospective study, which was intended to sample variations encountered in standard radiologic practice. Regions of ICH were manually outlined by the respective neuroradiologists and stored as binary masks that were used as ground truth for the training analysis. Ground truth masks were randomly split into training (80%) and testing (20%) groups.

RAPID Machine Learning Detection

Rapid ICH was developed using a deep convolutional neural network based on a U-Net architecture, consisting of an encoder (contracting feature maps) and a decoder section (with expanding feature map dimensions) with connections between the encoder and decoder sections. The system was trained on a cohort of more than 500 NCCT examinations. 16 All head CTs analyzed by RAPID ICH were postprocessed into 5 mm thick axial slices, with 48 images per study analyzed for the training dataset.

The training dataset included NCCT with intraparenchymal hemorrhage, intraventricular hemorrhage, extra-axial hemorrhage, subarachnoid hemorrhage, and no hemorrhage. To improve specificity of ICH detection, particularly for those occurring extra-axially, a final brain masking step was implemented to segment out brain tissue and adjacent bone regions. In addition, to reduce motion artifacts, a motion correction model which consists of a spatial and temporal parameterization of human head motion was applied.

Statistical Analysis

ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by RAPID ICH 3 compared to manual detection were calculated for the purposes of analysis. All calculations were conducted using SPSS v.28 software (SPSS Inc., Chicago, Illinois, USA).

Results

A total of 881 NCCT scans were included in the primary analysis. Mean processing time with the software was <40 seconds per scan. The scans were derived from a cohort of patients that included 463 male (52.55%) and 363 female (41.20%) overall with an average age of 62.54 (SD: 19.53) years. Demographic data was missing for 55 cases (6.20%).

The software correctly identified 453/463 ICH-positive cases and 416/418 ICH-negative cases validated by neuroradiologists ( Table 1 ). Examples of ICH detection are included in Figure 1 . In comparison to neuroradiology expert review, RAPID ICH detection resulted in a sensitivity of 97.84% (CI: 96.06–98.96) and specificity 99.52% (CI: 98.28–99.94), positive predictive value 99.56% (CI: 98.27–99.89), and negative predictive value 97.65% (CI: 95.75–98.71) for ICH detection. The positive and negative likelihood ratios for ICH detection were similarly favorable at 204.49 (CI: 51.31–814.99) and 0.02 (CI: 0.01–0.04) respectively. Use of RAPID ICH 3.0 resulted in two false-positive cases as well as 10 false-negative cases ( Figure 2 ). Review of the two false-positive indicated that the automated software inappropriately classified ¹ a component of a chronic subdural and ² a component of the parietal bone as a suspected ICH. Review of the 10 false-negative cases indicated the automated software failed to identify 10 small ICHs with a median volume of 1.60 ml (IQR: 0.55–2.42). No ICH above 4 ml was missed by the RAPID ICH 3.0 detection.

Fig. 1: Representative images depicting RAPID ICH ability to identify ICH. (A) A primary right temporo-occipital ICH depicted in the full panel NCCT with (B) RAPID ICH detection with representative images. (C) A primary left frontal ICH depicted in full panel NCCT with (D) RAPID ICH detection with representative images.

Fig. 2: Examples of false-positive and false-negative ICH cases. (A) An example of a false-negative showing a small left subcortical ICH that was missed by the software. The only two false-positive cases are shown which identified (B) a component of a chronic subdural and (C) a component of the parietal bone as a suspected ICHs.

Table 1.: Comparison of ICH detection by radiologist vs RAPID ICH.

Conclusions

Using a large data set of nearly 900 patients, the RAPID ICH 3.0 automated software maintained not only a high sensitivity for detection of ICH, but also essentially eliminated false-positive identifications. It was previously reported 10 that version 1.0 of this software had a sensitivity of 97% and specificity of 95%; importantly, enhancements to the newest iteration reported herein yielded improvements in specificity now exceeding 99% in a large and diverse retrospective cohort, without any decrement in sensitivity.

In comparison to other studies using deep learning tools 17 – 18 , the results above suggest the potential for significantly improved levels of performance for automated ICH detection tools, particularly with respect to false-positive identifications. The software resulted in only two false-positive identifications, including misclassification of a chronic subdural in one case and volume averaging with the parietal bone in the other. We observed 10 false-negative ICH cases, most of which were due to failure to detect small lesions with a median volume of 1.60 ml.

The strengths of this analysis include the large number of real-world cases reviewed from multiple different centers. Limitations of this work include the retrospective nature of the analysis that is limited in its generalizability to the institutions involved; nevertheless, the use of imaging obtained in varied environments and across mixed vendor platforms is projected to facilitate translation in other settings. Additionally, RAPID ICH detection is not designed to detect ICHs < 0.4 ml and further enhancements to improve performance beyond a priori detection limits are important avenues for further development in machine learning tools. We choose not to assess scans with major artifacts or a neurosurgical procedures as these impose barriers for interpretation even for experienced radiologists, however, are common occurrences in practice. Finally, while the utility of this software can be helpful in clinical management, it requires oversight from physicians with neuroradiologic expertise to identify rare false-positives and negatives.

In the same way automated CT-perfusion analysis influenced ischemic stroke management, 14 , 15 we propose that an automated detection tools for ICH can facilitate ICH management in the acute setting. As methods to improve the sensitivity and specificity of this technique are rapidly improving, implementation ICH detection software can help prevent erroneously treating a patient with a thrombolytic when a quick radiology review of the scan does not detect a brain hemorrhage, but the software does. It can also provide faster identification of larger ICHs that need emergent therapy as suggested by some existing AI assisted workflows, 19 , 20 as well as identification of small or subtle ICHs in settings lacking subspecialty imaging expertise.

Notas

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.