Deep Learning in Neuroradiology: A Systematic Review of Current Algorithms and Approaches for the New Wave of Imaging Technology

Introduction

Artificial intelligence (AI) describes the use of computers or machines to mimic human cognition ^1,2. Machine learning (ML) is a subfield of AI in which models are designed to generate a desired output from an existing set of data without explicit instructions. ML can be further subdivided into supervised and unsupervised learning. In supervised learning, models predict a specific label or outcome—for example, predicting whether an image portrays a cat or dog or determining whether a chest radiograph is normal or abnormal. This requires the input data to be explicitly labeled with the desired output. In unsupervised learning, models produce a representation of the data based on their underlying distribution, which does not require explicit labeling. An overview of the characteristics of supervised and unsupervised learning can be seen in Figure 1 . Deep learning (DL) can be considered a subfield of ML that uses deep neural networks (DNNs) as models for various tasks that include both supervised and unsupervised learning. A representation of the different ML architectures is shown in Figure 2 .

Figure 1: Classification of artificial intelligence architectures.

Figure 2: Overview of supervised versus unsupervised learning.

DNNs are modeled after synaptic connections in the human brain. DNNs can be described as a series of neuronal layers with weighted connections; these weights reflect connection strength and can be adjusted by comparing a model’s output with the labeled ground truth of the data. DL has seen the most success in image recognition with the development of convolutional neural network (CNN) architectures. CNNs are DNNs designed specifically to accommodate the intrinsic structure of images, although their use also extends beyond image recognition. For further details, see the review article published by Chartrand et al ².

DL has shown substantial potential in neuroimaging because of its high volume of multimodal data and its critical role in diagnosis of neurologic diseases. DL may be implemented at multiple points in patient care and has demonstrated efficacy in triaging high-priority images, recommending appropriate follow-up studies, and detecting lesions for long-term management ^3–5. In addition, DL applications in neuroradiology aim to reduce time to diagnosis and physician error while providing clinicians and researchers with the ability to work with extraordinarily large or impractical datasets.

To date, there have been many published reviews and peer-reviewed primary articles describing the role of AI in neuroradiology ^2,4,6–10. The goal of this systematic review was to synthesize the current literature, analyze various approaches and current results, and discuss opportunities for future research and standardization to develop the first thorough compendium of DL applications in neuroradiology.

Materials and Methods

An initial systematic literature search following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for systematic review was conducted through September 1, 2019, using PubMed, Cochrane, and Web of Science databases. This search was conducted in two phases—one from January 1, 2000, to June 1, 2019, and a second identical search conducted from June 1, 2019, to September 1, 2019, to provide more recent and updated information. Boolean search terms included any of the following: “artificial intelligence” OR “neural network” OR “machine learning” OR “deep learning” in combination with any the following terms: “neuroradiology” OR “brain imaging” OR “neuroimaging.” Medical subject headings (or MeSH) were used to further stratify studies. A university librarian was consulted to evaluate search phrases and databases. Additional studies were also included through article references identified in primary literature review. A PRISMA diagram is shown in Figure 3 .

Figure 3: PRISMA Guidelines for Systematic Review Flow Diagram. A = Records screened were based on brief review of the publication title or abstract; excluded papers included those that were not in English, were published before the year 2000, and did not describe deep learning research. B = Only peer-reviewed articles that evaluated deep learning modalities in neuroradiology were included. Excluded publications reviewed the technological field, provided qualitative assessment of deep learning, provided low-quality quantitative analysis per GRADE Working Group Criteria, or did not apply deep learning to solve an imaging task. Please refer to the Methods for a more detailed description. C = Secondary literature search was conducted through September 1, 2019, using identical search criteria and methodology with updated findings. GRADE = Grading of Recommendations Assessment, Development and Evaluation, PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Only peer-reviewed primary literature articles published in English after January 1, 2000, were included in this study. Articles were required to describe DL or neural network modalities and quantitatively evaluate DL or neural network modalities. Excluded articles included those that reviewed the field of AI or DL, described the technology underlying the field, provided qualitative assessment of AI, did not describe DL in the context of neuroradiology, or provided indirect, unclear, or imprecise study methods or results per modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group criteria ( http://www.gradeworkinggroup.org/ ) . Abstracts and conference proceedings that were not published in peer-reviewed journals were not included to preserve study quality and to avoid double representation by follow-up studies. Exclusion was determined by two readers (A.D.Y., D.L.C.) and was independently confirmed by two separate readers (I.P., F.K.). Citations for articles considered for analysis are listed separately in Appendix E1 (supplement).

Data were independently extracted by two trained student neuroradiology researchers (A.D.Y., D.L.C.) with backgrounds in systematic review, neuroimaging, and data science research. An experienced DL practitioner (F.K.) then reviewed extracted data and, if necessary, resolved any inconsistent findings. Data were extracted into Microsoft Excel using an internal data form and categorized as described later in this article. Missing data were obtained (I.P.) through careful article review and additional literature search if needed; information not included in this review was not readily available.

Included articles were divided by imaging modality. Included modalities were CT, functional MRI, MRI, PET, and US. Articles were further categorized based on neural network format (convolutional vs nonconvolutional) and by application (disease, application). For each application, imaging task, purpose, data source, algorithm type, outcome metrics, and qualitative results were evaluated and reported using descriptive statistics.

Results

On initial literature review, 6920 articles were identified; after duplicate removal, article exclusion, and subsequent literature search update, 155 articles were included for analysis. Details of the literature search can be found in Figure 3 . Articles were divided by imaging modality: MRI ( n = 115), functional MRI ( n = 19), CT ( n = 9), PET ( n = 18), and US ( n = 1). Seven articles described multimodal applications. A diagram describing the PRISMA flowchart can be seen in Figure 3 . Because of the large number of articles and DL applications, details of each article have been described separately. An extensive table with details for individual applications and relevant citations may be seen in Appendix E1 (supplement).

MRI Articles

One hundred fifteen studies were conducted describing DL applied to MRI. Most of the projects described algorithms tasked with image classification ( n = 25), segmentation ( n = 42), generation ( n = 5), detection ( n = 10), reconstruction ( n = 8), or prediction ( n = 20). Applications were extremely varied and ranged from determining brain age and prediction of onset of Alzheimer disease to brain tumor segmentation and white matter lesion detection. Importantly, data sources and qualitative assessment metrics were vastly different between each project. Notable data sources included the Alzheimer’s Disease Neuroimaging Initiative (ADNI), Brain Tumor Segmentation Challenge, the Ischemic Stroke Lesion Segmentation Challenge, and various private institutional datasets. Of these studies, 77 used datasets where n was greater than 100; only 18 used datasets where n was greater than 1000. Most of these studies used CNNs ( n = 94). Articles were evaluated using a variety of metrics, including area under the curve (AUC), Dice similarity coefficients (DSC), accuracy, sensitivity, specificity, or other metrics. A full list of application purposes, data sources, model types, and assessment metrics may be seen in Table E1 (supplement).

Functional MRI Articles

Nineteen articles described DL applications in functional MRI. These projects were primarily composed of CNNs ( n = 10) and were designed to evaluate brain activity during rest, sensorimotor tasks, and mild cognitive impairment. Data sources included the ADNI, Consortium for Reliability and Reproducibility, and institutional databases. Articles were assessed using different metrics. The most common metric was accuracy ( n = 18), followed by sensitivity ( n = 8) and specificity ( n = 8). Findings may be seen in Table E2 (supplement).

CT Articles

Nine articles were identified describing the use of DL in CT imaging. Of these, all were two-dimensional or three-dimensional CNNs. Tasks included image generation ( n = 2), detection ( n = 2), classification ( n = 3), prediction ( n = 1), and segmentation ( n = 1) with purposes ranging from reconstruction of CT data to identification of brain changes after hemorrhage, Alzheimer disease, and traumatic events. Data sources included the Icahn School of Medicine AI Consortium, CT Low-Dose Grand Challenge, and various private institutional databases. Metrics varied between studies and included AUC, sensitivity, specificity, positive predictive value, negative predictive value, time, and mean absolute error. Findings are presented in Table E3 (supplement).

PET Articles

Eighteen articles discussed the use of DL in PET imaging. Many conducted multimodal analysis of PET and MRI. Models were varied and included CNNs, multitrait DL, stacked autoencoder, conditional average estimator neural networks, and generative adversarial networks. Imaging tasks were varied and included classification ( n = 9), detection ( n = 2), generation ( n = 4), reconstruction ( n = 1), and prediction ( n = 2) used primarily for Alzheimer disease and amyloid burden classification. Data from these studies were obtained from the ADNI, Seoul National University Hospital, Parkinson’s Progression Markers Initiative, and Korean Brain Aging Study databases; the vast majority were externally validated through ADNI ( n = 13). Algorithm assessment varied and included accuracy, sensitivity, specificity, and time. Results may be seen in Table E4 (supplement).

US Articles

One article was identified using a two-dimensional CNN to segment and classify transventricular fetal US. This study used institutional data ( n = 19 838) and was analyzed using DSC, precision, and recall. This project is described in Table E5 (supplement).

Discussion

The role of AI in neuroradiology has demonstrated remarkable growth in the past 2 decades. In this study, we identified 155 articles detailing DL applications in neuroimaging. Of these, the majority (74%) described MRI applications and 42 of 115 (37%) MRI algorithms were tasked with image segmentation. This was expected because MRI uses a large volume of raw three-dimensional image data and easily allowed for the early study of basic ML tasks, such as segmentation. In this article, however, we highlighted multiple studies describing the use of DL in five separate imaging modalities—MRI, functional MRI, CT, PET, and US—used in neuroradiology and applied to an extremely wide range of neurologic diseases ranging from Alzheimer disease and vascular injuries to fetal brain development and cerebral neoplasms. Recent research has also seen DL expand from image segmentation to more complex tasks, such as image classification, generation, regression, detection, and reconstruction, among others. These new imaging techniques and a growing number of available assessment metrics underscore the rapid expansion and clinical potential of DL in neuroradiology.

Generalizability, Reproducibility, and Clinical Value

Despite these successes, the articles identified in this study demonstrated limited clinical applicability. Although a small number of groups have initiated clinical trials, most DL applications were designed primarily for research or architecture development; therefore, there was no standardized reporting scheme for ML models. In this study alone, we identified more than 15 unique quantitative assessment methods, each with differing measurement approaches and units. The most commonly used metrics included AUC ( n = 47), DSC ( n = 57), sensitivity ( n = 60), specificity ( n = 55), and accuracy ( n = 95). As a result, study interpretation and direct comparison or meta-analysis was challenging for readers both familiar and unfamiliar with ML.

Of the 78 articles identified in this study, 65 (42%) were tested on institutional data; only 16 of the 65 were validated against external data. Furthermore, 53 (34%) studies used a combined dataset of less than 100, and 124 (80%) used a combined dataset of less than 1000. These findings limit model generalizability; because the quality and size of reported datasets may significantly influence results, findings drawn from limited or internal data sources may not reflect true algorithm quality when tested against external resources. Although this finding may be challenging in circumstances when specific large datasets are not publicly available, most of the current DL research used raw MRI and CT data that were readily available through open source datasets such as ADNI, Brain Tumor Segmentation Challenge, and Ischemic Stroke Lesion Segmentation Challenge. As the number of such imaging databases—specific and otherwise—are gathered, DL researchers will continue to find it easier to validate their algorithms against larger, broader datasets.

Reproducibility was also severely limited; almost all articles identified in this study failed to describe their algorithm implementation methods. Whereas medical data are under strict privacy restriction, models and associated weights are not. As the field continues to advance and algorithms require validation for clinical use, researchers may need to release their models and code for readers to evaluate, reproduce, and test on external data.

Finally, very few articles provided clinical validation for their models. Whereas retrospective and local model validation was useful, lack of clinical value may prevent a DL development from being implemented in a health care setting. When DL was first being developed, emphasis was placed on creating research-based, proof-of-concept models. As this technology continues to progress, DL has become increasingly democratized and the value of training a model without discussing its potential utility is rapidly dissipating.

This disconnect between AI scientists and physicians may be influenced by limited physician understanding of AI and DL technology. However, growing collaboration between researchers, physicians, and computer scientists may soon allow DL to be effectively integrated in clinical practice. Similarly, greater standardization will allow for future comparisons and meta-analyses to provide clinical value to DL research. Although there is an inherent incentive against sharing data because of competition between researchers, repressed information directly hinders reproducibility and limits the development of this field.

Limitations and Future Directions

Several limitations exist in this study. First, systematic reviews were inherently limited by search criteria; recent studies or studies that did not fall under the described search may not have been included or may have been subjectively excluded from review. This holds especially true because of the rapidly expanding nature of the AI field. This limitation was minimized by using overlapping Boolean search phrases and consultation with a university librarian. In addition, this review only described peer-reviewed literature and did not include unpublished or internal data. Because of the review method, there was also a risk of study selection bias that was addressed by using multiple independent reviewers. Furthermore, outside the initial literature search, formal quality assessment of included articles by GRADE criteria was not conducted. Despite these limitations, however, our findings consistently demonstrated the need for further research standardization in the field of DL in neuroradiology.

In this study, we identified substantial limitations to neuroradiology DL reporting standardization, generalizability, reproducibility, and clinical validity. We suggest that future researchers propose a framework with specific reporting criteria for future DL studies. Researchers may benefit from using a structure similar to the table presented in Appendix E1 (supplement) of this article; commonly used and potentially appropriate standardized metrics may include AUC, DSC, sensitivity, specificity, and accuracy. Further research validating current algorithms against larger, external databases is also needed to demonstrate model generalizability—especially if developed and tested using only private internal data. Finally, we recommend that DL researchers share or provide open-source code for their algorithms to allow other institutions to further validate their data. This crosstalk may allow researchers to draw inspiration from similar studies and propel a new wave of DL development. By addressing each of these challenges and demonstrating clinical utility, the field of DL may soon be implemented in patient care.

Conclusion

The field of AI in neuroradiology has grown tremendously in the past decade. In this study, we presented a compendium of current DL research, compared recent findings, and discussed directions for future work. As the field continues to grow and clinicians become accustomed to AI, DL technology may soon play a major role in the imaging, diagnosis, prognosis, treatment, and follow-up of neurologic diseases.