Postdeployment Monitoring and Surveillance Methods, Guidelines, and Possibilities for AI in Radiology
Vasantha Kumar Venugopal; Suyash Anil Khubchandani; Charlene Liew; Felipe Kitamura; Rohit Takhar; Gerald Lip
RadioGraphics - Volume 46, Number 7 - https://doi.org/10.1148/rg.250173
Summary
As radiology AI systems move from predeployment validation to routine radiology practice, attention is shifting toward postdeployment monitoring and postmarket surveillance in a total product life cycle (TPLC) paradigm. In an operational sense, the human clinical oversight can be positioned along a spectrum encompassing human-in-the-loop (HITL), human-in-a-parallel loop (HIPL), human-on-the-loop (HOTL), human-over-the-loop (HOVL), and human-out-of-the-loop (HOOTL) models. Each of these models offers a trade between the verification workload and autonomy and risk. The authors provide a deeper understanding of the definitions first and then present HOTL as a pragmatic model of human-AI oversight for high-stakes imaging, demonstrating a balance between the trade-offs and benefits. The proposed monitoring system is based on two families of data points that do not require immediate determination of the ground truth, namely temporal stability of inputs and outputs and predictive divergence relative to a deployment initiation baseline. The authors also bring uncertainty quantification into the fray as a third element in helping prioritize reviews when labels are delayed or not continually feasible. The described threshold-based alerting system is paired with tiered escalation mechanisms and root cause analysis to distinguish degradation of the AI algorithm from data or integration pipeline issues. The result is an education-first proactive road map for postdeployment monitoring that allows preservation and prioritization of patient safety while enabling responsible scaling of radiology AI. © RSNA, 2026 See the invited commentary by Rouzrokh and Rouzrokh in this issue.
Figure 1: Conceptual depiction of an RCA workflow for AI monitoring alerts in a proposed HOTL system. env. = environment.
Figure 2: Graph shows predictive divergence (Jensen-Shannon divergence, weekly rolling divergence scores) for studies from a single scanner. A software update in week 6 preceded a threshold breach in week 7, prompting an RCA and a scoped rollback during week 8 as depicted by the arrow. During this period, the use of the triaging AI algorithm was paused while the errors were fixed in preprocessing. After retesting, the use of the AI algorithm was reinstated with divergence scores then reverting back to normal in week 10.
Keywords:
MEDLINE | Product (mathematics) | Interventional radiology | Patient safety
References
1. Hill DLG. AI in imaging: the regulatory landscape. Br J Radiol 2024;97(1155):483–491.
2. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25(1):44–56.
3. Wiens J, Saria S, Sendak M, et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med 2019;25(9):1337–1340. [Published correction appears in Nat Med 2019;25(10):1627.]
4. Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019;17(1):195.
5. McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature 2020;577(7788):89–94. [Published correction appears in Nature 2020;586(7829):E19.]
6. de Vries CF, Colosimo SJ, Staff RT, et al; iCAIRD Radiology Collaboration. Impact of different mammography systems on artificial intelligence performance in breast cancer screening. Radiol Artif Intell 2023;5(3):e220146.
7. Sahiner B, Chen W, Samala RK, Petrick N. Data drift in medical machine learning: implications and potential remedies. Br J Radiol 2023;96(1150):20220878.
8. Roschewitz M, Khara G, Yearsley J, et al. Automatic correction of performance drift under acquisition shift in medical image classification. Nat Commun 2023;14(1):6608.
9. Floridi L, Cowls J, Beltrametti M, et al. AI4People—An ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach (Dordr) 2018;28(4):689–707.
10. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA 2017;318(6):517–518.
11. U.S. Food and Drug Administration. Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. https://www.fda.gov/media/145022/download . Published January 2021. Accessed August 18, 2025.
12. U.S. Food and Drug Administration. Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles. https://www.fda.gov/medical-devices/software-medical-device-samd/predetermined-change-control-plans-machine-learning-enabled-medical-devices-guiding-principles . Published October 2022. Accessed August 18, 2025.
13. Brady AP, Allen B, Chong J, et al. Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement from the ACR, CAR, ESR, RANZCR and RSNA. Radiol Artif Intell 2024;6(1):e230513.
14. Ross J, Hammouche S, Chen Y, et al; Royal College of Radiologists AI Working Group. Beyond regulatory compliance: evaluating radiology artificial intelligence applications in deployment. Clin Radiol 2024;79(5):338–345.
15. Dolin P, Li W, Dasarathy G, Berisha V. Statistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health. arXiv 2025. Preprint posted online June 6, 2025. https://arxiv.org/abs/2506.05701 .
16. Strickland NH. Quality assurance in radiology: peer review and peer feedback. Clin Radiol 2015;70(11):1158–1164.
17. Daye D, Wiggins WF, Lungren MP, et al. Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How? Radiology 2022;305(3):555–563. [Published correction appears in Radiology. 2022 Oct;305(1):E62.]
18. Oala L, Suresh H, Madras D, et al. Monitoring machine learning models in healthcare. In: Proceedings of the NeurIPS 2020 Workshop on Machine Learning for Health (ML4H). 2020.
19. Dombrowski M, Prenner A, Kainz B. Bias Assessment and Data Drift Detection in Medical Image Analysis: A Survey. arXiv 2024. Preprint posted online September 26, 2024; doi:.
20. Koch LM, Baumgartner CF, Berens P. Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. NPJ Digit Med 2024;7(1):120.
21. Roschewitz M, Mehta R, Jones C, Glocker B. Automatic dataset shift identification to support root cause analysis of AI performance drift. arXiv 2024. Preprint posted online November 12, 2024. https://arxiv.org/abs/2411.07940 .
22. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology 2023;307(4):e222176.
23. Patel BN, Rosenberg L, Willcox G, et al. Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit Med 2019;2:111. [Published correction appears in NPJ Digit Med 2019;2:129.]
24. U.S. Food and Drug Administration (FDA). Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing-submission-recommendations . Published online January 2025. Accessed August 18, 2025.
25. Agência Nacional de Vigilância Sanitária (ANVISA). Resolução de Diretoria Colegiada – RDC No. 657, de 24 de março de 2022: Dispõe sobre Software como Dispositivo Médico (SaMD). https://in.gov.br/en/web/dou/-/resolucao-de-diretoria-colegiada-rdc-n-657-de-24-de-marco-de-2022-389603457 . Published online March 2022. Accessed August 18, 2025.
26. European Union. Regulation (EU) 2017/745 on medical devices. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745 . Published online 2017. Accessed August 18, 2025.
27. Health Canada. Draft Guidance on Pre-market Requirements for Machine Learning-enabled Medical Devices. https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/activities/guidance-documents/draft-guidance-machine-learning.html . Published online March 2023. Accessed August 18, 2025.
28. Medicines and Healthcare products Regulatory Agency (MHRA). Software and AI as a Medical Device Change Programme. https://www.gov.uk/government/publications/software-and-ai-as-a-medical-device-change-programme . Published online October 2023. Accessed August 18, 2025.
29. Pharmaceuticals and Medical Devices Agency (PMDA). Basic Concept of the Evaluation of Medical Device Programs Using Artificial Intelligence. https://www.pmda.go.jp/english/safety/info-services/ai-healthcare/0021.html . Published online June 2022. Accessed August 18, 2025.
30. Senado Federal do Brasil. Projeto de Lei n\textsuperscripto 2338. de 2023: Estabelece princípios, direitos e deveres para o uso da inteligência artificial no Brasil. https://www25.senado.leg.br/web/atividade/materias/-/materia/157233 . Published online 2023. Accessed August 18, 2025.
31. Singapore Health Sciences Authority (HSA). AI in Healthcare Guidelines (AIHGle). https://www.hsa.gov.sg/ai-in-healthcare . Published online 2022. Accessed August 18, 2025.
32. Venugopal VK, Gupta A, Takhar R, Mahajan V. New Epochs in AI Supervision: Design and Implementation of an Autonomous Radiology AI Monitoring System. arXiv 2023. Preprint posted online November 24, 2023. doi:.
33. Chen M, Wang Y, Wang Q, et al. Impact of human and artificial intelligence collaboration on workload reduction in medical image interpretation. NPJ Digit Med 2024;7(1):349.
34. Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J 2019;6(2):94–98.
35. Gawlikowski J, Tassi CRN, Ali M, et al. A survey of uncertainty in deep neural networks. Artif Intell Rev 2023;56(S1):1513–1589.
36. Benjamin M, Engelhard G, Aisen A, Aradi Y, Benjamin E. A Multisite, Report-Based, Centralized Infrastructure for Feedback and Monitoring of Radiology AI/ML Development and Clinical Deployment. arXiv 2020. Preprint posted online August 31, 2020. https://arxiv.org/abs/2008.13781 .
37. Integrating the Healthcare Enterprise (IHE) International. AI Interoperability in Imaging: IHE Radiology White Paper (Revision 1.1). https://www.ihe.net/resources/technical_frameworks/#Radiology . Published online October 2021. Accessed August 18, 2025.
38. Cummings ML. What self-driving cars tell us about AI risks. IEEE Spectrum . https://spectrum.ieee.org/self-driving-cars-2662494269 . Published July 30, 2023. Accessed August 20, 2025.
39. Chow J, Lee R, Wu H. How do radiologists currently monitor AI in radiology and what challenges do they face? An interview study and qualitative analysis. J Imaging Inform Med 2025. 10.1007/s10278-025-01493-8 . Published online April 8, 2025.
40. Abdar M, Pourpanah F, Hussain S, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inf Fusion 2021;76:243–297.
41. Faghani S, Moassefi M, Rouzrokh P, et al. Quantifying Uncertainty in Deep Learning of Radiologic Images. Radiology 2023;308(2):e222217.
42. Royal College of Physicians. National Early Warning Score (NEWS) 2: Standardising the Assessment of Acute-Illness Severity in the NHS. Royal College of Physicians ; 2017.
43. Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007;16(5):387–399.
44. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12(6):458–464.
45. Liew CJY. Autonomous algorithmic monitoring of a deep-learning chest radiograph AI model using temporal divergences in a real-world clinical setting. Abstract RPS 1205:276 In: European Congress of Radiology (ECR) 2024; March 1, 2024; Vienna, Austria. https://connect.myesr.org/course/recent-development-in-ai-for-lung-nodule-detection/ .
46. National Institute of Standards and Technology (NIST). AI Risk Management Framework (AI RMF 1.0) Playbook. https://airc.nist.gov/airmf-resources/playbook/ . Published online 2023. Accessed August 18, 2025.
47. Mello-Thoms C, Mello CAB. Clinical applications of artificial intelligence in radiology. Br J Radiol 2023;96(1150):20221031.
48. Erdal BS, Dikici E, Bigelow M, Prevedello LM, White RD. Integrating AI into Radiology Workflow: Levels of Research, Production, and Feedback Maturity. arXiv 2019. Preprint posted online October 14, 2019. https://arxiv.org/abs/1910.06424 .
49. Cardoso MJ, Moosbauer J, Cook TS, et al. RAISE -- Radiology AI Safety, an End-to-end lifecycle approach. arXiv 2023. Preprint posted online November 24, 2023; doi:.
50. World Health Organization. Regulatory Considerations on Artificial Intelligence for Health. https://www.who.int/publications/i/item/9789240074148 . Published 2023. Accessed August 18, 2025.
51. U.S. Department of Commerce. National Institute of Standards and Technology. AI Risk Management Framework (AI RMF 1.0) . https://doi.org/10.6028/NIST.AI.100-1 . Published 2023. Accessed August 20, 2025.
52. Gichoya JW, Banerjee I, Bhimireddy AR, et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit Health 2022;4(6):e406–e416.
53. Busch F, Kather JN, Johner C, et al. Navigating the European Union Artificial Intelligence Act for Healthcare. NPJ Digit Med 2024;7(1):210.
54. Health Sciences Authority (Singapore). Public Consultation on the Proposed Exemption from Manufacturer’s Licensing and Product Registration Requirements for Artificial Intelligence–Software as a Medical Device Developed by Selected Public Healthcare Entities for Use in Public Healthcare. https://www.hsa.gov.sg/announcements/public-consultation/proposedexemption-AI-SaMD . Published online May 19, 2025. Accessed August 20, 2025.
Other resources
The RSNA Pediatric Bone Age Machine Learning Challenge
Purpose The Radiological Society of North America (RSNA) Pediatric Bone Age Machine Learning Challenge was created to show an application of machine learning (ML) and artificial intelligence (AI) in medical imaging, promote collaboration to catalyze AI model creation, and identify innovators in medical imaging. Materials and Methods The goal of this challenge was to solicit individuals and teams to create an algorithm or model using ML techniques that would accurately determine skeletal age in a curated data set of pediatric hand radiographs. The primary evaluation measure was the mean absolute distance (MAD) in months, which was calculated as the mean of the absolute values of the difference between the model estimates and those of the reference standard, bone age. Results A data set consisting of 14 236 hand radiographs (12 611 training set, 1425 validation set, 200 test set) was made available to registered challenge participants. A total of 260 individuals or teams registered on the Challenge website. A total of 105 submissions were uploaded from 48 unique users during the training, validation, and test phases. Almost all methods used deep neural network techniques based on one or more convolutional neural networks (CNNs). The best five results based on MAD were 4.2, 4.4, 4.4, 4.5, and 4.5 months, respectively. Conclusion The RSNA Pediatric Bone Age Machine Learning Challenge showed how a coordinated approach to solving a medical imaging problem can be successfully conducted. Future ML challenges will catalyze collaboration and development of ML tools and methods that can potentially improve diagnostic accuracy and patient care. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Siegel in this issue.
The RSNA International COVID-19 Open Radiology Database (RICORD)
The coronavirus disease 2019 (COVID-19) pandemic is a global health care emergency. Although reverse-transcription polymerase chain reaction testing is the reference standard method to identify patients with COVID-19 infection, chest radiography and CT play a vital role in the detection and management of these patients. Prediction models for COVID-19 imaging are rapidly being developed to support medical decision making. However, inadequate availability of a diverse annotated data set has limited the performance and generalizability of existing models. To address this unmet need, the RSNA and Society of Thoracic Radiology collaborated to develop the RSNA International COVID-19 Open Radiology Database (RICORD). This database is the first multi-institutional, multinational, expert-annotated COVID-19 imaging data set. It is made freely available to the machine learning community as a research and educational resource for COVID-19 chest imaging. Pixel-level volumetric segmentation with clinical annotations was performed by thoracic radiology subspecialists for all COVID-19–positive thoracic CT scans. The labeling schema was coordinated with other international consensus panels and COVID-19 data annotation efforts, the European Society of Medical Imaging Informatics, the American College of Radiology, and the American Association of Physicists in Medicine. Study-level COVID-19 classification labels for chest radiographs were annotated by three radiologists, with majority vote adjudication by board-certified radiologists. RICORD consists of 240 thoracic CT scans and 1000 chest radiographs contributed from four international sites. It is anticipated that RICORD will ideally lead to prediction models that can demonstrate sustained performance across populations and health care systems. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Bai and Thomasian in this issue.
