The Evolution of Radiology Image Annotation in the Era of Large Language Models
Summary
Although there are relatively few diverse, high-quality medical imaging datasets on which to train computer vision artificial intelligence models, even fewer datasets contain expertly classified observations that can be repurposed to train or test such models. The traditional annotation process is laborious and time-consuming. Repurposing annotations and consolidating similar types of annotations from disparate sources has never been practical. Until recently, the use of natural language processing to convert a clinical radiology report into labels required custom training of a language model for each use case. Newer technologies such as large language models have made it possible to generate accurate and normalized labels at scale, using only clinical reports and specific prompt engineering. The combination of automatically generated labels extracted and normalized from reports in conjunction with foundational image models provides a means to create labels for model training. This article provides a short history and review of the annotation and labeling process of medical images, from the traditional manual methods to the newest semiautomated methods that provide a more scalable solution for creating useful models more efficiently. Keywords: Feature Detection, Diagnosis, Semi-supervised Learning © RSNA, 2025
Figure 1: Hierarchy of annotation levels. Labels can be applied at many levels in the hierarchy, with the patient at the top of the structure, with imaging examinations and their component parts (eg, examination, series, image, and pixel). Annotations can be both human and machine generated.
Figure 2: Linkage between annotations, markup, and specific images. While annotations may appear as a markup (eg, line, bounding box, arrow, text) on a DICOM viewer, the actual annotation is stored in another discrete file or varying type (eg, JSON, DICOM SR, DICOM SEG) with linkages to specific instance unique identifier and pixel location. DICOM viewers will read both files and use the annotation file to display an overlay of the annotation on the original image. DICOM = Digital Imaging and Communications in Medicine, JSON = JavaScript Object Notation, SEG = segmentation information object, SR = structured report.
Figure 3: Traditional annotation workflow using human experts. An idiosyncratic labeling solution is developed for a specific research use case, which is then tested and revised following consultation with project designers. Experts are then recruited, trained, and monitored as they label examinations independently. QA = quality assurance.
Figure 4: Example of employing a large language model (LLM) to produce a tabular output of image features from a free-text report with an inventory of the feature, laterality, and mapping to a controlled lexicon ( International Classification of Diseases, Tenth Edition [ICD-10]). The LLM can then convert the content to a JavaScript Object Notation (JSON) object to make the content machine readable. PA = posteroanterior.
Figure 5: Automated generation of examination-level labels using original clinical reports and “helper” artificial intelligence (AI). In this context, Helper-AI is any tool that can provide automatic labels from pixel data. The Helper-AI tool evaluates the pixels in each image and creates normalized tags for examination and series name, imaging plane or projection, modality, body part, and contrast phase. A modified large language model extracts concepts from the report and normalizes the extracted concept to the most appropriate lexicon. A small group of experts perform random curation of the applied labels. NLP = natural language processing.
