Study design

This study was approved by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen (approval number 23-11557-BO, 20.02.2024). Due to the study’s retrospective nature, the requirement of written informed consent was waived by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen. This study adheres to the TRIPOD-LLM [17] guidelines where applicable. Figure 1 illustrates the dataset preparation steps and general workflow of the study.

Fig. 1

Methods and study design. The complete workflow of the study is summarized, including steps for dataset preparation, division into training and test set and fine-tuning. LLM, large language model; GOÄ, Gebührenordnung für Ärzte. The illustration was created with BioRender.com

Inclusion and exclusion criteria

The study cohort included all radiology reports from privately insured adults (≥ 18 years) treated at the University Hospital Essen between 1999 and 2025 with documented GOÄ codes in the Picture Archive Communication System (PACS). No other demographic exclusions were made. For manual evaluation, 500 Real-World Samples were drawn from the validation set, excluding reports consisting solely of invariant normal-finding templates to ensure clinical diversity. From this subset, a cleaned sample (n = 350) was derived by excluding 150 reports dominated by institution-specific surcharge logic rather than extractable clinical text. Additionally, 163 individual codes (out of 586 total codes) representing pure administrative surcharges without textual correlations were removed from the ground truth to isolate semantic understanding.

Dataset

A total of 499,601 radiology reports from 124,497 patients were retrieved using the Python programming language with pandas [18] (Version 2.3.1) and slqalchemy [19] package (Version 2.0.41). Demographic information of the patients was analyzed, and dataset characteristics like the number of tokens were calculated using the nltk [20] package (Version 3.9.1) and are presented in Table 1.

Table 1 Patient demographics and radiology report characteristics

The radiology reports were originally retrieved from the hospital’s PACS in base64 encoded format. The first author performed the necessary preprocessing, which was limited strictly to decoding these strings into plain text using the built-in Python library base64 (Version 3.13.5). No additional text normalization techniques, such as stemming, lemmatization, or stop-word removal, were applied.

Ground truth

A total of 1,799,971 manually documented GOÄ codes were retrieved from the separate hospital billing system to serve as ground truth. Comprising 238 unique codes, these labels are generally robust, but they inherently reflect historical documentation biases and potential human errors that the model may reproduce. The dataset exhibits class imbalance (see Supplementary Fig. S1), where a small subset of high-frequency codes dominates the distribution.

Data partitions

The dataset was split 90/10% into a training and a test set on a patient basis, so that a patient’s data was either part of the training or the test set. After the initial split, five-fold cross-validation was performed on the training data. Five models were trained for each cross-validation fold. The initial 10% hold-out test set was used for evaluation.

Model training

The MediPhi-Instruct model from Microsoft was used as the basis for further fine-tuning and evaluation. This model has 4B parameters and has already been trained on clinical data and achieved state-of-the-art results on medical and clinical scenarios [21]. This model was further fine-tuned using pairs of radiology reports and GOÄ codes described in the previous section.

To benchmark domain adaptation against general capabilities, we fine-tuned Ministral-3-8B-Instruct (Mistral AI) as a state-of-the-art general-purpose baseline. The model was trained using identical hyperparameters and cross-validation splits to ensure strict comparability. All models were applied exclusively to German-language radiology reports in plain text format. No image data was processed. The input prompt, incorporating the radiology report as context, is detailed in Supplementary Fig. S2. Hyperparameters and Python packages used in the training phase are reported in Supplementary Note S1.

Evaluation

To evaluate the performance of the model, machine learning metrics like precision (P), recall (R), and F1-score were computed. Additionally, accuracy was calculated as subset accuracy, where a prediction is considered correct only if the predicted set of GOÄ codes perfectly matches the ground truth set, without any false positives or false negatives. Additionally, given that this is a multi-label classification problem with an imbalanced label distribution, micro-average scores were calculated. While macro-averaging would provide equal weight to rare codes, which can be financially important, our primary objective was to evaluate the model’s potential to reduce the manual administrative workload. This workload is driven by high-frequency standard procedures. Therefore, micro-averaging was chosen to reflect the system’s effectiveness in automating the bulk of routine documentation, aiming to free up human expert time for the verification of complex, low-frequency cases. In the context of the evaluation task at hand, recall is considered to be the most important metric, given that potential financial losses may occur when the model fails to predict GOÄ codes that were relevant for billing.

Validation set

The dataset used for evaluation was the 10% hold-out test set (see Data Partitions section). The set consisted of 50,126 radiology reports and GOÄ code pairs. The fine-tuned models were evaluated against the total hold-out test set. For this, the five models were evaluated separately on the test set by building micro average scores. Additionally, the model results were combined using an ensemble method. Only if a code was predicted by at least three models was it considered an ensemble choice and compared against the ground truth. The results of the ensemble were calculated using micro-average scores for each report separately (instance-based).

Comparison with larger LLMs

To compare the trained models with state-of-the-art LLMs, 500 samples from the hold-out test set were drawn. More information about the selection is described in the Inclusion and Exclusion Criteria section. For data privacy reasons, the radiology reports were anonymized using a de-identification pipeline developed at the investigating site [22]. The reports were also manually checked, and all sensitive information contained in the reports was removed to comply with European General Data Protection Regulation laws. The predicted codes of the models were compared against the ground truth, and the performance of the fine-tuned models was evaluated using the ensemble technique.

The results of the fine-tuned ensemble models were compared against different state-of-the-art LLMs. The models that were used for comparison were GPT-5, GPT-4.1 [23], GPT-oss, kimi-K2, deepseek-R1 [24], deepseek-V3 [25], gemini-2.5-pro [26], gemini-2.5-flash [26], Llama-3.3-70B-Instruct [27], and Qwen3-235B-A22B-FP8 [28]. A comprehensive specification of all evaluated models, including their accessibility status (open source vs proprietary), manufacturer company, parameter counts, and access methods, is provided in Table 2. Other generation parameters like temperature, top-p, maximum tokens, and number of generations per case are displayed in Supplementary Table S1. Further information about the evaluated models is also provided in Supplementary Note S2.

Table 2 Information about evaluated models

Inference on all models was performed using the OpenAI Python library (Version 1.97). During inference of larger models (GPT, Deepseek, Kimi), occasional server timeouts occurred. Therefore, a retry mechanism was implemented by automatically re-attempting the request until it was successful and a valid response was returned.

Prompt engineering

Initial zero-shot experiments (Supplementary Fig. S3) yielded poor performance, necessitating a context-rich approach. Consequently, a few-shot prompt was developed (Supplementary Fig. S4) incorporating official GOÄ code definitions structured by section and two consistent examples of high-frequency examinations (standard chest and combined chest/rib X-rays). Prompts were iteratively refined by a senior medical data scientist in consultation with a senior medical controller with more than 15 years of professional experience documenting GOÄ codes from radiology reports to ensure clinical accuracy. The final output was strictly constrained to a comma-separated list of codes without reasoning steps to ensure consistent parsing across all models.

Report types overview

The sample dataset was categorized into different categories using the described imaging protocols mentioned in the reports. In Fig. 2, the distribution of imaging protocols present in the reports is visualized.

Fig. 2

Frequency of imaging techniques described in the radiology report samples. a Real-world sample—report category frequency. b Real-world sample—distribution percentage. c Cleaned sample—report category frequency. d Cleaned sample—distribution percentage. The plots show the distribution of imaging procedures for the sample evaluation data

The most frequent imaging procedure was X-ray, followed by CT. For the real-world samples, MRI was the third most frequent procedure. Because the documentation of GOÄ codes for MRI reports often contains surcharge-based codes, most samples were filtered out for the second comparison.

Statistical analysis

All variables and metrics in the manuscript were reported using mean and standard deviation (SD) if normally distributed and otherwise using median and interquartile range (IQR). The normality assumption was checked using the Shapiro–Wilk test [29]. In addition, 95% confidence intervals (95% CI) were reported based on cross-validation model performance metrics using 10.000 iterations of bootstrapping [30]. Performance metrics between the fine-tuned and best-performing zero-shot LLM were compared using a paired t-test [31]. All statistical tests were conducted using the Python package sklearn [32] (Version 1.7.0) and scipy [33] (Version 1.15.3).