Datasets

This IRB-approved retrospective study utilized two distinct datasets: An in-house dataset from the University Hospital Erlangen, Germany (called the in-house dataset) and a subset of the ACRIN-6698 dataset (called ACRIN-6698) (Fig. 1).

Fig. 1

Flowchart illustrating the selection and exclusion criteria for the in-house (a) and ACRIN-6698 (b) datasets. From an initial 1610 in-house breast MRI examinations, 138 cases with volumetric segmentations were selected for the performance evaluation dataset, while the remaining 1472 cases formed the basis of the evaluation of the impact on human reader lesion segmentation subset. After excluding cases with BI-RADS < 3, a random selection of 20 cases from the remaining 656 was included in the final analysis. For the ACRIN-6698 dataset, 865 cases were initially available. After filtering for pre-treatment examinations and limiting to Siemens-manufactured data, 36 cases remained. From these, the seven cases with the highest DWI misalignment and 13 additional random cases were selected, yielding a total of 20 cases. The final lesion segmentation dataset consisted of 40 cases from both datasets combined

The in-house dataset contains clinically indicated diagnostic routine breast MRI examinations performed at the Institute of Radiology of the University Hospital Erlangen, excluding patients with breast implants, comprising n = 1610 MRI examinations performed between 2015 and 2020, with a mean patient age of 50 years. (± 13 years). MRI scans were performed in prone position on both 1.5 Tesla and 3 Tesla systems (Magnetom Aera 1.5T and Skyra Fit 3T; Siemens Healthineers, Erlangen, Germany) using a dedicated 18-channel breast coil (Siemens Healthineers). This dataset included a multiparametric MRI protocol with pre- and postcontrast T1-weighted sequences (T1w), T2-weighted sequences, and multi-b-value DWI sequences, acquired with a single-shot echo-planar imaging (EPI) sequence, in axial orientation with b-values of 50 (b50), 750 (b750), and 1500 (b1500) s/mm2 (detailed acquisition parameters for DWI are provided in Supplemental Table 1). For DWI vendor specific distortion correction steps have been applied during image reconstruction. Histopathology results served as the ground truth and were retrieved from the hospital information system.

Apart from the prior work on co-registration of ADC, published as part of the 11th International Workshop, WBIR 2024 conference proceedings [11], the data were previously included in different studies on virtual breast image generation [13,14,15,16,17], detection and forecasting of artifacts [10, 18,19,20], incidental finding [21] and skin pathology imaging [22].

The second dataset consisted of the ACRIN-6698 dataset, which included n = 865 cases with invasive breast cancer [23]. Details on the dataset, including the underlying study and MRI protocols, can be found in the respective reference. For comparability with our in-house data, we filtered it to retain only MRI examinations performed with systems from vendor Siemens Healthineers and which were acquired prior to adjuvant chemotherapy. This resulted in n = 36 breast MRI examinations. The MRI systems used included 1.5 Tesla and 3 Tesla Siemens models (Magnetom Avanto 1.5T, TrioTim 3T, and Prisma fit 3T; Siemens Healthineers, Erlangen, Germany) with sequences including pre-contrast T1w, post-contrast T1 subtraction series, and DWI, acquired with a single-shot EPI sequence, with b-values of 0 (b0), 100 (b100), 600 (b600), and 800 (b800) s/mm2.

All data were transferred to dedicated research workstations. The goal of the first part of the study was to evaluate the best-performing co-registration methods, based on a subset of the in-house dataset. The sub-dataset with volumetric segmentations of lesions on b1500 images of n = 138 (mean age: 53.2 ± 11.7 years) cases comprised n = 68 benign and n = 70 malignant lesions. For the n = 138 cases, only the largest lesion in each dataset was segmented in 3D on b1500 images using the integrated draw-function of Slicer3D (version 4.11.20210226) [24]. In four of the malignant and one of the benign cases, two lesions were annotated per patient, as there were two major lesions visible. In cases with necrosis, only viable tumor tissue was segmented, and necrotic areas were excluded to ensure consistent representation of intralesional tissue. The segmentations were performed by a medical student (J.E.) with two years of experience in breast DWI under the supervision of a board-certified radiologist (S.B.) with more than 10 years of experience in breast DWI. In the following, this dataset will be called performance evaluation dataset.

For the second part of this study, the evaluation of the impact on human reader lesion segmentation analysis, two datasets were used. The first dataset was derived from the in-house dataset, excluding cases used in the performance evaluation, leaving n = 1469 available datasets. From these, n = 20 (mean age: 49 years) cases were selected randomly, but ensuring an equal number of malignant (n = 10) and benign (n = 10) cases. The second dataset was derived from the ACRIN-6698 study. For the n = 36 Siemens pre-therapy examinations, a visual reading was performed to score the overall image alignment between b0 and b800 DWI sequences, focusing on the alignment of breast tissue and lesions. One reader (L.B.), with one year of experience in breast DWI, evaluated and scored the images using a 5-point Likert scale (1: Very Poor Alignment, 5: Perfect Alignment). All cases received a score of either 4 or 5, indicating at least good alignment. From these, the seven datasets scored as 4 were selected, along with 13 randomly chosen cases, resulting in a total of 20 datasets (mean age: 47 years). The resulting 20 datasets from the ACRIN-6698 study composed of 11 cases from a 1.5T scanner and 9 cases from a 3T scanner. For the evaluation of the impact on human reader lesion segmentation, both datasets were combined, forming a lesion segmentation dataset of 40 cases.

Experimental setup overview

Figure 2 shows the experimental workflow of the study performed in order to evaluate the co-registration of the different b-values prior to ADC map calculation. First, our initial n = 16 different co-registration approaches, which are explained in detail below in Sect. "Co-registration methods", were applied to the DWIs of the performance evaluation dataset, and ADC maps were generated from the co-registered and the non-registered DWIs.

Fig. 2

Experimental workflow to assess whether inter-b-value co-registration prior to ADC map creation improves accuracy of ADC and clinical analysis. With the performance evaluation, which is divided into stability analysis and diagnostic analysis, the best-performing methods were selected in comparison to no co-registration based on different comparison criteria and then further validated using the lesion segmentation dataset in an evaluation of the impact on human reader lesion segmentation

A stability analysis of the ADC mean values inside the 2D lesion segmentations using Coefficient of Variation (CoV) and outlier detection was performed to ensure that co-registration does not strongly alter the ADC mean values of lesions and thereby reduce the stability of ADC maps (see Sect. "Stability analysis"). From the 16 initial co-registration methods, we selected the so-called ‘stable’ methods by comparing their performance to that of no co-registration. The exact criteria that were used for selecting the methods after each evaluation step are further described in the following sections. The classification methods that passed the stability analysis as better or equally performing in comparison with no-coreg were further assessed in a diagnostic performance analysis, focusing on accuracy, sensitivity, and specificity in distinguishing benign from malignant lesions (see Sect. "Diagnostic performance analysis"). Evaluating the stability analysis, followed by the diagnostic analysis on the 16 co-registration methods, the best-performing methods were selected.

In the second part of the study, ADC maps generated by the best-performing coreg methods were tested against no-coreg ADC maps by evaluating the impact on human reader lesion segmentation using the lesion segmentation dataset.

Co-registration methods

The aim of using different co-registration methods is to identify the optimal alignment order and transformation type (rigid or non-rigid) for DWI images, in order to generate the most accurate ADC maps. Accurate ADC maps are essential for differentiating between benign and malignant lesions.

The co-registration approaches are summarized in Table 1. Similar to the WBIR study [11], co-registration methods were based on the ANTs library [12], with the difference of adding b750 as a possible target for co-registration in this study, resulting in n = 16 co-registrations using different co-registration types from the ANTs library, with b50, b750, or b1500 images as co-registration targets and applied iteratively or non-iteratively.

Table 1 Overview over 16 different co-registration methods with the co-registration type, target and whether it is an iterative or non-iterative method

Both rigid and non-rigid methods were used, with non-rigid registration addressing potential three-dimensional breast deformation during imaging and distortion artifacts, while rigid registration provided robustness since it preserves the overall shape and volume of structures, avoiding potential unrealistic deformations that can occur with non-rigid transformations.

Across all targets, four types of transformations from the ANTs library were applied: Rigid (rotation and translation only), Similarity (scaling, rotation, and translation), Affine (rigid registration with additional scaling), and SyN (symmetric normalization with affine and deformable transformations)[25]. We focused on the type of transformations for the evaluation and kept the default cost function from the registration function in the ANTs library, which is mutual information.

When b1500 images were the target, co-registration was additionally performed iteratively for each co-registration type, first aligning b750 images to b1500 images, creating an “aligned b750”, then b50 images to the aligned b750 images. For all other targets, co-registration was performed only non-iteratively.

Since we evaluated the co-registration methods based on the ADC mean values inside the lesions, it was necessary to register also the lesion segmentations to the target, when b1500 images were not the target because the segmentations were originally performed on b1500 images (see 2.1 Dataset). In these cases, the b1500 image was first co-registered to the target, and the segmentation was aligned to the target using the same transformation as the b1500 image. In comparison to the WBIR study, the segmentation was then binarized at a 0.3 threshold to maintain consistency with pre-co-registration masks.

A baseline method, no-coreg, was included for comparison to evaluate potential improvements. In an additional analysis, we evaluated the influence of masking during co-registration. This analysis has been moved to the supplementary material to keep the main text concise (Supplemental Chapter 1, Supplemental Fig. 2, Supplemental Table 2).

Table 2 Patient and lesion characteristics for benign and malignant cases

Recent literature indicates that using only two b-values for ADC calculations maintains comparable predictive value and repeatability to using four b-values [7, 26]. Hence, our study focused on two-b-value based ADC calculations, computed voxel-wise using Eq. (1) where S(b1) and S(b2) represent the signal intensities at two different b-values, one close to 0 and one close to 750 s/mm2 [27]. Choosing these specific b-values is based on a study of Pereira et al., who found slight improvements using these b-values [28]. For in-house data, the closest values to this are b50 and b750, and for ACRIN-6698 data, they are b0 and b800.

$$\text= \frac\left(\frac_\right)}_\right)}\right)}_- _}$$

(1)

Voxel-wise ADC maps were generated for the entire breast. In the first part, where the best co-registration method is evaluated on the performance evaluation dataset, mean lesion ADC was obtained by averaging all voxel values within the volumetric tumor segmentation. In the second part, focused on evaluating the impact on human reader lesion segmentations on the lesion segmentation dataset, mean ADC was derived analogously but restricted to the 2D segmentation on the slice with the largest lesion diameter.

Best co-registration methods analysis

In the first part of the study, the goal was to identify from the n = 16 co-registration methods presented in Sect. "Co-registration methods" the best performing co-registration methods. This analysis was done on the performance evaluation dataset and was performed in an iterative way as described in Sect. "Experimental setup overview". First, the stable methods were filtered by the stability analysis. Afterwards, the stable methods were evaluated based on their diagnostic performance analysis. After each analysis, the methods that performed worst in the analysis according to the criteria presented in the sub-sections were discarded from further analysis, resulting at the end in the best performing methods for part two of the study.

Stability analysis

We aimed to investigate the stability of the co-registration and ensure that its application does not introduce additional variation in the resulting mean ADC values of the volumetric lesion segmentations. To do this, we analyzed the number of outliers in benign and malignant lesions. Outliers were identified as cases in which the mean value of the ADC in the lesion exceeded 1.5 times the interquartile range (IQR) [29]. Co-registration methods for which the number of outliers exceeded the number of outliers in the non-co-registered data were not considered for further analysis.

Additionally, within-lesion CoVs and Mean were analyzed for benign and malignant lesions. Statistical significance compared to no-coreg was assessed using repeated measures ANOVA with Dunnett's post hoc analysis [30], considering p-values < 0.05 as significant. The t-values from the ANOVA with Dunnett’s post hoc test indicate whether the CoV in each method differs significantly from the no-coreg condition. Methods with non-significant or lower t-values were considered to have stable or reduced variability. In particular, methods with t-values close to zero or less (indicating no increase or a decrease in CoV compared to no-coreg) were retained, while those with large positive t-values (indicating increased CoV) were filtered out.

Diagnostic performance analysis

The diagnostic analysis focused on the clinical impact of co-registration on distinguishing benign from malignant lesions. Performance metrics—including area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, and accuracy—were calculated for each method using receiver operating characteristic (ROC) curve analysis and on individually calculated optimal cutoff based on the Youden index (called individual threshold in the results section) [31]. Additionally, the previously published optimal ADC cutoff (1.53 × 10⁻3 mm2/s) and conservative cutoff (1.68 × 10⁻3 mm2/s) from the A6702 trial were used to calculate diagnostic metrics. The optimal cutoff was originally identified to achieve the best trade-off between sensitivity and biopsy reduction, with values below this threshold indicating a higher likelihood of malignancy. An alternate conservative cutoff was proposed for fewer false negatives, though at the expense of a smaller reduction in biopsy rates [2, 6]. From here on these two thresholds of the A6702 trial will be referred to as A6702 trial thresholds. Statistical significance was assessed for the cutoff’s predictive accuracy of the co-registration methods compared to the no-coreg method using McNemar's test [32].

The three methods with the highest AUROC and the best accuracy were selected for the second part of the study.

Evaluation of the impact of co-registration on human reader lesion segmentation

In the second part of the study, the previously selected best-performing co-registration methods were further analyzed to evaluate the impact of the co-registration on the segmentations a human reader does on the ADC maps.

The lesion segmentation dataset, consisting of n = 40 (30 malignant, 10 benign) cases, was used for this evaluation. The ADC maps of the three best-performing co-registration methods that were selected based on the stability analysis (Sect. "Stability analysis") and the diagnostic performance analysis (Sect. "Diagnostic performance analysis")—and no-coreg ADC maps—were used for this purpose. To maintain consistency in naming, methods applied to the ACRIN-6698 dataset followed the same b-value nomenclature as the in-house dataset. While ACRIN-6698 data used b0 and b800 instead of b50 and b750, we retained the original naming convention (e.g., b750-registration type) to ensure clarity. The resulting ADC maps, along with their respective b-value images and contrast enhanced T1-weighted subtraction images at the second time point (T1w-sub2), were randomly assigned into four reading folders. For in-house data, the BI-RADS scores and radiologist reports were added, and for the ACRIN-6698 dataset, it was known to the readers that all lesions were malignant. Three independent readers (D.S., three years of experience in DWI Breast MRI; C.E., two years of experience in DWI Breast MRI; D.H., 15 years of experience in DWI Breast MRI) reviewed these folders, with a two-week interval between readings to minimize recall bias. Blinded to the ADC map generation process, they segmented the main lesion at the slice of its largest diameter using Slicer3D (version 5.6.2) [24]. To find the lesion with the largest diameter, the comments from the radiological report were taken into account for the in-house data and the lesion was first identified in T1w-sub2 and b750/b800 images, to afterwards use the b750/b800 images as an overlay over the ADC map, and to then segment on the ADC map using the integrated draw-function of Slicer3D. With regard to intralesional heterogeneity, readers were explicitly instructed to exclude necrotic components from their segmentations. In cases of differing segmentation targets chosen by the readers, the most experienced reader (D.H.) determined the correct lesion, and segmentations were adapted to this lesion. The analysis assessed segmentation accuracy and ADC values to evaluate co-registration performance.

ADC map segmentation evaluation

Segmentation accuracy on the ADC maps was evaluated using several metrics. Bland–Altman analysis compared 2D lesion segmentation areas between non-co-registered and co-registered ADC maps, plotting the mean area difference against the overall mean area across readers. Spatial consistency was evaluated by calculating the mean distance between segmentations from no-coreg and co-registered ADCs, differences in lesion positioning were evaluated using the Friedman test, assuming a p value < 0.05 to be significant [33].

Reader agreement was evaluated to determine whether image registration influenced the consistency of lesion segmentations across readers. Agreement was quantified using the Intraclass Correlation Coefficient (ICC), applied separately to distances between reader segmentations and lesion area measurements. ICC3k was applied, as it assesses absolute agreement in a two-way mixed-effects model, appropriate when the set of raters is fixed and the focus is on the consistency of their ratings across all subjects [34]. While overlap-based measures such as the Dice coefficient are commonly used for segmentation evaluation, we did not apply Dice in this context because co-registration can introduce slight in-plane or through-plane shifts of the lesion. These small spatial displacements can substantially reduce Dice scores despite segmentations being clinically equivalent, potentially resulting in misleading conclusions. Therefore, our evaluation focuses on lesion segmentation position and size, as these metrics more directly capture the impact of co-registration on lesion extent and displacement.

Additionally, pairwise comparisons of the distances between lesion segmentations of different methods were performed to evaluate the impact of registration on reader agreement. For each reader pair, the Wilcoxon signed-rank test was applied to compare the distances between the reader segmentations of the reference method (no-coreg) against each registration method, with a p value < 0.05 meaning statistically significant. The same was done for the relation of the segmentation areas between the reader pairs for the different methods.

Quantitative ADC values analysis

Firstly, we tested for statistical significance between the ADC mean values in the 2D lesion segmentation of no-coreg and the co-registration methods for benign and malignant lesions separately using repeated measures ANOVA with Dunnett's post hoc analysis, considering p values < 0.05 as significant. Afterwards, for each method, the diagnostic performance was analyzed, individually for each reader and method, using the AUROC and a DeLong’s test for statistical significance evaluation. Additionally, AUROC was created for the averaged ADC means inside the 2D lesion segmentations across the three readers for each method. For the averaged ADC means, sensitivity, specificity, and accuracy were evaluated at three distinct cutoffs similar to the diagnostic performance evaluation from above: the optimal individual cutoff determined for each method, the A6702 trial optimal and conservative cutoff. The statistical significance of the differences in the predictive accuracy between the methods and no-coreg at the different cutoffs was calculated using McNemar’s test, with p < 0.05 meaning statistical significance.