Computational pathology (CPath) (Song et al., 2023), which utilizes whole-slide images (WSIs) stained with hematoxylin and eosin (H&E) for diagnosis, is an emerging paradigm in medical research. Artificial intelligence (AI)-based pathology diagnostic models not only reduce the workload of pathologists in clinical practice (Greenwald et al., 2022), but also offer valuable insights for researchers (Holzinger et al., 2017). Furthermore, an increasing number of foundational models (Chen et al., 2024, Vorontsov et al., 2024) demonstrate excellent feature extraction capabilities from pathological images, enabling downstream tasks such as tissue segmentation (Chan et al., 2019), subtype classification (Couture et al., 2018), staging (Skrede et al., 2020), grading (Huo et al., 2024), and pan-cancer analysis (Chen et al., 2022, Wu et al., 2024). However, current models occasionally misclassify unknown data as known content, causing a trust crisis for patients and doctors. Consequently, addressing and correcting AI’s misjudgments is essential for advancing research in medical image analysis.

Socrates stated, “Awareness of ignorance is the beginning of wisdom”. However, under the closed-set assumption of deep learning theory, models are compelled to predict and interpret unfamiliar data, which can cause intelligent systems to appear unwise. To address this, a filtering mechanism based on out-of-distribution (OOD) detection theory can be established to identify ambiguous or unknown samples, enabling the model to reject uncertain predictions. These theories are vital for safety-critical applications, particularly in medical image analysis (Linmans et al., 2023). The corresponding OOD detection approaches are broadly categorized into post-hoc inference methods and training methods (Zhang et al., 2023). Post-hoc inference methods do not intervene in the model training process but operate solely during model inference (Hendrycks and Gimpel, 2017, Liang et al., 2017). In contrast, training methods integrate auxiliary OOD training data (Yu and Aizawa, 2019) and introduce some novel loss functions (Hendrycks et al., 2018) to improve OOD detection during model training. In the paper, to ensure that the original performance of the pathology model is not altered and the existing specific tasks are not affected, we explore adding a new module to the existing models for OOD detection.

However, current post-hoc inference methods for OOD detection are primarily designed for natural image tasks. They often fail or do not meet the requirements in pathological image OOD detection tasks. Table 1 presents an overview of representative OOD detection theories. “Taxonomy” refers to which part of the model’s output is used to compute the OOD score. “Training Data-Free” indicates whether the inference process depends on the model’s training data. “Grad. Info.” specifies whether gradient information is utilized during OOD detection. Preventing the leakage of training data and model weight information is a crucial requirement in medical image analysis. Therefore, once a model is trained, access to its training data and model weight information should be restricted. Moreover, we show the distribution plots of OOD scores for several OOD detection methods that meet medical safety standards in Fig. 1. However, these methods do not effectively distinguish between ID and OOD samples in the context of pathological OOD detection. Therefore, it is imperative to design a novel OOD detection method specifically tailored for pathological images.

To address this issue, we propose a plug-and-play OOD detection approach specifically tailored for digital pathology images, termed Stability Distance (StaDis). When faced with unseen data, models often make erratic predictions, generating chaotic features (Lee and AlRegib, 2020). Consequently, StaDis requires calculating the distance between a sample and its own perturbation. As a result, compared to OOD data, models with prior knowledge of in-distribution (ID) samples exhibit a lower stability distance to perturbations in ID data, as shown in Fig. 2. In traditional distance-based OOD detection theory, obtaining the OOD score focuses on calculating the distance between a sample and the prior knowledge derived from the training data. In contrast, our approach preserves the privacy of digital pathology data, without relying on training data or gradient information. In addition to detecting patch-level OOD data, we also analyze slide-level tasks within the multi-instance learning framework. To thoroughly assess the pathological OOD detection capabilities, we design four OOD detection benchmarks and utilize ten backbones to closely simulate the OOD data encountered by computational pathology models in real-world applications. The experimental results indicate that our method significantly outperforms the compared approaches, demonstrating that StaDis is an effective method for pathological OOD detection at the current stage. The key contributions of the paper are as follows:

We propose a novel OOD detection approach, StaDis, to compute the distance between a sample and its perturbations. StaDis is specifically designed for CPath tasks to distinguish between in-distribution and out-of-distribution samples.

We develop a slide-level OOD detection framework based on multiple instance learning (MIL), an area that, to our knowledge, has been rarely explored. Our framework allows the integration of any OOD detection module into an MIL model.

We design a comprehensive set of pathological OOD detection benchmarks, including rare case detection, anomaly tissue detection, and frozen section (FS) detection scenarios.

We conduct experimental analyses on these pathological OOD detection benchmarks and different models. The results demonstrate that our method achieves state-of-the-art performance across these benchmarks.