We train and evaluate the proposed method on the EndoScapes-CVS201 benchmark, a public dataset specifically designed for automated CVS recognition in laparoscopic cholecystectomy [11]. The dataset comprises endoscopic video frames collected across multiple institutions, encompassing diverse patient anatomies, camera viewpoints, lighting conditions, surgical styles, and device configurations. Such heterogeneity poses a significant challenge for generalizable CVS recognition.

All experiments strictly adhere to the official dataset split and evaluation protocol defined in the benchmark [11]. The original training, validation, and test partitions are preserved without alteration to avoid data leakage and to ensure fair comparison with prior work. No auxiliary annotations or supervisory signals from other EndoScapes subsets (including bounding box or segmentation annotations) are utilized at any stage of training or inference.

EndoScapes-CVS201 provides monocular RGB frames without depth information. To incorporate geometric cues, we generate per-frame metric depth maps using Depth Anything V3 [25], a state-of-the-art monocular depth estimation model trained on large-scale real and synthetic data. The predicted depth maps are spatially aligned with the corresponding RGB frames and normalized to a consistent scale.

Each RGB frame is concatenated with its corresponding depth map to form a four-channel RGB-D representation. Prior to concatenation, all frames undergo a center square crop to remove black borders introduced by varying laparoscopic camera aspect ratios, followed by resizing to a fixed spatial resolution. This preprocessing step reduces background bias and ensures consistent spatial alignment across cases.

The training process consists of two sequential stages designed to decouple representation learning from task-specific supervision.

Stage I: Unsupervised Foundation Model Adaptation

We initialize the visual foundation branch using DINOv2 with a Vision Transformer backbone (ViT-B), configured with a patch size of \(16 \times 16\), an input resolution of \(224 \times 224\), and an embedding dimension of 768, loaded from publicly available pretrained weights.

To extend the pretrained patch projection to four-channel RGB-D input, we apply a weight inflation strategy. The original 3-channel convolutional weight tensor \(W \in \mathbb ^\) is extended to \(W' \in \mathbb ^\) by initializing the fourth-channel slice as the channel-wise mean of the RGB slices: \(W'[:, 3,:,:] = \text (W, \dim =1)\). This preserves the pretrained representation at initialization.

Stage I self-distillation is conducted on all RGB-D frames from the EndoScapes-CVS201 training split (\(\sim \)18,000 frames) using the DINO student–teacher objective, without CVS annotations. Training runs for 50 epochs with AdamW (\(\text =5\times 10^\), weight decay 0.05, batch size 64). To promote stable convergence on this relatively small dataset, the patch embedding and modality projection are fine-tuned for the first 10 epochs before the full ViT-B backbone is unfrozen. Depth maps are normalized per-frame to [0, 1]; RGB channels follow ImageNet normalization. Upon convergence, the DINOv2 parameters are frozen for all subsequent training stages.

Stage II: Supervised Temporal Modeling

In Stage II, the frozen DINOv2 foundation encoder, whose parameters remain fixed throughout this stage to preserve the domain-robust representations acquired during Stage I, is integrated with a supervised Swin Transformer and the proposed spatio-temporal Mamba module for end-to-end CVS recognition. The frozen foundation branch provides stable, generalizable spatial features, while the Swin Transformer and Mamba module are free to specialize toward the task-specific anatomical and temporal patterns defined by the CVS criteria.

The Swin Transformer is initialized from the Swin-B pretrained weights and configured to accept four-channel RGB-D inputs directly, following the same channel extension strategy as the foundation branch. Its output embedding dimension is set to 768 to match the patch-level embedding dimension of the DINOv2 ViT-B backbone, ensuring that the token concatenation \(\textbf_t = [\textbf_t^} \Vert \textbf_t^}]\) described in Section Swin transformer for hierarchical RGB-D spatial encoding is dimensionally consistent and that neither branch dominates the joint representation by virtue of embedding size alone.

The spatio-temporal Mamba module operates on fixed-length temporal windows of \(T4\) frames, corresponding to 4 s of surgical video at the 1 fps sampling rate used throughout all experiments. As discussed in Section Ablation study, this window size is justified empirically and clinically: 4 s is sufficient to confirm stable anatomical exposure of the hepatocystic triangle while remaining within a single coherent dissection phase. The state-space convolution kernel size is set to \(d3\), which we found provides adequate receptive field over the flattened spatio-temporal sequence without introducing unnecessary parameter overhead.

The Swin Transformer, spatio-temporal Mamba module, and classification head are optimized end-to-end using supervised CVS labels via AdamW with weight decay, while the DINOv2 backbone remains frozen. To account for the inherent subjectivity of CVS annotation, where reasonable inter-annotator disagreement exists particularly near the boundary between partial and full CVS achievement, we apply label smoothing with \(\epsilon 0.1\), which prevents the model from becoming overconfident on ambiguously labeled frames and improves calibration under the class imbalance characteristic of surgical video datasets. All remaining hyperparameters, including learning rate, weight decay coefficient, and batch size, are selected solely on the basis of validation set performance to prevent any leakage of test-set information into model selection.

To ensure a fair and consistent comparison with prior studies, we adopt the same evaluation protocol and metrics commonly used for CVS recognition. Specifically, performance is quantified using mean Average Precision (mAP) and Balanced Accuracy (BAcc).

The mAP metric evaluates the quality of probabilistic predictions by measuring the area under the precision-recall curve for each CVS criterion, thereby capturing the model’s capability to rank positive frames ahead of negative ones under varying decision thresholds. Balanced Accuracy is defined as the average of sensitivity and specificity, which is particularly suitable for CVS assessment due to the inherent class imbalance between positive and negative observations in surgical videos. Both metrics are computed independently for each of the three CVS criteria, and the final reported scores correspond to the unweighted average across all criteria [11].

We benchmark the proposed RGB-D spatio-temporal framework against a set of established baselines, including ResNet50 [26], ResNet50-MoCov2 [11, 26, 27], DeepCVS [4, 28], LG-CVS [11, 28], and SV2LSTG [28, 29]. Notably, several state-of-the-art graph-based methods leverage extended Endoscapes subsets, such as Seg201 [11], to enhance anatomical-level awareness. In contrast, our approach is trained solely on the CVS201 sub-dataset.

As summarized in Table 1, the proposed model consistently outperforms most baselines across both mAP and Balanced Accuracy, demonstrating substantial gains in capturing the clinically relevant CVS criteria. This improvement is particularly pronounced when compared to conventional RGB-only architectures and temporal models lacking foundation-model pretraining or spatio-temporal state-space modeling. Despite using a more restricted dataset, the integration of DINOv2, Swin Transformer, and the depth-aware spatio-temporal Mamba enables superior generalization and fine-grained temporal reasoning, underscoring the effectiveness of our RGB-D pipeline in endoscopic CVS recognition.

To gain deeper insight into the internal behavior of the proposed framework, we conduct a qualitative analysis based on module-level attention and activation visualizations. Specifically, we examine the attention patterns of the DINOv2-based foundation branch, the supervised Swin Transformer branch, and the temporal activation dynamics of the spatio-temporal state-space module. Representative visualizations are shown in Fig. 2.

We first visualize the self-attention maps produced by the DINOv2-based foundation encoder. The resulting heatmaps exhibit broad yet structured attention distributions across the scene. In particular, the model attends to regions corresponding to typical instrument locations, high-contrast anatomical boundaries, and depth-discontinuous edges introduced by the RGB-D representation. These attention patterns suggest that the foundation model captures globally informative semantic and geometric cues without being overly localized, reflecting its role in learning general-purpose and transferable representations.

In contrast, attention maps from the supervised Swin Transformer branch are markedly more concentrated. The Swin Transformer focuses predominantly on the hepatocystic (hepatic) triangle region and dynamically shifts its attention across frames toward specific anatomical structures relevant to CVS, such as the cystic duct and cystic artery. This behavior indicates that supervised fine-tuning refines the broad foundation features into task-specific spatial representations aligned with clinical criteria.

To analyze temporal reasoning, we further visualize the mean activation of the Spatio-temporal mamba module across a four-frame temporal window. The activation scores vary over time, reflecting the model’s sensitivity to progressive anatomical exposure. Notably, the highest activation consistently occurs when the hepatocystic triangle is fully exposed under an optimal viewing angle, corresponding to clinically valid CVS conditions. Frames with partial exposure or transient occlusions yield substantially lower activation, demonstrating the model’s ability to suppress premature CVS predictions.

Qualitative visualization of module-level attention and temporal activation in the proposed framework. (Row 1) Continuous 4 input frames. (Row 2) Attention maps from the DINOv2-based foundation branch, showing broadly distributed yet structured responses over instrument regions, anatomical boundaries, and depth-discontinuous edges, reflecting globally informative semantic-geometric representation learning. (Row 3) Attention maps from the supervised Swin Transformer branch, exhibiting highly localized focus on the hepatocystic triangle and dynamically shifting across frames to anatomically relevant structures associated with the Critical View of Safety (CVS). (Row 4) Mean temporal activation (y-axis) of the spatio-temporal mamba module over a four-frame window (x-axis), where higher activation corresponds to frames with complete exposure of the hepatocystic triangle under optimal viewing conditions, indicating effective temporal aggregation for CVS identification

To systematically quantify the contribution of individual components, we perform an ablation study on the Endoscapes-CVS201 dataset and further evaluate cross-dataset generalization on the independent MT-CVS dataset [12]. The experiments examine the impact of RGB-D input, foundation-model pretraining, spatio-temporal Mamba, and alternative temporal modeling architectures.

We compare the full RGB-D pipeline against a baseline using only RGB frames. Incorporating depth significantly improves both mean Average Precision and Balanced Accuracy across all CVS criteria, demonstrating the value of geometric cues for disambiguating occlusions, tissue separation, and instrument interference.

To assess the contribution of self-supervised foundation representations, we remove the DINOv2 encoder while keeping all other supervised components unchanged. The performance consistently drops, confirming that DINOv2 embeddings provide robust, semantically rich representations that enhance generalization.

We evaluate the proposed depth-aware spatio-temporal Mamba against a standard 4-frame Mamba [9] and the attention-based TimeSFormer [16]. The Spatio-temporal Mamba achieves superior mAP and Balanced Accuracy, illustrating the advantages of linear-complexity state-space modeling with selective spatial-temporal gating for capturing progressive anatomical changes relevant to CVS formation.

To validate robustness, we perform inference on the MT-CVS dataset using models trained exclusively on Endoscapes-CVS201. Models incorporating both the DINOv2 foundation and depth-aware Spatio-temporal Mamba exhibit the highest transfer performance, highlighting that foundation-model pretraining and spatio-temporal state-space reasoning enhance adaptability to unseen clinical data distributions.

Component ablation results are summarized in Table 2. The combination of RGB-D input, DINOv2 backbone, and spatio-temporal Mamba consistently achieves the best performance across in-domain and cross-dataset evaluations, confirming that each component contributes meaningfully to accurate and generalizable CVS recognition.

Finally, to justify the choice of temporal window size, we ablate \(T \in \\) under a fixed video sampling rate of 1 fps, such that \(T1\), \(T2\), \(T4\), and \(T8\) correspond to 1, 2, 4, and 8 seconds of surgical context respectively. At \(T1\), the spatio-temporal Mamba module receives tokens from a single frame only, reducing it to a frame-level classifier with no temporal aggregation. Performance improves consistently as T increases from 1 to 4, confirming that temporal context is essential for reliable CVS recognition. At \(T8\), performance plateaus and marginally declines despite the larger temporal context. This behavior is consistent with the clinical structure of CVS formation: at 1 fps sampling, a 4-second window captures sufficient evidence to confirm whether the hepatocystic triangle is stably and completely exposed, while an 8-second window routinely spans qualitatively distinct dissection sub-phases — instrument exchanges, irrigation pauses, and retraction adjustments — whose inclusion introduces cross-phase ambiguity that reduces the selectivity of the depth-conditioned gating mechanism. The window size \(T4\) is therefore the principled choice that aligns temporal context with the clinical rhythm of CVS confirmation. These results are reported in Table 3.