To benchmark visual odometry and motion detection, we use the DynaSCARED test split, which includes 100 sequences (300 frames each) across 8 scenarios. To demonstrate our model’s generalization, we perform extensive experiments on StereoMIS [4], a robotic-assisted MIS dataset with realistic scene dynamics and labeled endoscope poses. We also assess DynaSCARED’s effectiveness in training our model by comparing performance when trained on: 1) a synthetic motion detection dataset from [3] (SynMotion), and 2) the StereoMIS training split from [4]. We report relative position error (RPE) and absolute trajectory error (ATE) to evaluate pose estimation accuracy. Additionally, we report Intersection over union (IoU) on the inferred motion probability M (with ground truth motion in DynaSCARED) to quantitatively assess motion detection, aiding in pose estimation diagnostics. For qualitative analysis, we visualize the learned weight map W, which complements M.

We train our model on DynaSCARED (training & validation contains 4000/500 sequences with 30 frames each) with a batch size of 8. We use the Adam optimizer with a learning rate of \(1 \times 10^\). with a warm up-decay strategy as in [3]. All our ablated models are trained on a single 32GB NVIDIA RTX V100 GPU with 7000k iterations. We augment the dataset with color jittering and blurring, coupled with the frame skipping strategy of [3] during training. For the differentiable pose optimizer, we set 30 iterations as the optimization stopping limitation across all experiments.

In this section, we evaluate our method on the test split of DynaSCARED. In addition, we ablate critical modifications in our method, and conduct an experiment to justify the importance of adopting end-to-end training. In addition to RPE and ATE, we analyze the improved pose estimations performance with the assisted motion detection metric IoU. The scenes are categorized based on the motion status of the scene and camera.

We summarize our ablations as follows: 1) Compared to Naive, where the inferenced weight map in Ours is replaced with a constant weight map of 1, Ours(end-to-end pose model) outperforms across all dynamic scenarios, both with a fixed camera (S1) and a challenging moving camera (S0). This is due to effective detection of scene dynamics, including rigid tool movement and non-rigid tissue deformation, as reflected by high IoU values. 2) We show that using the 3D alignment (Ours(res.3D)) error instead of a 2D reprojection error (Ours) significantly degrades performance, which justifies our design for target residual as referred in Sec. 3.2.2. 3) Comparing Ours(rf.W) to Ours, we see improved performance with our proposed test time refinement strategy as detailed in Sec. 3.2.2, while the vanilla refinement Ours(rf.N) shows worse performance compared to Ours, reinforcing the value of our learned weights.

In this section, we compare our method with five baseline methods and one OCLR−extended [3] method, including rigid SLAM methods ORBSLAM3 [1] and ElasticFusion [2] following the choices in recent work [4]; and non-rigid methods Hayoz [4] and additional one extended upon OCLR [3], which are most close to ours in methodology. We further compared our method with endoscope tracking method EDaM [35] and a learning based SLAM method DroidSLAM [13]. To be specific, we extend the motion detection method OCLR for pose estimation purposes, by replacing the adopted learned weight map in weighted pose optimization, with inferred motion probability.

Notice the endoscope in StereoMIS is often still, to sufficiently benchmark methods, from which we extract short snippets (26 frames each), captured when camera is moving; we group the obtained snippets to liver/bowel scene (5174/3796 frames) and report the evaluation results in Table 2. For completeness, we also experiment on still-camera snippets as reported in Table 3, where the snippets are grouped into scenario without presence of surgical tools (containing breathing and pulsation) and scenario involving severe tool-tissue interaction. The former scenario is further grouped to liver/bowel scene(2600/1326 frames), while we observe the second only captures bowel scene therefore we grouped based on their source sequence P2_6/P2_7 1066/1248 frames). we refer Sec. C.1 in supplements for detailed strategy in obtaining the snippets.

For evaluations conducted on snippets captured with moving camera as reported in Table 2. , our model outperforms Hayoz et al. [4] significantly in both RPE and ATE when camera moves as shown in Table 2. This is likely due to the well simulated and distributed motion signals in our training data (DynaSCARED), and our transformer-based architecture, adapted from the state-of-the-art motion detection network OCLR [3], which captures scene ambiguities better than their CNN. As shown in Fig. 4, Hayoz’s model can fail in properly attenuating colon deformation (row C) and is overly sensitive to motion signals (large areas around the moving tool are attenuated in rows D-F), potentially wasting information. Compared with the visual odometry method extended upon OCLR [3], our model shows improved detection of non-rigid tissue deformation (e.g., row A C, F in Fig. 4). This is likely because OCLR’s training data are less surgery-specific and designed for amodal motion segmentation, a binary task, unlike our focus on pose estimation; on top of that, the end-to-end training with pose supervision potentially also contribute toward the target pose estimation task, as demonstrated with the deteriorated performance of Ours (motion) in Table 2, Table 1. Notice our method also outperforms EDaM [35], an visual odometry approach proposed for endoscopic videos, it can handle tissue deformation to an extent, while it assumes slow tissue deformations. We also put our methods in comparison with SLAM methods, our method generally performs better due to its ability to handle scene dynamics, though it, like other VO baselines, underperforms in ATE due to the lack of global optimization. For evaluation conducted on snippets captured with static camera (involving extensive tissue breathing, pulsation, and tool-tissue interactions), as reported in Table 3, our method achieves overall best performance among prior-free baselines, while ranks second to Hayoz et al.  [4]. We emphasize that their method relies on continuous surgical tool masking to enhance performance. This dependency is demonstrated through an ablation study in tool-tissue interaction scenarios: when evaluated without tool mask preprocessing (denoted as metrics within brackets for Hayoz et al). exhibits significant performance degradation. In contrast, our framework maintains robustness without requiring explicit tool segmentation, establishing it as the most effective prior-free solution for dynamic surgical environments. Whereas as we have argued, consistently masking objects out based on semantic priors regardless of their motion status, can lead to potential information wasting for accurate localization and has limitations in generalization across unseen tool categories. On the other hand, as expected, Hayoz et al. [4] shows competitive results on static camera scenario, this is because the method has a tendency of over-sensitivity to motion signals (shown in row F in Fig. 4, more severe in row E), thereafter “prefers” the still camera snippets (the pose optimization is initialized with identity transformation). The results on full sequence align with our conclusion, as shown with the representative trajectory in Fig. 3, and the reported metrics in Table 2 of supplementary material.

Trajectories on full StereoMIS sequences. In sequence P2_2 (up), there are extreme scene dynamics (endoscope out of trocar in frame 805, quickly reentering), leading briefly to no anchoring for relative pose estimation, and consequently tracking failure until end. Our method shows strong relative pose estimation, indicated by 1) a similar trajectory shape to that of ground truth 2) the reported mean RPE metrics. In sequence P2_4 (down), our method is the only one which consistently tracks

Visualization of learned weight map. Row A & B represent example results on DynaSCARED. Other results are on StereoMIS. Values in the weight maps are on a color scale from blue (small) to yellow (large). We mark regions containing non-ideal/reasonable inference with red/cyan boxes. Red arrows indicate over-sensitivity to motion signals. We refer Fig. 1 and the video link in the supplementary for more examples and enhanced visualization

Finally, we validate the effectiveness of DynaSCARED by training our model (pose model) and motion model with two other datasets [3, 4] as shown in Table 4. Additionally, to explore the effect of chosen technique for tissue deformation simulation in generation of DynaSCARED, we also compare our choice of thin plate spline technique with alternatives, namely, B-spline based free form deformation and elastic distortion algorithm, and generated two variants of DynaSCARED. We again train our models with the referred variants and report results in Table 4. The evaluation is conducted on StereoMIS snippets containing camera motion.

Compared with models trained with other datasets, as reported in Table 4, our motion model Ours(motion) outperforms the motion model trained on SynMotion [3], Ours(m.SYN). Our pose model, surprisingly, significantly outperforms a pose model trained on real surgical data StereoMIS Ours(p.SM). The above demonstrates the effectiveness of our exquisitely designed data modeling strategy toward well-distributed, surgery-specific semi-synthetic data as described in Sec. 3.3; and serves as strong evidence for its contribution to the model learning of our method. On the other hand, the outperformance of our pose model over motion model additionally verifies the benefit of leveraging end-to-end learning, despite the practicability of motion supervision with our synthetic DynaSCARED. Compared with models train with other variants (p.DynaSCAREDffd, p.DynaSCAREDed) of DynaSCARED, we observe comparable performance. This is potentially because our motion detection module only takes optic flow as input and the model is directly trained with pose supervision toward motion-aware pose estimation, both reduce the model’s sensitivity to a specific deformation pattern simulated in the training data. Moreover, the target task of pose estimation can also alleviate the tendency of our model in overfitting to the intermediate motion detection, and consequently contributes in bridging the synthetic-real gap.