A non-arthritic scapula phantom no. 1021 (Sawbones, Vashon, WA, USA) was digitized using an LLP HD 3D scanner (Faro, Lake Mary, FL, USA), such that a closed 3D surface mesh was obtained. From this, a rigid tracker was designed to fit along the coracoid process, as this is typically accessible, and support a constellation of four retroreflective infrared markers (Northern Digital Inc., Waterloo, Ontario, Canada). The tracker was fabricated in PA-12 using selective laser sintering (EOS, Krailling, Germany) and affixed, with IR markers, to the phantom (Fig. 1a). This assembly was likewise digitized, resulting in a second surface mesh from which the 3D centroids of the IR markers were determined through sphere fitting (Fig. 1b).

Based on this surface mesh, a ground-truth glenoid trajectory was defined using Mimics (Materialise, Leuven, Belgium) and comprised an entry point and direction at the glenoid face (Fig. 1c). A coordinate system was defined representing anatomical directions at the entry point: superior-inferior (SI), anterior-posterior (AP), medial-lateral (ML).

A rigid tracker consisting of five passive IR markers was affixed to a System 7 surgical drill (Stryker, Kalamazoo, MI, USA) equipped with a 3.2 mm twist drill. Similarly to the phantom, the drill was 3D scanned, whereafter the coordinates of the IR markers were determined through sphere fitting. Additionally, and from the same scan, the position and orientation of the drill bit were identified. For all trackers, the coordinates of their respective IR markers were used to define a local coordinate system in which relevant data were referenced, e.g., a vector or point.

To provide an AR workflow encompassing visualization, phantom registration, and drill guidance as described below, an AR application was developed in Unity3D (Unity Technologies, San Francisco, CA, USA), version 2019.4.40f1. The app was deployed to a HoloLens 2 (Microsoft, Redmond, WA, USA) AR-HMD and integrated an inside-out tracking algorithm for pose estimation of rigid trackers using the device’s short-throw IR sensor.

Registration between the virtual and physical phantom was achieved in two steps. First, the investigator used a handheld tracked stylus to identify four predefined points along the glenoid margin. These point pairs provided an initial alignment through least-squares registration. Next, the investigator collected additional points through tracing along the surface of the scapula, limited to regions accessible during deltopectoral approach (Fig. 1c). Final registration was achieved through an iterative closest point (ICP) method, minimizing the point-to-plane distance between the collected data and the scapular surface. Adding additional points allowed for refinement of registration until the investigator was visually satisfied with the alignment between the virtual and physical scapula (Fig. 2a, b).

First person view of augmented reality (AR)-guided pin placement workflow during a point collection for registration, b visual assessment of final registration result, c alignment of the drill to the target trajectory, d drilling with the drill aligned to the planned trajectory

Image processing overview for infrared (IR) tool tracking. a IR reflectivity map from the HoloLens 2 device. b Detected IR markers (pink crosses). c Estimated 3D pose (red–green–blue axes) for both drill and coracoid tracker

Visualization technique for augmented reality (AR) guided drilling, during a alignment of the drill to the target trajectory and b drilling of aligned drill to trajectory. Dashed lines are purely illustrative and not shown in the AR app

Pose estimation of each IR labeled tracker was accomplished in four steps (Fig. 3). First, the IR reflectivity map of the HMD was acquired, Gaussian smoothed and histogram equalized based on the upper 20th percentile. Second, blob detection was used to determine the image position of illuminated IR markers. Then, using both the IR markers’ local coordinates within each physical tracker and their image positions, the pose in six degrees of freedom (DoF) was estimated using perspective-n-point geometry [34]. Lastly, this was transformed into the world coordinate system using the extrinsic transforms of the IR sensor relative to the HMD and world. Four IR markers are required for tracker detection, while tracking can be maintained with a minimum of three. A greater number of IR markers per tracker allow for partial occlusion of each tracker during the workflow.

Drilling of the scapula was performed in two steps. First, the app guided the user to place the drill tip at a position upon the glenoid surface such that its distance to the trajectory (\(\delta \)) was minimized. Secondly, the drill was oriented about its tip such that it minimized its angle with respect to the planned trajectory (\(\alpha \)). To aid in alignment, two red dots were visualized on planes perpendicular to the planned trajectory’s start and end point, indicating intersection of the tracked drilling trajectory through both planes. When the drill achieved the correct position and orientation, the red dots, planned trajectory, and drill trajectory showed co-alignment (Fig. 4). During the positioning and orientation process, the color of the planned trajectory and stylus tool-tip changed independently from red to yellow and to green according to Eq. 1. The red/yellow colors indicate a “no-go” and the green color a “go” condition. This is appreciable in Fig. 2c, d during AR navigation.

$$\begin color= \text , & \text \ \alpha \le ^\circ \text \delta \le \,} \\ \text , & \text \ \alpha \le ^\circ \text \delta \le \,} \\ \text , & \text \end\right. } \end$$

(1)

All measurements were performed by a surgical trainee and an expert in medical image processing, both having extensive experience with the AR-HMD hardware. For each evaluation, a new undrilled phantom scapula with a phantom skin envelope no.1509-24-2 (Sawbones, Vashon, WA, USA) was clamped into a support. The skin envelope was incised to provide an appropriate surgical field. Two drilling techniques were used: first, a control, relying on a proprioceptive guided freehand drilling technique based on prior imaging data and anatomical knowledge; and second, an end-to-end AR-navigated technique providing superposed drilling guidance, as described above, of registered planning. After drilling, a pin of 3.1 mm was inserted into the glenoid. Each of the two investigators performed each placement technique three times, resulting in 12 measurements.

In addition, a series of experiments were performed to investigate the contribution of system errors from tool tracking, phantom registration, and different navigation feedback positions upon the quality of the final pin placement.

To quantify IR tracking performance, a phantom scapula with an additional printed tracker was created (Fig. 5). This second tracker defined a specific anatomical coordinate system at the glenoid surface. During the experiment, the investigator was free to move about the proximity of the phantom while wearing the AR-HMD. Measured drift between the fixed spatial relationship of the trackers gives an estimate of the tracking uncertainty. As such, the pose for each tracker with respect to the AR world coordinate system was logged to the device for analysis a posteriori.

Phantom scapula for infrared (IR) tracking evaluation; trackers (yellow) with local coordinate systems (red–green–blue axes), passive IR retroreflective markers (blue)

Registration quality was evaluated by comparing a series of registration outcomes between a moving AR scapula surface mesh and a fixed phantom scapula, as described above. The two investigators each performed eight registrations. For each attempt, the registered pose (6-DoF), with respect to the phantom tracker, was logged to the device after each attempt for analysis a posteriori. Two investigators performed each series five times.

To assess the effect of visualization techniques on experimental outcomes, a series of scapula drilling experiments were conducted using two AR guidance visualization techniques: superposed and juxtaposed. Superposed navigation placed the AR drilling guidance at the drilling site upon the glenoid surface, while juxtaposed navigation allowed the investigator to manually shift the AR guidance to a position adjacent to the scapula phantom. To eliminate errors associated with registration, all phantoms were preregistered to the IR tracker. Two investigators performed each series three times.

As no human or animal subjects or material were included as part of this work, approval was not required from the institutional review board.

Pin placement outcomes were assessed through 3D scanning of the scapula with the fixed pin in place. From these scans, the pin’s entry point and direction with respect to the glenoid anatomical basis were determined. The difference between the achieved and planned ground-truth entry point and direction is reported as directional errors (inclination and version) [\(^\circ \)] and as positional errors (anterior-posterior and inferior-superior) [mm].

To assess the tracking performance, logged transforms for the phantom’s two IR trackers were transformed into the glenoid basis. From this, their difference is reported as displacement [mm] and rotation [\(^\circ \)].

To assess registration performance, the logged registration was used to transform the planned trajectory into the ground-truth glenoid coordinate system. From this, the difference between the registered and ground-truth entry point and direction is reported as displacements [mm] and rotation [\(^\circ \)]. The number of registration attempts is also reported.