The rapid development of large language models (LLMs) has significantly expanded the possibilities of artificial intelligence (AI) in healthcare, particularly in medical education and clinical decision-making [1,2]. Orthopaedic surgery, encompassing diverse areas such as anatomy, trauma, sports medicine, and regenerative approaches, represents a field where continuous updates of guidelines and research findings create a strong need for efficient knowledge management. In this context, LLMs may serve as “knowledge assistants,” potentially improving literature review, physician training, and clinical planning. However, LLMs rely on statistical pattern recognition rather than genuine human-like understanding. Their reliability in complex, nuanced tasks—such as accurately interpreting diagnostic images, applying evolving guidelines, and integrating patient factors—remains uncertain. Orthopaedic decision-making often depends heavily on radiographic findings and requires contextual integration of comorbidities, patient activity levels, and social considerations. Although some newer LLMs demonstrate better reasoning capabilities, it is unclear if these models can effectively replicate specialist-level clinical judgment, especially in high-stakes testing situations like board certification examinations.

GPT-4o (4o) is a widely recognized LLM; more recently, GPT-o1 pro (o1 pro) was introduced with enhanced chain-of-thought reasoning abilities. While preliminary studies suggest that o1 pro surpasses 4o on certain complex inference tasks, including specialized exams, its performance on comprehensive medical board questions that assess basic science, clinical reasoning, and imaging interpretation remains insufficiently explored.

To address this gap, we evaluated the performance of both 4o and o1 pro on the Japanese Orthopaedic Association (JOA) Board Certification Examination. This exam covers a broad spectrum of topics ranging from foundational orthopaedic knowledge to challenging imaging-based diagnoses. By comparing the two models, we aimed to determine whether o1 pro's advanced reasoning translates into improved accuracy on real examination questions and to identify specific strengths and weaknesses relevant to clinical practice. The findings from this study could inform future development and safe integration of LLMs in orthopaedic education and patient care.