The Impact of Healthcare Professionals’ Characteristics on the Evaluation of Clinical Decision Support Systems: Insights from a Cross-Country Usability and Technology Acceptance Study of the iCARE Tool

K. Kawamoto, C. A. Houlihan, E. A. Balas, and D. F. Lobach, “Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success,” BMJ, vol. 330, no. 7494, p. 765, 2005. [Online]. Available: https://www.bmj.com/content/330/7494/765

A. X. Garg, N. K. J. Adhikari, H. McDonald, M. P. Rosas-Arellano, P. J. Devereaux, J. Beyene, J. Sam, and R. B. Haynes, “Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review,” JAMA, vol. 293, no. 10, pp. 1223–1238, 03 2005. [Online]. Available: https://doi.org/10.1001/jama.293.10.1223

R. Kunhimangalam, S. Ovallath, and P. K. Joseph, “A clinical decision support system with an integrated EMR for diagnosis of peripheral neuropathy,” Journal of Medical Systems, vol. 38, no. 4, pp. 1–14, 2014. [Online]. Available: https://doi.org/10.1007/s10916-014-0038-9

R. Kwok, M. Dinh, D. Dinh, and M. Chu, “Improving adherence to asthma clinical guidelines and discharge documentation from emergency departments: Implementation of a dynamic and integrated electronic decision support system,” Emergency Medicine Australasia, vol. 21, no. 1, pp. 31–37, 2009. [Online]. Available: https://doi.org/10.1111/j.1742-6723.2008.01149.x

P. J. Helmons, B. O. Suijkerbuijk, P. V. Nannan Panday, and J. G. W. Kosterink, “Drug–drug interaction checking assisted by clinical decision support: A return on investment analysis,” Journal of the American Medical Informatics Association, vol. 22, no. 4, pp. 764–772, 2015. [Online]. Available: https://doi.org/10.1093/jamia/ocu010

S. Eslami, N. F. de Keizer, D. A. Dongelmans, E. de Jonge, M. J. Schultz, and A. Abu-Hanna, “Effects of two different levels of computerized decision support on blood glucose regulation in critically ill patients,” International Journal of Medical Informatics, vol. 81, no. 1, pp. 53–60, 2012. [Online]. Available: https://doi.org/10.1016/j.ijmedinf.2011.10.003

L. G. Armando, G. Miglio, P. de Cosmo, and C. Cena, “Clinical decision support systems to improve drug prescription and therapy optimisation in clinical practice: A scoping review,” BMJ Health & Care Informatics, vol. 30, no. 1, p. e100683, 2023. [Online]. Available: https://doi.org/10.1136/bmjhci-2022-100683

J. M. Wohlgemut, E. Pisirir, R. S. Stoner, Z. B. Perkins, W. Marsh, N. R. Tai, and E. Kyrimi, “A scoping review, novel taxonomy and catalogue of implementation frameworks for clinical decision support systems,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, p. 323, 2024. [Online]. Available: https://doi.org/10.1186/s12911-024-02739-1

A. E. Davidson, J. M. Ray, A. K. Patel, Y. S. Levites, P. Rashidi, and A. Bihorac, “An iterative, user-centered design of a clinical decision support system for critical care assessments: Co-design sessions with icu clinical providers,” arXiv preprint arXiv:2503.08814, 2025. [Online]. Available: https://arxiv.org/abs/2503.08814

B. C. Stagg, B. Tullis, A. Asare, J. D. Stein, F. A. Medeiros, C. Weir, and K. Kawamoto, “Systematic user-centered design of a prototype clinical decision support system for glaucoma,” Ophthalmology Science, vol. 3, no. 3, p. 100279, 2023. [Online]. Available: https://doi.org/10.1016/j.xops.2023.100279

E. Cuvelier, L. Robert, E. Musy, C. Rousseliere, R. Marcilly, S. Gautier, P. Odou, J.-B. Beuscart, and B. Decaudin, “The clinical pharmacist’s role in enhancing the relevance of a clinical decision support system,” International Journal of Medical Informatics, vol. 155, p. 104568, 2021. [Online]. Available: https://doi.org/10.1016/j.ijmedinf.2021.104568

M. Nuutinen and R.-L. Leskela, “Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system,” Health and Technology, vol. 13, no. 4, pp. 557–570, 2023. [Online]. Available: https://doi.org/10.1007/s12553-023-00763-1

Q. Q. Zhou, J. Wang, W. Tang, Z. C. Hu, Z. Y. Xia, X. S. Li, R. Zhang, X. Yin, B. Zhang, and H. Zhang, “Automatic detection and classification of rib fractures on thoracic ct using convolutional neural network: Accuracy and feasibility,” Korean Journal of Radiology, vol. 21, no. 7, pp. 869–879, 2020. [Online]. Available: https://doi.org/10.3348/kjr.2019.0651

L. Wu, J. Zhang, W. Zhou, P. An, L. Shen, J. Liu, X. Jiang, X. Huang, G. Mu, X. Wan, X. Lv, J. Gao, N. Cui, S. Hu, Y. Chen, X. Hu, J. Li, D. Chen, D. Gong, X. He, Q. Ding, X. Zhu, S. Li, X. Wei, X. Li, X. Wang, J. Zhou, M. Zhang, and H. G. Yu, “Randomised controlled trial of wisense, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy,” Gut, vol. 68, no. 12, pp. 2161–2169, 2019. [Online]. Available: https://doi.org/10.1136/gutjnl-2018-317366

F. Dhombres, P. Maurice, L. Guilbaud, L. Franchinard, B. Dias, J. Charlet, E. Blondiaux, B. Khoshnood, D. Jurkovic, E. Jauniaux, and J. Jouannic, “A novel intelligent scan assistant system for early pregnancy diagnosis by ultrasound: Clinical decision support system evaluation study,” Journal of Medical Internet Research, vol. 21, no. 7, p. e14286, 2019. [Online]. Available: https://doi.org/10.2196/14286

J. Su, Z. Li, X. Shao, C. Ji, R. Ji, R. Zhou, Z. Li, C. Chen, M. Zhang, X. Fan, P. Ji, Q. Zhou, F. Zhou, C. Wang, Z. Ai, M. Chen, and G. Wang, “Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: A prospective randomized controlled study,” Gastrointestinal Endoscopy, vol. 91, no. 2, pp. 415–424, 2020. [Online]. Available: https://doi.org/10.1016/j.gie.2019.08.026

J. Kirakowski, M. Corbett, and M. Sumi, “The software usability measurement inventory,” British Journal of Educational Technology, vol. 24, no. 3, pp. 210–212, 1993. [Online]. Available: https://doi.org/10.1111/j.1467-8535.1993.tb00076.x

F. D. Davis, “Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Quarterly, vol. 13, no. 3, pp. 319–340, 1989. [Online]. Available: https://doi.org/10.2307/249008

V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User acceptance of information technology: Toward a unified view,” MIS Quarterly, vol. 27, no. 3, pp. 425–478, 2003. [Online]. Available: https://doi.org/10.2307/30036540

N. Toda, M. Hashimoto, Y. Iwabuchi, M. Nagasaka, R. Takeshita, M. Yamada, Y. Yamada, and M. Jinzaki, “Validation of deep learning-based computer-aided detection software for pulmonary abnormalities on chest radiographs and examination of factors that influence readers’ performance and final diagnosis,” Japanese Journal of Radiology, vol. 41, no. 1, pp. 38–44, 2023. [Online]. Available: https://doi.org/10.1007/s11604-022-01330-w

T. Wataya, M. Yanagawa, M. Tsubamoto, T. Sato, D. Nishigaki, K. Kita, K. Nakajima, J. Tsuchiya, T. Aoki, H. Kataoka, and O. U. R. Team, “Radiologists with and without deep learning-based computer-aided diagnosis: Comparison of performance and interobserver agreement for characterizing and diagnosing pulmonary nodules/masses,” European Radiology, vol. 33, no. 1, pp. 348–359, 2023. [Online]. Available: https://doi.org/10.1007/s00330-022-08948-4

Q. Wei, S. Zeng, L. Wang, Y. Yan, T. Wang, J. Xu, X. Luo, X. Chen, X. Liu, X. Wei, and L. Pan, “The added value of a computer-aided diagnosis system in differential diagnosis of breast lesions by radiologists with different experience,” Journal of Ultrasound in Medicine, vol. 41, no. 6, pp. 1355–1363, 2022. [Online]. Available: https://doi.org/10.1002/jum.15816

M. Georgsson and N. Staggers, “Quantifying usability: Evaluation of a diabetes mhealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics,” Journal of the American Medical Informatics Association, vol. 23, no. 1, pp. 5–11, 2016. [Online]. Available: https://doi.org/10.1093/jamia/ocv099

G. B. Raglan, B. Margolis, R. A. Paulus, and J. Schulkin, “Electronic health record adoption among obstetrician/gynecologists in the united states: Physician practices and satisfaction,” Journal for Healthcare Quality, vol. 39, no. 3, pp. 144–152, 2017. [Online]. Available: https://doi.org/10.1111/jhq.12072

H. Varonen, T. Kortteisto, and M. Kaila, “What may help or hinder the implementation of computerized decision support systems: A focus group study with physicians,” Family Practice, vol. 25, no. 3, pp. 162–167, 2008. [Online]. Available: https://doi.org/10.1093/fampra/cmn020

F. R. Jeddi, E. Nabovati, R. Bigham, and R. Khajouei, “Usability evaluation of a comprehensive national health information system: Relationship of quality components to users’ characteristics,” International Journal of Medical Informatics, vol. 133, p. 104026, 2020. [Online]. Available: https://doi.org/10.1016/j.ijmedinf.2019.104026

S. L. Decker, E. W. Jamoom, and J. E. Sisk, “Physicians in nonprimary care and small practices and those age 55 and older lag in adopting electronic health record systems,” Health Affairs, vol. 31, no. 5, pp. 1108–1114, 2012. [Online]. Available: https://doi.org/10.1377/hlthaff.2011.1121

C. M. DesRoches, E. G. Campbell, S. R. Rao, K. Donelan, T. G. Ferris, A. K. Jha, R. Kaushal, D. E. Levy, S. Rosenbaum, A. E. Shields, and D. Blumenthal, “Electronic health records in ambulatory care - a national survey of physicians,” New England Journal of Medicine, vol. 359, no. 1, pp. 50–60, 2008. [Online]. Available: https://doi.org/10.1056/NEJMsa0802005

E. O. Hoogendijk, G. Onder, L. Smalbil, D. L. Vetrano, J. P. Hirdes, E. P. Howard, J. N. Morris, D. Fialova, K. Szczerbinska, E. C. M. Kooijmans, M. Hoogendoorn, A. Declercq, J. De Almeida Mello, R. L. Leskela, J. Hasa, J. Edgren, G. Ruppe, R. Liperoti, K. J. Joling, H. P. J. van Hout, and I-CARE4OLD consortium, “Optimising the care for older persons with complex chronic conditions in home care and nursing homes: Design and protocol of I-CARE4OLD, an observational study using real-world data,” BMJ Open, vol. 13, no. 6, p. e072399, 2023. [Online]. Available: https://doi.org/10.1136/bmjopen-2023-072399

interRAI, “Instruments,” https://interrai.org/instruments/, 2025, accessed 31 Mar. 2025.

M. Nuutinen, R.-L. Leskelä, D. Fialova, I. Haavisto, H. Finne-Soveri, J. Häsä, J. Edgren, H. van Hout, D. E. da Cunha Leme, J. P. Hirdes et al., “Effect of discontinuing antipsychotic medications on the risk of hospitalization in long-term care: a machine learning-based analysis,” BMC medicine, vol. 23, no. 1, p. 484, 2025.

J. N. Morris, E. P. Howard, E. Schachter, S. Burney, A. Laytham, D. Fialova, E. O. Hoogendijk, R. Liperoti, H. P. van Hout, and D. L. Vetrano, “Cognitive change among nursing home residents: Cogrisk-nh scale development to predict decline,” Journal of the American Medical Directors Association, vol. 24, no. 9, pp. 1405–1411, 2023.

Article  PubMed  Google Scholar 

J. N. Morris, E. P. Howard, E. Schachter, V. Pešic, A. K. Laytham, and S. N. Burney, “Risk of death among nursing home residents: A cross-national perspective,” Journal of the American Medical Directors Association, vol. 25, no. 9, p. 105142, 2024.

Article  PubMed  Google Scholar 

A.-M. Hiltunen, I. Haavisto, M. Nuutinen, M. Lahelma, A. Salminen, J. de Almeida Mello, R. Liperoti, E. P. Howard, D. Fialova, K. Szczerbinska, M. Alon, I. Baranska, C. Exmann, A. Declercq, W. Boorsma, G. Onder, H. P. J. van Hout, R.-L. Leskela, and I.-C. Consortium, “Protocol of the pilot study to test and evaluate the iCARE tool: A machine learning-based e-platform tool to make health prognoses and support decision-making for the care of older persons with complex chronic conditions,” BMJ Open, vol. 15, no. 4, p. e101234, 2025. [Online]. Available: https://doi.org/10.1136/bmjopen-2025-101234

J. R. Lewis, “IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use,” International Journal of HumanComputer Interaction, vol. 7, no. 1, pp. 57–78, 1995. [Online]. Available: https://doi.org/10.1080/10447319509526110

M. P. Gagnon, E. Orruño, J. Asua, A. B. Abdeljelil, and J. Emparanza, “Using a modified technology acceptance model to evaluate healthcare professionals’ adoption of a new telemonitoring system,” Telemedicine and e-Health, vol. 18, no. 1, pp. 54–59, 2012.

Article  PubMed  Google Scholar 

G. Strudwick, “Predicting nurses’ use of healthcare technology using the technology acceptance model: an integrative review,” CIN: Computers, Informatics, Nursing, vol. 33, no. 5, pp. 189–198, 2015.

PubMed  Google Scholar 

A. A. AlQudah, M. Al-Emran, and K. Shaalan, “Technology acceptance in healthcare: a systematic review,” Applied Sciences, vol. 11, no. 22, p. 10537, 2021.

Article  Google Scholar 

S. Hajesmaeel-Gohari, F. Khordastan, F. Fatehi, H. Samzadeh, and K. Bahaadinbeigy, “The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health,” BMC Medical Informatics and Decision Making, vol. 22, no. 1, p. 22, 2022.

Article  PubMed  PubMed Central  Google Scholar 

S. Valizadeh-Haghi, S. Rahmatizadeh, M. M. Forouzanfar, Z. Kohzadi, and F. Fatehi, “Usability evaluation of emergency department information system: A case of discharge and admission sections of an educational hospital.” Frontiers in Health Informatics, vol. 12, 2023.

B. Patel and A. Thind, “Usability of mobile health apps for postoperative care: systematic review,” JMIR Perioperative Medicine, vol. 3, no. 2, p. e19099, 2020.

Article  PubMed  PubMed Central  Google Scholar 

S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: https://doi.org/10.2307/4615733

L. J. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika, vol. 16, no. 3, pp. 297–334, 1951.

Article  Google Scholar 

J. M. Cortina, “What is coefficient alpha? an examination of theory and applications,” Journal of Applied Psychology, vol. 78, no. 1, pp. 98–104, 1993.

Article  Google Scholar 

H. Abdi and L. J. Williams, “Principal component analysis,” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 2, no. 4, pp. 433–459, 2010. [Online]. Available: https://doi.org/10.1002/wics.101

S. P. Lloyd, “Least squares quantization in PCM,” IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 129–137, 1982. [Online]. Available: https://doi.org/10.1109/TIT.1982.1056489

J. MacQueen, “Some methods for classification and analysis of multivariate observations,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1. University of California Press, 1967, pp. 281–297. [Online]. Available: https://dl1.icdst.org/pdfs/files/89776553ba539ae6ab958d2fb6b97757.pdf

H. F. Kaiser, “An index of factorial simplicity,” Psychometrika, vol. 39, no. 1, pp. 31–36, 1974. [Online]. Available: https://doi.org/10.1007/BF02291575

G. W. Snedecor and W. G. Cochran, Statistical Methods, 8th ed. Ames, IA: Iowa State University Press, 1989.

Google Scholar 

P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics, vol. 20, pp. 53–65, 1987. [Online]. Available: https://doi.org/10.1016/0377-0427(87)90125-7

H. P. Fransen, A. M. May, M. D. Stricker, J. M. A. Boer, C. Hennig, Y. Rosseel, M. C. Ocke, P. H. M. Peeters, and J. W. J. Beulens, “A posteriori dietary patterns: How many patterns to retain?” Journal of Nutrition, vol. 144, no. 8, pp. 1274–1282, 2014. [Online]. Available: https://doi.org/10.3945/jn.113.188680

M. G. Thorpe, C. M. Milte, D. Crawford, and S. A. McNaughton, “A comparison of the dietary patterns derived by principal component analysis and cluster analysis in older australians,” International Journal of Behavioral Nutrition and Physical Activity, vol. 13, no. 1, p. 30, 2016. [Online]. Available: https://doi.org/10.1186/s12966-016-0353-2

J. Sauro and J. R. Lewis, Quantifying the User Experience: Practical Statistics for User Research, 2nd ed. Amsterdam, The Netherlands: Morgan Kaufmann / Elsevier, 2016. [Online]. Available: https://www.elsevier.com/books/quantifying-the-user-experience/9780128023082

C. D. Dziuban and E. C. Shirkey, “When is a correlation matrix appropriate for factor analysis? some decision rules,” Psychological Bulletin, vol. 81, no. 6, pp. 358–361, 1974. [Online]. Available: https://doi.org/10.1037/h0036316

S. Michie and T. Marteau, “Non-response bias in prospective studies of patients and health care professionals,” International Journal of Social Research Methodology, vol. 2, pp. 203–212, 1999. [Online]. Available: https://doi.org/10.1080/136455799295014

M. Estrela, G. Semedo, F. Roque, P. Lopes Ferreira, and M. T. Herdeiro, “Sociodemographic determinants of digital health literacy: A systematic review and meta-analysis,” International Journal of Medical Informatics, vol. 177, p. 105124, 2023. [Online]. Available: https://doi.org/10.1016/j.ijmedinf.2023.105124

Comments (0)

No login
gif