External Validation of a Machine Learning Model to Predict Postpartum Hemorrhage in a US Northeastern Healthcare System

ABSTRACT

Introduction Postpartum hemorrhage (PPH) is a major cause of maternal morbidity and mortality. Timely prediction may prevent adverse maternal outcomes, and efforts are needed to develop accurate predictive tools. A high-performing machine learning model to predict PPH using data from the US Consortium for Safe Labor (CSL) remains to be widely validated in contemporary clinical settings using electronic health record (EHR) data. Our goal was to evaluate the performance of the CSL PPH predictive model using EHR data across a large healthcare system in the Northeastern US.

Methods We conducted a retrospective cohort study across eight hospitals in the Northeast US between 05/2015 and 05/2024. We used the same sociodemographic, clinical diagnoses, family history, laboratory, and vital signs available on labor and delivery admission in the EHR that were used to train the original CSL model. The binary outcome was PPH, defined as estimated blood loss of 1000 ml or more at delivery or blood transfusion within 24 hours postpartum. We then refit a new model using the original features to assess whether model performance could be further improved in our study population using the best-performing machine learning approach (XGBoost) from the original CSL model. We evaluated model discrimination as measured using the area under the curve (AUC), feature importance, calibration, and decision analysis curves of both the original CSL model with external validation and the further refit model.

Results Among 87,662 deliveries, the incidence of PPH was 7.7%. The original CSL model demonstrated modest discrimination for predicting PPH with an AUC of 0.60 (95% CI, 0.58– 0.61). Refitting a new model with XGBoost resulted in improved discrimination with an AUC of 0.75 (95% CI, 0.74–0.76). Calibration analyses demonstrated that the refit model overestimated PPH risk across a range of predicted probabilities.

Conclusion A promising PPH predictive model had substantially reduced performance with external validation using contemporary EHR data across an eight-hospital health system in the Northeastern US. These findings highlight the importance of external validation, local adaptation, and ongoing surveillance for assessing model performance in an era of evolving prevention, management, and treatment strategies for PPH.

Key Points

This study aimed to externally validate a previously published machine learning model for predicting postpartum hemorrhage (PPH) and assess its portability across eight hospitals using electronic health record data on labor and delivery admission.

We found that the original model demonstrated modest discrimination (area under the curve, AUC: 0.60) with external validation.

A refit model achieved improved discrimination (AUC: 0.75) but remained poorly calibrated and overestimated the risk of PPH across a range of predicted probabilities.

These findings underscore the importance of local validation and adaptation of external models, and ongoing performance monitoring before clinical deployment of PPH prediction models in an era of evolving prevention, management, and treatment strategies for PPH.

Competing Interest Statement

VPK reports funding from the NIH/NHLBI grants 1K08HL161326-01A1, Anesthesia Patient Safety Foundation (APSF), and BWH IGNITE Award. VPK reports consulting fees from Avania CRO unrelated to the current work and patent #WO2021119593A1 for the control of a therapeutic delivery system assigned to Mass General Brigham. DWB reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from ValeraHealth, equity from Clew, equity from MDClone, personal fees and equity from AESOP, personal fees and equity from FeelBetter, personal fees and equity from Guided Clinical Solutions, outside the submitted work. KJG reports consulting fees for BillionToOne, Aetion, Roche, and Janssen Global, outside the submitted work. JEJ reports institutional funding from NICHD, NIDDK, and FDA, honoraria from UpToDate.

Funding Statement

Supported by NIH/NHLBI grants 1K08HL161326-01A1and the Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Womens Hospital.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Mass General Brigham Institutional Review Board

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Footnotes

Twitter (X) post: Check out our new paper, highlighting that even promising AI models predicting bleeding after childbirth lose accuracy when validated externally. It emphasizes the need for local adaptation and ongoing surveillance in maternal care.

More from this channel

External Validation of a Machine Learning Model to Predict Postpartum Hemorrhage in a US Northeastern Healthcare System

Comments (0)