NUR 633 Module 8 Decision Support Analysis Example

When the Alert Is Wrong More Often Than Right: Evaluating a Proprietary Sepsis Prediction Model

[Student Name]

Southern New Hampshire University

NUR 633: Informatics and Communication Technology

Module Eight Decision Support Analysis

[Instructor Name]

[Date]

When the Alert Is Wrong More Often Than Right: Evaluating a Proprietary Sepsis Prediction Model

Predictive models are arriving in electronic records faster than hospitals can evaluate them. Many are proprietary: their inputs and weighting are not published, and their performance is reported by the company that sells them. Nurses are usually the first to receive their alerts and the first to feel the consequences when those alerts are unreliable. This paper examines one widely implemented sepsis prediction model after an independent evaluation. It argues that the model performed far worse outside its developer's reports, that its alerts would burden nurses while missing most sepsis cases and that nurses should insist on local validation and on specific performance measures before any predictive alert is switched on.

The Tool and the Test

The model calculates a sepsis risk score for hospitalized adults every 15 minutes using data in the record, and hospitals choose a score threshold at which clinicians are alerted. It had been adopted by hundreds of hospitals, but its developer's reports of its accuracy had not been independently confirmed. Wong and colleagues tested it retrospectively at one academic health system, where it had been running in the background, on 38,455 hospitalizations among 27,697 adults. Sepsis, defined by criteria combining infection and organ dysfunction, occurred in 2,552 hospitalizations, or 7% (Wong et al., 2021).

What the Numbers Mean

Three measures describe the model's performance, and each matters differently to a nurse. The C statistic, shorthand for the area under the ROC curve, summarizes how well a model separates patients who develop the outcome from those who do not; 0.5 is no better than chance and 1.0 is perfect. The model's hospitalization-level value was 0.63, far below the figures its developer had reported, and generally considered poor discrimination.

Sensitivity is the proportion of patients with sepsis whom the model flags, and missed cases are the price of low sensitivity. At the commonly used threshold of 6, the model generated alerts for 6,971 hospitalizations, 18% of all patients, yet it failed to flag 1,709 of the 2,552 patients who developed sepsis, 67% of them (Wong et al., 2021). Its sensitivity was therefore about 33%. The model also identified only 183 septic patients, 7%, who had not already received timely antibiotics from clinicians, meaning that in most cases it recognized sepsis after clinicians had.

Positive predictive value is what a nurse holding the alert most wants to know: when the alert fires, how likely is this patient to have sepsis? The study data allow a rough calculation. If about 843 septic patients were flagged among 6,971 alerts, the positive predictive value was about 12%. Roughly seven of every eight alerts would concern a patient who did not develop sepsis.

Why Performance Fell

Several explanations are possible, and the study authors and commentators discussed them. Models built at some hospitals can perform worse elsewhere because patient populations, documentation habits and coding practices differ. The model may rely partly on actions clinicians take once they already suspect sepsis, such as ordering blood cultures, which would make it good at confirming clinicians' suspicions and poor at anticipating them. And performance reported by a developer, without independent peer review, may not reflect use in routine practice. A commentary accompanying the study concluded that the findings showed why external validation should be required before such models are deployed (Habib et al., 2021).

Consequences for Nurses

An alert that fires for nearly one in five patients and is wrong about seven times in eight is a recipe for alarm fatigue, the desensitization that follows exposure to large numbers of nonactionable alerts (Sendelbach & Funk, 2013). Nurses who learn that the sepsis alert is usually wrong will begin to dismiss it, including in the minority of cases where it is right. Meanwhile, two thirds of septic patients would never trigger it, so a unit that relied on the tool could become less vigilant rather than more. The harm is not only wasted time; it is misplaced trust. There is also an equity dimension. Proprietary models trained on one population may perform differently across age, race or insurance groups, and when the inputs are hidden, hospitals cannot easily check whether some patients are flagged more or less accurately than others. Independent validation should therefore report performance for important subgroups, and hospitals should ask vendors for that information before deployment.

What Makes Decision Support Work

Decision support can improve care when it is designed well. A systematic review of 70 randomized trials found that decision support systems improved clinical practice in 68% of trials and identified four features independently associated with success: decision support delivered automatically as part of clinician workflow, recommendations rather than assessments alone, support arriving where and when the decision is made and computer-based delivery (Kawamoto et al., 2005). A sepsis risk score that simply announces risk, without a recommended action and with poor accuracy, satisfies few of these conditions.

Five Questions Nurses Should Ask

Before any predictive alert goes live on their units, nurses and nurse leaders should ask five questions. First, has the model been validated on this hospital's own patients, and what were its sensitivity and positive predictive value at the proposed threshold? Second, how many alerts per shift will a nurse receive, and how many of those will be true? Third, what action does the alert recommend, and who is responsible for taking it? Fourth, how will the alert be monitored after go-live, including missed cases? Fifth, who can turn it off or change the threshold if it performs poorly? Until a vendor or the hospital's analysts can answer all five, the alert should stay switched off.

Conclusion

The independent test of a widely used sepsis model found poor discrimination, low sensitivity and a heavy burden of false alerts, a stark contrast with its developer's claims. For nurses, the lesson extends beyond one product: predictive alerts should be judged on local evidence, on the measures that matter at the bedside and on whether they tell a clinician what to do. Nurses have both the standing and the responsibility to ask.

References

Habib, A. R., Lin, A. L., & Grant, R. W. (2021). The Epic Sepsis Model falls short: The importance of external validation. JAMA Internal Medicine, 181(8), 1040-1041. https://doi.org/10.1001/jamainternmed.2021.3333

Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005). Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ, 330(7494), 765-768. https://doi.org/10.1136/bmj.38398.500764.8F

Sendelbach, S., & Funk, M. (2013). Alarm fatigue: A patient safety concern. AACN Advanced Critical Care, 24(4), 378-386. https://doi.org/10.1097/NCI.0b013e3182a903f9

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626