NUR 651 Module 6 Algorithm Governance Paper Example

Governing Clinical Algorithms: A Sepsis Alert, a Biased Risk Score and a Review Process for a Health System

[Student Name]

Southern New Hampshire University

NUR 651: Advanced Concepts for Nurse Executive Leaders

Module Six Algorithm Governance Paper

[Instructor Name]

[Date]

Governing Clinical Algorithms: A Sepsis Alert, a Biased Risk Score and a Review Process for a Health System

Algorithms now influence which patients nurses are told to check, which patients receive extra services and which alerts interrupt a shift. Many arrive inside the electronic health record or from vendors with performance figures from other organizations, and many are switched on without local testing. Nurses experience the results directly, in alerts that fire too often or in patients who are never flagged. In the composite Northfield system, two tools illustrate the problem: a vendor sepsis prediction alert that nurses describe as constant noise and a commercial risk score used to select patients for a care management program. This paper examines what can go wrong with clinical algorithms, using published evaluations of tools like these, and proposes a governance process. It argues that algorithms should be treated like other clinical interventions, validated locally, checked for bias, explained to users and monitored after launch, with nursing at the decision table.

When Advertised Performance Does Not Hold

Wong et al. (2021) evaluated a proprietary sepsis prediction model, widely implemented in United States hospitals, using nearly 40,000 hospitalizations at one academic health system. The model's ability to separate septic from nonseptic patients was substantially lower than the developer had reported, with an area under the curve of about 0.63. At the alert threshold in use, it missed about two thirds of sepsis cases while generating alerts on about 18% of all hospitalized patients, many of whom did not develop sepsis. The authors concluded that the model's poor performance, combined with its wide use, raised concerns about alert fatigue and about hospitals relying on vendor claims without independent validation.

Northfield's experience matches this pattern. In a month-long audit, the sepsis alert fired on 21% of adult inpatients on medical units, and nurses dismissed nine of ten alerts within a minute. When an alert is usually wrong, nurses learn to ignore it, which means the rare correct alert is ignored too.

When an Algorithm Encodes Bias

The second risk is harder to see. Obermeyer et al. (2019) examined a commercial algorithm used by health systems to identify patients for care management programs. The algorithm predicted future health care costs and used that prediction as a proxy for health needs. Because less money is spent on Black patients than on White patients with the same level of illness, reflecting unequal access to care, the algorithm assigned Black patients lower risk scores than equally sick White patients. Two patients with an identical score were not equally sick: the Black patient typically carried more chronic disease. The authors estimated that correcting the problem would more than double the proportion of Black patients selected for extra help. The algorithm was not designed to be biased; the bias came from the choice of what to predict.

Northfield's care management program uses a commercial risk score whose target variable has never been reviewed. An analysis of last year's enrollment found that Black patients made up 19% of the system's adult primary care population but 11% of those selected for care management, a gap that warrants investigation of how the score is built.

A Governance Process

Northfield will establish a clinical algorithm governance committee reporting to the quality committee of the board, led jointly by nursing's top executive and the physician in charge of clinical informatics, with members from nursing, medicine, pharmacy, informatics, data science, equity and patient advocacy. No algorithm that influences patient care will be activated or kept in use without the committee's review.

The review has four parts. First, every tool must have a model facts label. Sendak et al. (2020) proposed such labels for clinical machine learning models, modeled on nutrition labels, summarizing in plain language what the model predicts, how it was developed, its performance, the populations in which it was validated, warnings about inappropriate uses and how it should inform decisions. Vendors will be asked to provide this information, and the committee will complete the label locally. Second, before activation, the tool will run silently on Northfield's own patients for at least three months, and its performance, including alert rate, sensitivity and positive predictive value, will be compared with the vendor's claims. Third, performance will be examined by race, ethnicity, sex, age and payer, and the target variable will be reviewed for proxies, such as cost, that may encode unequal access. Fourth, after activation, performance and alert burden will be monitored quarterly, since models can drift as patient populations and practice change.

Applying the Process to the Two Tools

The sepsis alert will be moved back to silent mode on all but one pilot unit while it is validated locally. If local performance resembles the Wong findings, the committee will consider raising the threshold, limiting the alert to certain units or replacing it with a nurse-driven screening protocol. The care management risk score will be reviewed for its target variable; if it predicts cost, the committee will ask the vendor for a version based on health measures such as active chronic conditions, or build a local model, and will compare selection rates by race before and after. In both cases, nurses on the affected units will be asked to describe how the tool fits their workflow, since usability shapes whether a tool helps or harms.

Why Nursing Must Be at the Table

Nurses receive most bedside alerts, act on many risk scores and see when a tool's recommendations do not match the patient in front of them. Their experience of alert fatigue is a safety signal, not a complaint. Nursing representation on the committee, and a simple way for nurses to report problems with any algorithm, ensures that the people who use these tools most can influence whether they stay in use. Nursing informatics specialists can also help translate model facts labels into guidance that makes sense on the unit.

Conclusion

Clinical algorithms can help nurses find deteriorating patients and direct scarce services, but published evaluations show they can also perform far worse than advertised and encode inequity through the choice of what they predict. A governance committee with nursing leadership, model facts labels, local silent validation, bias review and ongoing monitoring gives Northfield a way to use these tools with the same scrutiny it applies to medications and devices.

References

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342

Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020). Presenting machine learning model information to clinical end users with model facts labels. npj Digital Medicine, 3, Article 41. https://doi.org/10.1038/s41746-020-0253-3

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626