Epic’s algorithms aren’t performing well in the real world, study shows

A new meta-analysis in SpringerNature shows what researchers have long said about Epic’s artificial intelligence: its out-of-the-box tools don’t perform well in the real world. The insight isn’t necessarily brand new, but this study is the best research to date on the issue. 

Doctors, researchers, health IT, and technology staff at Northwell Health conducted the analysis. 

The study reviewed Epic’s Deterioration Index, Sepsis Model, Unplanned Readmission Model, End of Life Care Index, and Risk of No Show Patient Model. In the research, the models achieved modest real-world performance, with none surpassing an AUROC of .79 (​​an acceptable level of performance). Researchers’ confidence intervals for three of the models (ESM, EURM, EEOL-CI) were below Epic’s self-reported statistics. 

What does that mean? In the words of researchers: “For many clinical use cases, especially high-stakes applications like sepsis detection or ICU transfer, this level of discrimination may result in excessive false positives or missed cases, potentially limiting clinical utility.” 

They also say that false positives can contribute to unnecessary diagnostics, anti-microbial use, and, of course, the dreaded alert fatigue. 

But the bigger takeaway, researchers wrote, is that health systems must locally validate these tools before deploying them—not an easy task. 

The reason that this analysis matters is that Epic’s electronic health record, along with its various tools, is widely used by hospitals. As of the end of 2024, Epic had a 42.3 percent share of acute care systems, according to KLAS Research. Epic also makes it very easy to turn on its tools. 

In a prepared statement, Epic said the meta-analysis examined studies that largely assessed its first-generation models. In second-generation models, for example, its sepsis model, it has included built-in capabilities that help fit the algorithm to the health system’s local environment, and pointed me to this study. The company agrees with the paper’s central takeaway: that local validation and customization are necessary to make its tech work optimally. 

Big picture: Electronic health record companies, including Epic, are increasingly competing to be all-in-one platforms that have it all. They are competing not only with other electronic health record companies, but also with health tech companies that offer individual products, like a sepsis algorithm. As the health industry's interest in AI booms, Epic and other electronic health records have put increasing effort and money into developing those tools.

A good recent example of this is in the ambient scribing and documentation space, where VC-backed companies like Abridge and Ambience Healthcare have found market share. Epic and Microsoft teamed up on their own offering in August of last year, as part of their ongoing efforts to release AI  applications to the market.

Local, recurrent, clinical validation: Every health system has a unique infrastructure and patient population, which is why validating AI products before they’re deployed is important.  

However, not all health systems have the money, resources, or knowledge necessary to do that. 

Since 2024, there have been several efforts to create transparency around and validation of health AI performance to make AI validation more accessible to more health systems. Under President Joe Biden, the Office of the National Coordinator for Health IT created transparency rules for AI made by the electronic health record platforms it certifies. However, the Trump administration is poised to sunset those rules. 

The Coalition for Health AI, an industry group tasked with creating consensus over how health AI should be vetted, had at one point pushed the idea of having independent bodies validate health AI, called assurance labs. However, those failed to take hold. 

The study highlights the need for local ongoing validation, but there is still no consensus on how to do that.  

Instead, validating artificial intelligence products, whether made by Epic or others, is largely up to a health system to figure out. Creating an infrastructure for testing and shaping AI for an individual institution’s needs is expensive and time-consuming. That means large systems, with big budgets, are best poised to adopt non-Epic AI products.   

The lack of consensus also creates problems for startups that have to go through a different process of validation with each and every client.    

Industry perspective: For now, peer-reviewed studies are the best broad validation tool for AI products. 

Martin Doerfler, former SVP of clinical strategy at Northwell Health and now an advisor to Bayesian Health, which makes sepsis and deterioration detection AI that competes with Epic’s algorithm, told Second Opinion that peer-reviewed research remains a valuable way to help health systems—especially less resourced ones—see the value of exploring alternatives to Epic’s AI.

He notes that health systems are under a lot of cost pressures. Epic, with its many tools, makes it very easy to adopt its frequently free tools. And even though doctors complain of too many false positives, IT departments may push to adopt them.   

“​​It actually becomes a real tension between clinical operations and IT,” said Doerfler. “And their budgets are running in opposite directions. You're wasting my manpower as a clinician to improve your budget on the IT side.”

Doerfler says independent peer-reviewed studies are important not just for Epic, but for startups wanting to prove they have superior products. He urges startups to shy away from putting out press releases on their own internal research and instead push clients to publish unsponsored peer-reviewed studies that evaluate their products. 

He quotes the famed author Stephen Covey, “Nothing happens faster than the speed of trust— and that's incredibly true in healthcare.”