A new meta-analysis in SpringerNature shows what researchers have long said about Epic’s artificial intelligence: its out-of-the-box tools don’t perform well in the real world. The insight isn’t necessarily brand new, but this study is the best research to date on the issue.
Doctors, researchers, health IT, and technology staff at Northwell Health conducted the analysis.
The study reviewed Epic’s Deterioration Index, Sepsis Model, Unplanned Readmission Model, End of Life Care Index, and Risk of No Show Patient Model. In the research, the models achieved modest real-world performance, with none surpassing an AUROC of .79 (an acceptable level of performance). Researchers’ confidence intervals for three of the models (ESM, EURM, EEOL-CI) were below Epic’s self-reported statistics.
What does that mean? In the words of researchers: “For many clinical use cases, especially high-stakes applications like sepsis detection or ICU transfer, this level of discrimination may result in excessive false positives or missed cases, potentially limiting clinical utility.”
They also say that false positives can contribute to unnecessary diagnostics, anti-microbial use, and, of course, the dreaded alert fatigue.
But the bigger takeaway, researchers wrote, is that health systems must locally validate these tools before deploying them—not an easy task.
The reason that this analysis matters is that Epic’s electronic health record, along with its various tools, is widely used by hospitals. As of the end of 2024, Epic had a 42.3 percent share of acute care systems, according to KLAS Research. Epic also makes it very easy to turn on its tools.
In a prepared statement, Epic said the meta-analysis examined studies that largely assessed its first-generation models. In second-generation models, for example, its sepsis model, it has included built-in capabilities that help fit the algorithm to the health system’s local environment, and pointed me to this study. The company agrees with the paper’s central takeaway: that local validation and customization are necessary to make its tech work optimally.
Big picture: Electronic health record companies, including Epic, are increasingly competing to be all-in-one platforms that have it all. They are competing not only with other electronic health record companies, but also with health tech companies that offer individual products, like a sepsis algorithm. As the health industry's interest in AI booms, Epic and other electronic health records have put increasing effort and money into developing those tools.