Candace diCarlo
When the federal government launched the Hospital Compare initiative earlier this decade, health care officials hoped the program would serve as a sort of Consumer Reports for the sick.
By grading hospitals on how well they treated a predetermined set of common health problems, Hospital Compare, officials said, would let patients know, for the first time, which hospitals were safe—and which weren’t.
If a new study from two Penn professors is any indication, though, the effort—while clearly well intentioned—isn’t quite living up to its billing.
In a study recently published by the Journal of the American Medical Association, Penn Assistant Professor of Medicine Rachel M. Werner and Wharton Professor of Marketing and Statistics Eric Bradlow report that hospital performance in the Hospital Compare measures actually has little impact on patient mortality rates. After combing through data from more than 3,600 hospitals, the researchers concluded that while Hospital Compare has some merit, it does not necessarily provide an accurate measure of quality of care between hospitals.
The data the system uses, they say, just isn’t sufficient to do so.
Werner, who focuses her research on health care quality issues, brought the idea for the study to Bradlow, an expert in data analysis who, coincidentally, had taught Werner when she was a student at Penn. Werner’s idea—to use data analysis to link hospital performance to patient outcomes—struck a chord with Bradlow, who has long been interested in health care.
“We wondered if these measures that supposedly have the ability to allow people to compare hospitals actually predict mortality very well,” Bradlow explains. “That was the basis of the study.”
The researchers focused their attention on Hospital Compare measures related to heart attack, heart failure and pneumonia. And while they found there was some statistical correlation between good performance in the quality measures and mortality, they also noted the differences in mortality between hospitals that did well by Hospital Compare standards and those that didn’t were small. Very small.
Across all heart attack performance measures, for instance, the reduction in risk-adjusted death rates between hospitals that performed in the 25th percentile versus those that were in the 75th percentile was just 0.005 for inpatient death, 0.006 for 30-day death, and 0.012 for death at one year. Among heart failure cases, the differences were even tinier: just 0.001 for inpatient death to 0.002 for 1-year death. Because the differences are so small, Bradlow and Werner say, it’s questionable whether the Hospital Compare data is accomplishing what it’s supposed to.
If a hospital 50 miles away from a heart attack patient’s home is only a tiny bit better than the hospital around the corner, they note, it’s probably not worth it to spend an extra hour getting to the supposedly “better” hospital. Especially if the hospital is only a fraction of a percent “better.”
“As we found in the study,” Bradlow says, “the impact of these measures is really not that large.”
Bradlow and Werner are quick to point out that the Hospital Compare system is not without merit.
Though not perfect, they say the effort is an important step toward the goal of finding a true means of measuring quality of care, which most researchers agree probably does vary from one hospital to the next.
The tough part, Werner says, will be figuring out how to prove that. Werner and Bradlow are now investigating ways of doing so.
“I think the biggest question right now is how we can measure quality in a way that better captures the difference between hospitals,” Werner says.
Adds Bradlow: “What we’re saying is, ‘Let’s find better measures. We’re not saying, ‘Let’s scrap the system.’”
Originally published on February 15, 2007.
