Medical evidence: why process counts in health policy decisions
It would be hard to find anyone or any group today making health policy recommendations who doesn't claim their conclusions are "evidence-based." But since recommendations vary considerably in some areas, such as whether there is a health benefit to reducing dietary sodium/salt, it should be important to all of us exactly how "evidence-based medicine" (EBM) works and how it fits into the overall process of producing health care or health policy decisions.
An interview in the June 19 issue of Health Affairs magazine illuminates the limits of EBM and the shadowy abuses of the term.
The interview is a conversation between Sean Tunis who has contributed his insights working in the federal Office of Technology Assessment and the Centers for Medicare and Medicaid Services who has now founded and directs the Center for Medical Technology Policy in San Francisco and David Eddy, founder and medical director of Archimedes Inc. in Aspen, Colorado.
“Many people think that EBM is a more or less automated process that delivers a final answer. It does not,” Eddy explains. Rather, EBM is the first step in a multiple-step process. After EBM determines what the evidence is, the process requires someone to apply this information about health outcomes of the options in making preferences and value judgments leading to a decision or policy. Claiming the output is evidence-based, even if true, doesn't tell the whole story.
Eddy continues: "the main contribution of EBM is to improve the first step. It establishes principles and methods for how that step should be taken. That might seem like a relatively small contribution, but it’s huge." He points out that, prior to EBM, an authority figure or reference to the consensus of practicing physicians would be the accepted basis for decisions.
If evidence was cited at all, it would likely have been a few papers cherry-picked to support a particular point of view. So the first contribution of EBM is to change the anchor for the decision from the beliefs of experts to evidence of effectiveness. If we stopped there, that would be very big. But there are several other benefits. One is the process for creating guidelines and related policies. In the recent past, a guideline would have been set by convening a group of experts in a room and asking them what they thought the appropriate practice was. The job might be completed in a day, and the guideline issued as a one-page statement. ... There would be no systematic search of the literature, no explicit description of the evidence, no statistical analysis of clinical trials, no formal synthesis of the evidence, no written rationale, or anything like that. Today that personal, informal, incomplete, undocumented, and easily biased approach has been replaced by a much more formal, systematic, explicit, informative, and reproducible approach. ... (B)y forcing a systematic evaluation of the evidence, it shines a spotlight on the quality of the evidence. Prior to EBM, most people, including physicians, simply assumed that if something was in common practice, there must be good evidence for it. It wasn’t until we began to explicitly focus on evidence—to demand to see it—that we learned that there were huge gaps in the evidence. This revelation has had at least two beneficial effects. First, it informs everyone about the lack of evidence, so people can be more realistic in their expectations and more aware of the uncertainty in their decisions. Second, it stimulates the search for better evidence and participation in clinical trials.
Moreover, Eddy declares, EBM creates an "evidence threshold" for policy.
While EBM does not tell us whether the benefits of a treatment outweigh its risks and costs, it does tell us that nothing should be affirmatively promoted unless there is good evidence of at least some benefit. A simple belief of effectiveness is not good enough.
Tunis agreed and added:
It helps highlight those aspects of decision making where differences in the decision are not primarily disagreements about evidence, but rather reflect differences of values, preferences, and perspective. The implication of this is that further improvements in evidence-based decision making will probably require greater awareness when differences among stakeholders are primarily nonscientific issues and better approaches to arriving at acceptable decisions despite those value differences.
A primary "value judgment" explored in the interview is the absence from EBM of any consideration of cost -- certainly something taxpayers might wish was intrinsic to the Medicare and Medicaid programs, but which Tunis laments in its absence.
And I think that all of this is consistent with what you have said about how costs might differentially influence the judgments of different stakeholders. My view is that Medicare decisionmakers didn’t consider costs any more or less than the members of the ACC (American College of Cardiology)/ AHA (American Heart Association) guideline committee did, but I think that our implicit level of concern about costs was different and perhaps influenced the analysis of evidence and conclusions drawn by each organization.
The entire interview is worth a read, but the take-away message is that EBM is an explicit, disciplined process, but it only describes the evidence-base. From there, policy advocates apply their various perspectives and values in advocating policy. Let's get the process straightened out. Let's agree on the evidence base. Then, perhaps, we can move beyond the current situation on salt and health where advocates of salt reduction point loudly to evidence pertaining to an intermediate variable (blood pressure) while skeptics in medical schools and industry keep saying: not so fast in moving towards "implementation" -- let's get the evidence sorted out properly as the first step.
This kind of article should be required reading for the federal Dietary Guidelines Advisory Committee.
