Not Everything that Works Works
Thursday, September 13, 2012
Yesterday, my friend and colleague Yoni Freedhoff posted a thoughtful comment on the launch of Gary Taubes’ new non-profit organization NuSI, whose stated mission is to, “improve the quality of science in nutrition and obesity research“, and whose implied mission (according to Yoni), “is to prove Gary Taubes’ carbohydrate hypothesis of obesity is as correct as he clearly believes it to be“.
In this post, Yoni makes the important distinction between what works in a laboratory or the closed confines of a controlled clinical trial and what ‘really’ works in the ‘reality’ of clinical practice.
This issue is commonly referred to in the medical literature as the difference between ‘efficacy’ and ‘effectiveness’ – the former refers to the proof that a treatment actually works when used as determined by the researchers – the latter refers to whether or not that same treatment works when widely used in an actual clinical setting or in the general population.
As an example, there is no doubt in anyone’s mind that the DASH diet (Dietary Approach to Stop Hypertension) is amazingly effective in lowering blood pressure – to an extent comparable to, or even exceeding that of, taking a blood pressure pill. Clearly this diet is ‘efficacious’ for lowering blood pressure.
However, were I to take 100 people with high blood pressure off the street and educate them all on the benefits of following the DASH diet, I’d be surprised if 6 months later even 5% of individuals would still be following and achieving blood pressure targets on this diet – what works fine in a group of highly motivated, self-selected volunteers with all of the attention and dedication of a research team behind them, does not easily translate to routine clinical practice or a population-wide intervention.
Imagine in contrast, if the same 100 patients were told to simply take a blood pressure pill every day. Chances are that perhaps as many as 50% of individuals will still be taking their pills and controlling their blood pressure at 6 months.
Thus, even if the ‘efficacy’ of the DASH diet for lowering blood pressure was greater than that of the blood pressure pill, the pill would in fact be far more ‘effective’ as a treatment for blood pressure in real life than any diet could ever be.
The same, no doubt, could be said for anything that Taubes (or any one else) comes up with in terms of “the” solution to obesity. Irrespective of the fact, that (as pointed out by Yoni), obesity is an incredibly heterogeneous disorder with multiple causes and complex psychophysiology (which is why no one, who knows anything about obesity, expects there to ever be ‘the’ solution), the notion that whatever is found to work in trials will also necessarily prove ‘effective’ in actual practice (read: real life!) will always remain to be proven..
This is hard for non-researchers to understand and, sadly, even all too many researchers believe that once they prove ‘efficacy’ their work is done.
Diets don’t work in real life simply because most people (exceptions simply prove the point) cannot maintain them forever.
This is not a failing of the people – in the end, it is a failing of the diets.
Patients never fail treatments – it is always treatments that fail patients.
AMS
Toronto, Ontario
photo credit: osiatynska via photo pin cc
Thursday, September 13, 2012
Thanks Arya, for the nice clarification of efficacy vs effectiveness and how research findings should be translated to and interpreted by the public!
Thursday, September 13, 2012
Re: efficacy vs effectiveness, the NuSI website includes this clear statement:
“Two types of studies are necessary [referring to efficacy and effectiveness] to make scientific progress and effect societal change, and NuSI will fund both types of studies.” See the “Our strategy” tab on their site.
Thursday, September 13, 2012
Thanks Garry, good to see…
Thursday, September 13, 2012
Dear Dr. Sharma,
Perhaps I am misunderstanding the concepts of efficacy and effectiveness as applied to research (studies)—and as applied to “real life” experiences (health outcomes) of populations. If so, please forgive me if I am constructing a false analogy with my following inquiries. I intend no disrespect.
However, regarding the “efficacy” vs. the “effectiveness” of surgical bariatric treatment(s) in terms of research studies showing improved (more “effective”) health outcomes after bariatric surgery vs. potential long-term effectiveness of bariatric surgery in population-wide interventions:
1) Could a very similar argument be made about outcome efficacy regarding “highly motivated, self selected volunteers” who choose bariatric surgery and who experience intensive support from follow-up contacts with research and/or medical team members (and, who, moreover, may have access to social and or material assets which are not, typically, as readily available to general populations)?
2) If a similar argument (about bariatric surgery) cannot be validly (or fairly) advanced or applied to bariatric research results, what is the critical difference I may be overlooking?
3) If a similar argument could also be made in relation to bariatric surgery research results, then what additional evidence might be required to justify the significant expenditures that would accompany social/political policies encouraging greater access and availability (of bariatric surgery) to a wider population?
Thanks for any clarification you are able to provide!
Thursday, September 13, 2012
There is a simular problem with drug trials for mental health consumers. Because the consumers report to a health care professional weekly they report good bad and indifferent feelings so there improvement is exqued because in real life the consumers do not see a mental health care councellors weekly. The well intended patient has an issue that being how do I really know if I have depression–since they are not seeing a councellor weekly they don’t feel that they are getting better. When that is combined with the fact that a person starts feeling better on other medications fairly soon they think they should be noticing that they are feeling better.
Before you (the doctor) can treat any person that person needs to admit that they have a problem be it obeityblood pressure, or metal illness. The thoght that clinical trials doesn’t match real life is not over this lay reader’s head but it could easily go over the head of lay people.
Friday, September 14, 2012
Hopefullandfree: You are right, in that the same also applies to surgical studies. What works great in the hands of experienced surgeons at high volume centres does not necessarily work as well at lower volume inexperienced centres. Similarly, the outcomes you see in highly motivated patients, who participate in prospective trials (and come back for regular visits) may no be typical of all patients. Unfortunately, many surgical studies lose large numbers of patients to follow-up and report only on thoses that stick with the program or at least stay in contact with the surgical centres – again, these may be the patients with the best adherence, compliance, and overall outcomes. This is why the best surgical long-term data comes from the Swedish SOS study, which has almost complete follow-up of patients over 15+ years – but these results may not be typical of surgery in other healthcare systems.
AMS
Friday, September 14, 2012
Dear Dr. Sharma,
Thank you for taking the time to reply with such clarity, honesty and insight.
The Swedish health care system (as one social factor) may indeed construct atypical results; in addition, the Swedish cultural milieu in general may offer unique social characteristics that enhance long-term outcomes—perhaps, for instance, a more homogeneous population, more favorable political/economic policies, traditions that strengthen social bonds and community support, and social norms that encourage reliance on community support, etc. It’s even possible that bariatric patients in Sweden approach surgery with unique, culturally-encouraged (socially constructed) motivations for change.
It would be very interesting to study any available research which may be available (or in the works) to analyze these kinds of social and cultural variables.
At the very least, it all makes me wonder if the research literature offers cross cultural comparisons with other surgical outcomes (analyzing, for instance, differences in outcomes for heart surgery in Sweden vs. other countries.)
Thanks again for leading a most provocative discussion!
Saturday, September 15, 2012
Absolutely, healthcare is delivered in a complex system of societal values, priorities, beliefs and cultures. I would be naive to assume that these differences in settings do not affect effectiveness (not efficacy!). As I noted in previous posts, medications only work when ou take them. Unfortunately, whether or not people actually take their meds depends on all of the above, culture, beliefs, priorities, affordability, and countless other factors. Much of this lies well beyond the confines of the health system (even outside the confines of public health or health promotion). As you point out, what works for Swedes may not work for the Southern United States or India.
AMS