How do you respond to “It worked for me”?

A regular reader of SBP describes a common reaction to the provision of science-based information, which I’ll paraphrase:

What do we do if someone is insisting on anecdotal evidence? In particular, someone who is positive that arnica, or Traumeel helped their joint issue? If someone says, “I couldn’t move my hand, or my finger, and as soon as I put it on, it was fine the next day” What do I say? I know what the science says, but what’s the most effective way to respond?

What response and strategies have you tried that have been effective in helping develop an understanding about anecdotal evidence? And what would you suggest under these circumstances? Please share your thoughts in the comments.

About these ads

28 thoughts on “How do you respond to “It worked for me”?

  1. It has limited utility for many people, but a nice response for me as a science writer is to say, “As a science writer, I have to stick to reporting what can be confirmed by a fair test.” Sometimes I throw in, “Like on MythBusters.” This has defused many an angry emailer. They may still cherish their experience and believe it constitutes proof of something I’m not acknowledging the truth of, but they usually concede that I have a higher bar to get over, leave me alone … and (this is the important part, really) the seed is planted that there might BE a higher standard for how to decide what is known.

  2. Talking out my butt, I’d tell them they were pretty lucky because those treatments rarely work. Discounting their experience will probably get you nowhere but countering with your own experience with many many more cases might help. People are wired to like stories more than data so rhetorically, it could help to express data as a story, or a set of stories.

  3. The Armchair Daddy says:

    Agreeing or disagreeing what the ‘it’ is. We can agree they are better but disagree on the cause.

  4. It’s important to acknowledge their experience – and that it’s good they did have an improvement. But I think I would probably move from that to try to explain about natural history of conditions, non-specific effects, correlation != causation and so on.

    Such an approach might well fail …

  5. Sue Kuisma says:

    or really listen – sometimes stuff works- or even better suggest a science based method/drug etc that might give relief! Some times a drink of Scotch works wonders. Guess you can tell I believe science but have a broader acceptance of other methods.

  6. And do remember the placebo effect is a real effect. People can improve for no other reason than believing that they are going to.

    My response is always that it is great that something works.

  7. Fred says:

    My friend told me he took a homeopathic jet lag remedy, so I explained the pre-scientific principles and dilutions involved in homeopathy and told him, “They’ve been selling you sugar pills – placebos – and you’ve actually been getting over your jet lag all by yourself. You’re really much tougher than you thought you were!!” He accepted my explanation and decided not to buy that product anymore.

  8. Personally, I express sincere feelings for their ailment and let them know that I am happy they found relief. I will not lend any official endorsement to their methods, but I see no point in winning an argument with a patient. Should they consult me for advice on how to treat something, I will follow standard procedure and methods for how they should proceed. If they ask what my thoughts are on some anecdotal method I’ll politely share my suspicions and concerns, but not belittle them or try to ‘win’ the debate.

  9. This happens a lot among CFS/fibromyalgia patients. The thing is, we always complain that our illness is totally inexplicable, can be fine one day and near death the next for no good reason. But people will still claim that this or that unproven treatment helped them. I normally don’t get involved, but if I have to, my response is a gentle reminder of how unpredictable our illness is, so the improvement can’t necessarily be attributed to the treatment. (And then I blog my frustrations away: http://newly-nerfed.net/2010/07/07/the-problem-with-causation/ on this topic.)

  10. This is one of the toughest issues to confront in promoting science-based medicine and challenging quackery, because persnal experiences and anecdotes are inherently deeply compelling even while being unreliable. It takes time, persistant exposure to the idea, and deep humility to acknowledge the limitations of our own experiences as a guide to what is true. I often try to offer specific examples of personal experiences that are convincing yet misleading and which are relevant to the subject at handd, and then I refer people to additional resources, such as the following:

    Books

    Don’t Believe Everything You Think: The 6 Basic Mistakes We Make in Thinking by Thomas Kida

    Why People Believe Weird Things
    by Michael Shermer.
    Becoming a Critical Thinker- A Guide for the New Millenium by Robert T. Carroll

    Blog Articles

    The Role of Anecdotes in Science-Based Medicine

    Why We’re Often Wrong

    A Budget of Anecdotes

    Why We Need Science: “I saw it with my own eyes” Is not Enough

    Videos

    Pitfalls of Thinking: Anecdotal Evidence

  11. Wouldn’t it be nice to have something a controlled trial to answer this question, too (about what works as a good reply) rather than just relying on our respective anecdotes? Not easy to do, perhaps, but I wonder if anyone has tried?

  12. Gib says:

    I say “I had such a bad back I couldn’t move. I rotated my salt shaker 180 degress in my cupboard and the next day my bad back was gone!”.
    Then I let them think it through themselves. If they object that I’m being silly, I ask them what it would take to test my “remedy” and go from there.

  13. There are probably many approaches to this which would be effective given our knowledge of the people involved. I have different friends and family which I would approach differently. However, the approach I tend to use most often if time allows is to politely and subtly turn the tables.

    I offer a counter anecdote that claims to support the exact opposite of what they are saying. Then, I ask how “they” would make sense of these two conflicting examples. When faced with opposing claims, how do we know which is true? This often engages them rather than shutting them down. If the tone is genuine and civil, they often offer really good ideas which can lead to the concepts I would have advocated to begin with.

    It’s tricky, but whenever possible I try to turn the tables and present the challenge back to them and ask for their input. Once they hit on a general area of true critical approach I then inject terms… “you’ve just described scientific consensus” or “that is basically the process of a blinded study”, or “that is what is known as peer review”, etc.

    But through it all, I try very hard to remind them that it’s ok to have been wrong, mistaken, or mislead. It’s even ok if they choose to continue their original line of thinking… just as long as they are open and informed of all the evidence. After all, we can’t make decisions for people… we can only help inform those decision and offer out best examples through out own choices.

  14. Azyrie says:

    I think I would approach it with an analogy.

    If a friend of mine has a lucky rabbit foot, and as long as he has this rabbit foot on him, he has passed all his chemistry midterms. As his friend, I may congratulate him, but I’m probably not going to tell him to stop studying for those tests just because he has a rabbit foot right?

    Similarly, a friend of mine once said that she feels better going to a naturopath. I mentioned that perhaps it is because they give much more time and attention during their consultations that makes you feel more comfortable than a relatively hurried and cold family doctor-type examination. She asked me what’s the harm if all I needed was the time of day to feel better, and I wasn’t getting it at the family doctor, why shouldn’t I go to the naturopath? I told her well, if all you needed was someone to give you the time of day, I feel obligated as a friend to tell you that you’re being ripped off (at $50 dollars a consultation) aren’t you better off getting the time of day somewhere else?

  15. Jackie says:

    Is there any chance that any of you might conceivably give some actual thought to whether a particular anecdote was more credible than the average anecdote might be?

    Here’s an example: I had suffered from chronic, near-disabling, unremitting back pain for nearly a decade. I had been to about 20 various specialists (MDs, PTs, DCs, massage therapists, etc.) and none of them offered more than a small amount of temporarily relief, at best.

    Then, I tried a given product on a whim, not expecting very much. In under a minute, 50% of the pain was GONE.

    How do you argue with that anecdote? If it were the placebo effect, wouldn’t it have kicked in long before that product (say, with one of the multitude of tried-and-discarded pain killers and physical therapy)? Are you really going to argue causation on that one?

    • Hi, Jackie. Many anecdotes are probably true stories. Yours may well be one of them, but I have no way of knowing that. It’s hard to know which stories to believe. Almost impossible. Like a witness in a murder trial, someone may have confidently identified the killer … but how can the jury really know, without corroborating hard evidence? “Reliable” witnesses have often persuaded juries to convict innocent people — at least 15% of convictions based on eyewitness testimony are later overturned by DNA evidence. Smart, honest witnesses are often just wrong.

      So when someone tells me a story, I always take it with a grain of salt, no matter how reliable the teller seems. But even if I believed every story that sounded good, I would still need corroborating evidence to justify endorsing a medical treatment. I would still have to see proof that their experience wasn’t some kind of fluke, because the only medical treatments that can be ethically recommend are the ones that are reliable and potent enough to pass a fair test.

      Rare and odd (healing) events certainly do occur, creating compelling and memorable stories told by smart, honest people. They aren’t lying, and they aren’t deluded. But their experience is, at best, a reason for researchers to dig deeper, to see if a reliable therapeutic effect can be found … not a reason to believe that the treatment is fabulous.

  16. Michael Kruse says:

    @Jackie

    It seems you have a misunderstanding of the history and progression of the scientific method. Anecdote was time and again seen to contradict the evidence of well-designed and controlled studies, which were then validated when repeated by independent investigators. Even doctors had to be dragged kicking and screaming to the table of evidence-based medicine because of their insistence on the truth of clinical experience over the truth of experiment.

    Causation itself is quite a tricky one and eventually led to the adoption of the Bradford-Hill criteria, which are science’s best answer, though not a perfect one, of determining causation. If you take even a casual glance at them, you will realise the weaknesses inherent in anecdote, no matter how powerful they can be to the individual.

    Humans suck at determining causation – we are biased by our pre-conceptions prior to the intervention, we do not intuitively possess the tools to determine the mechanism during the event, and cognitive dissonance and heuristics shape our memory of the event after the intervention.

    Before you decry, with vitriolic insistence, that anecdote = truth, perhaps you should research why we have abandoned anecdote as evidence despite the great irrational power it has over each of us.

  17. Jackie says:

    @Paul – Thank you for your considered reply. There are so many elements to this puzzle. Let me address two.

    Take the “she might be lying” component out of it. Imagine the person you trust most in the world, say an honest, reliable best friend of 20 years. Or, if even that isn’t enough, use yourself in the scenario instead of me. What would that experience tell you had it happened to you? And to raise my question again: aren’t some anecdotes more valuable than others?

    The other point would be “how much do a few good anecdotes ‘buy’?” For my part, I’ve taken to giving this product (a topical herbal) to friends and relatives whom I think could benefit from it. The only thing I ask in return is that they tell me whether it worked and, if it did, how well. Right now, I’m at about 50/50. It hasn’t helped about half of my sample. On the other hand, the other half almost uniformly rave about it. Many have said this herbal has done a better job than anything else they’ve ever tried.

    So, as you can see, my personal experience led me to run my own self-sponsored test. The results of that test have subsequently given me sufficient certainty to recommend the herbal to others (even without a free tube from yours truly to back it up).

    If I were a professional herbalist (I’m not) and I considered the combination of the cost (almost nil) and risk (almost nil) of this herbal, there would be some point at which my experiences and the experiences of others would lead me to say “I can, in good conscience, recommend this professionally.” It has been used successfully for hundreds of years and also by me personally, by my friends and family…so why not? So it doesn’t work for 50%? Who cares. Medicine isn’t an exact science, at least not with our current level of medical development. If it doesn’t work, no harm is likely to result from trying.

    And, yes, I can almost hear you saying “But there might be hidden side effects! Liver damage! Kidney damage! Chronic hiccups!” :)

    This is where one needs to realize that life is full of measured risk. Aspirin includes risk. So does caffeine. And no practicable, economically feasible study can assess all risks. As I’m sure you know, even some expensive, FDA-approved drugs have gone through “correct” clinical trials only to be pulled from the market later for wreaking all sorts of havoc.

    I guess what I’m saying is, yes, modern-day double-blind trials are a great tool. But they have their limits. Hundreds of years of use yields valuable data, too, no matter how many times someone says that “the plural of anecdote is not data.” Having both for my herbal product would be great. Failing that, I’ll take what I think is a small chance of some undefined “bad thing” happening as a result of this herbal over living a life with pain that no other remedy has ever even begun to touch.

    • Thanks for considering my reply to be considered, Jackie. ;-)

      You want to take the lying component out of it, but I cannot do that. Alas, I have less faith in people than you do. I have witnessed too many “white” lies in this context, and even comitted them myself: little rationalized truth bendings and embellishments to make my point a little better, a little more consistent with I want or hope, or to avoid acknowledgements that might be shaming or awkward. Everyone does this, routinely without even noticing, or without thinking it matters. There are a lot of people who have strong vested interests in the alleged efficacy of popular but unproven treatments. Their income is at stake, and their ego is at stake. Whenever anything is at stake, people lie in complex shades, blending their lies thoroughly with self-deception. Many anecdotes are indeed, in my opinion, hopelessly compromised by white lies ranging from the extremely complex and subtle to the amazingly dishonest.

      To answer your question as directly as possible, yes, I think “some anecdotes are more valuable than others” specifically for the purpose of helping us to decide what interesting things to investigate more carefully. I mentioned the MythBusters above. How do Adam and Jamie decide which myths to test? Suppose a drunk jerk staggers up to them in a bar and says, “I shink my toesh are magic. When I wiggle ‘em, my TV goes all funky. Swear ta god. I shink you should do an esipode. Huge fan. You rock.” Should they do a show on it? Of course not. But along comes someone they respect, someone well-trained in logical fallacies and cognitive distortions, saying “Seems like I’ve witnessed something amazing. I wonder if there’s something to it.”

      I hope they would do that show.

      Are some anecdotes better? Yes. But better for WHAT?

      No anecdote, no matter how credible, is ever good enough for anything except inspiring and justifying more careful investigation. History is riddled with examples of incredible collections of anecdotes that turned out to be mass delusions. It’s one of the most striking features of human nature.

      So how much do a few good anecdotes buy? Sorry, but they buy nothing but my raised eyebrow. If the anecdote is at odds with extensive fair testing, they will also buy a sigh. If the anecdote seems to be at odds with well-established physics and biology, those anecdotes will buy a groan.

      You wrote that “hundreds of years of use yields valuable data.” Some valuable data, sure. But if anecdotal evidence were actually reliable, then all of folk medicine would still be the best medicine available today. Stories also yield a tremendous amount of belief in silly things. Hundreds of years of anecdotal evidence has ALSO given us literally every silly belief you have ever heard of. Every naked superstition and outrageously dangerous quackery (bathing in radioactive springs, for instance) has had its zealots and converts, sometimes in shockingly large numbers. What are all those beliefs based on?

      Anecdotes!

      • Jackie says:

        Hi Paul,

        I’ve got to admit, my first impulse was to provide a hypothetical situation. It went: 1) you heard someone else say something positive about X and saw X’s beneficial effects on that person yourself, 2) by hook or by crook, that person got you to try this unproven X and it worked for you every single time you tried it, 3) more people hopped on the bandwagon and X worked for most of them in a demonstrable way. (In my hypo, you were all hard-partying frat guys who learned of a “folk” remedy for a hangover. You can only imagine the colorful details this abridged version of the story is omitting. ;) )

        So, that’s where I had been going. But I reread your post first. And, really, all I can think to say is that I’m sorry that all a good anecdote (even one that happened to you) buys is a raised eyebrow.

        I’m sorry that a cup of chamomile tea followed by a better night’s rest will be viewed as a coincidence or as the result of your body/mind deceiving you.

        I’m sorry that you’ll probably never try Rhodiola rosea and, if you do, you’ll think that extra, healthy energy boost is just happenstance, so why take it again?

        I’m sorry that you view your neighbors and even your own body as so unreliable as to be almost dismissed out of hand.

        I’m sorry that the only medicines you’ll likely ever take will be of the sort advertised on TV with an announcement like “sideeffectsmayincludeheadache,nausea,vomiting,suddenblindness,comaordeath.”

        Finally, I am genuinely sorry that I couldn’t find a way to do a better job with the “tone” of this posting. I did not mean for it to be snide or sarcastic in the least. I truly regret that someone like you — who really seems like a nice guy — is missing out due to what I would call “unhealthy skepticism.” The phrase “healthy skepticism” isn’t redundant.

        Anyway, I wish you health and joy (and, in doing so, made myself sound like a hippie. Ah, well.)

        Jackie

  18. Jackie says:

    @Michael – “Decry with vitriolic insistence”…huh?

    I’m not sure where that came from. The rest of your post was interesting and I’ll look into it.

  19. Thanks for the kind wishes, Jackie. Unfortunately, it’s going to be tough for me to experience health and joy through the haze of side effects I experience from my credulous ingestion of as many drugs as I can get some arrogant doctor to prescribe, not to mention all the fatigue from insomnia that could be cured by chamomile tea if only I would believe in my own experience, and if only I had more faith in humanity. I’m sure it’s just a matter of time before my good nature crumbles under the pressure, and I start lashing out, probably online at first. The first symptoms will probably be a pattern of making sweeping, sanctimonious assumptions about the lives of perfect strangers who disagree with me.

    Tragic!

  20. Jackie says:

    “Sweeping, sanctimonious assumptions”? I assume you mean my reply. Hopefully, *that* assumption is not itself too sweeping and sanctimonious.

    I merely paraphrased your comment and drew reasonable conclusions. I was trying to make a point but I was also trying to be nice.

    So, yes, your reply was indeed somewhere in the neighborhood of tragic, albeit not for me.

Comments are closed.