Anecdotal evidence

Cogito ergo sum Logic and rhetoric

Key articles

Logical fallacy Syllogism Argument

General logic

Argumentum ad populum False analogy Escape to the future Sanctioning the devil One single proof Apophasis

Bad logic

Self-refuting idea Galileo gambit Two wrongs make a right Broken window fallacy Fallacy of opposition Neyman's bias v - t - e

Also known as the meta-anecdote. (Source)

“”[Bloodletting] removes coma. Mr. Henry Clymer was suddenly relieved of this alarming symptom, in the fever of 1794, by the loss of twelve ounces of blood.

—Benjamin Rush, A Defense of Blood-letting

Anecdotal evidence (also proof by selected instances, or, more pejoratively, anecdata) is use of one or more anecdotes (specific instances of an event; stories) to either support or refute a claim. The use of anecdotal evidence to draw a conclusion is like using the NBA all-star teams to estimate the average height of Americans.

Whereas anecdotal evidence is sometimes the starting point of a proper scientific investigation, it is all too often the ending point and every point of a pseudoscientific investigation. In the world of pseudoscience, an anecdote is treated as the equivalent of a peer-reviewed, double-blind, repeatable scientific experiment with consistent results.

Anecdotal evidence is often used in politics, journalism, blogs, and many other contexts to make or imply generalisations based on very limited and cherry-picked examples, rather than reliable statistical studies. A classic instance was Ronald Reagan's story of a "welfare queen" who was abusing the system, who Reagan attempted to portray as indicative of the average welfare recipient. It turned out she didn't even exist when some reporters finally decided to look for her (even if she had, this would not actually prove anything about welfare recipients generally).

Anecdotal evidence is especially vulnerable to confabulation or outright deceit.

Remember: the plural of "anecdote" is not "data".

Form

P1: X happened once when Y.

P2: (unstated) Things that happen once will happen every time.

C: X happens every time when Y.

For example, "Billy-Bob ate 3 pounds of apples a day for a year and his cancer disappeared – so can you!"

It should be clear that P2 is invalid. Take the following syllogism:

P1: Abby choked once when she ate some goat.

P2: (unstated) Things that happen once will happen every time.

C: Everyone will choke when they eat goat.

Consume that goat, and face the wrath of the Goat Overlord!

Another form:

P1: Horatio started drinking orange juice after being diagnosed with cancer and then his cancer went into remission.

P2: (unstated) There is no other plausible explanation for Horatio's cancer going into remission than the OJ.

C: Orange juice can cure cancer.

It's clear from the above that fallacies like post hoc ergo propter hoc and argument from incredulity can play supporting roles in the perception of anecdotes as convincing.

Legitimate use

In two instances, it is possible to use anecdotes non-fallaciously:

• If you use one or more anecdotes to refute the claim that there are no instances of the event that the anecdote describes. This is not fallacious because one counterexample is all it takes to prove a universal rule false, or an existential rule true.

• If you use one or more anecdotes as an example of a general rule which is already supported by a broad, comprehensive investigation (i.e., your evidence/argument does not rely on the anecdotes, they are just used to illustrate the point).

Problems

“”If you are basing your claims on anecdotal experience, then any treatment will seem to work for anything and everything.

—Steven Novella, The CAM Worldview

While anecdotes — when believed to be true, at least — are nice illustrative stories, they do not constitute good repeatable scientific evidence. This is because anecdotes only ever apply to individuals or individual experiences and are subject to the biases that this brings with it. Even without bias, the small scope of an anecdote means that it cannot be generalized to a larger group, due to variations between, say, the physiology of individual humans (and of individual pathogens), the structural integrity of rocks, or whatever the anecdote is about meaning that what works to cure an illness in one person or shatter one stone may or may not work to cure the same illness in another person or to shatter another stone of the same kind. And even if those differences could be accounted for, that still leaves the circumstances of an anecdote unaccounted for; the influence of environmental factors on a given procedure's result should not be overlooked. Anecdotes also tend to begin when a problem is first noticed, without any awareness of how long it may have been festering quietly without being observed; thus, the root cause of the problem is often unaccounted for. For all of these reasons and more, it is impossible to say that an individual anecdote is representative, and it is also impossible to actually detect the real cause of the anecdote.

For instance, with life-saving medical treatments (say, pills that reduce heart disease and subsequently lower the death rate), there are some deaths that occur whether or not the medication was taken (and not all of these deaths are even caused by heart disease). Therefore, if someone who is on the medication dies, you cannot tell if they would have died anyway without it — you can't prove that the medical intervention worked, or not, from the one case study.

It is very rare for an intervention to be, by itself, a sufficient cause of something. Rather, they tend to change the probability of a given event occurring. This means, obviously, that one can cherry pick examples that show something does or does not work, regardless of what effect it actually has. For instance, if the municipal government of some city enacts a law to reduce crime, one can find anecdotes "demonstrating" the exact opposite of the law's actual effect, whatever it may be. If it's effective: "Before the law I never had any problems with crime, but since its enactment I've been mugged once and had my house burglarized! This law is useless!" If it's not: "Before the law, I was robbed twice, but ever since the law, I haven't been robbed once! It must be working!"

Ultimately, anecdotal evidence is very prone to false positives (and is also prone to false negatives).

Regression to the mean

See the main article on this topic: Regression to the mean

If someone with a rash gets better after taking a homeopathic remedy, that doesn't mean the remedy caused the improvement, as it could simply have been an exception to the rule. Maybe it would have gotten better anyway — for example, it could have just been an allergic reaction, or a sweat-rash or anything along those effects. The anecdote doesn't tell you anything about all the people who did not get better; many other people with rashes could have taken homeopathic remedies to no effect.

Adulteration

Not to mention that these remedies can contain actual active substances, which are misleadingly listed under "non-medicinal ingredients". A "homeopathic" anti-acne cream may contain tea tree oil, which is an effective antifungal.

In addition, some people selling alternative medicines sometimes adulterate them with drugs in order to give them the desired medicinal effect and thus generate return customers. For instance, a "herbal" cold remedy may be adulterated with paracetamol and antihistamines. In these sorts of cases, the improvements may very well be caused by the drugs rather than the herbs.

Improbability

There are also anecdotes in which a person suffered from a disease for many years, and no conventional treatment was able to cure them, but they improved after applying some alternative remedy for a few days/weeks/months. For instance:

I having ministered [tobacco] to many people as well men as women, in great number, and being grieved of ten, and of twenty years they have healed old rotten sores in legs, and other parts of the body, with only this remedy to the great admiration of all men.

The implication is that the improvement cannot possibly have been a coincidence, because it was so unlikely. Of course, since millions of people use alternative medicines every day, such coincidences are certain to happen all the time to many people, even if the odds of it happening in one specific instance are low. The odds of a particular person winning the lottery are extremely low, but somebody always does, regardless. Not to mention that most people get sick many times in the course of their life, and the number of such events may number in the thousands or even tens of thousands, if one includes such mundane things as pain, cuts, nausea, or blemishes. Some improbable coincidence is bound to happen during at least one, if not several, of these many illnesses. It is worth also considering a practitioner's situation. Over the course of her career, she will see countless thousands of patients. Even if her treatments are effective, many of her patients' recoveries (and possible website testimonials) will be unrelated to her treatments, and the large number of cases almost certainly means that some of these coincidental recoveries will be extremely unlikely.

While "case studies" exist within medicine that are essentially anecdotal, these are treated mostly as curiosities or communications between doctors, rather than hard evidence on par with real trials.

Selection bias

See the main article on this topic: Selection bias

Selective reporting is the main reason that popularly disseminated anecdotes will be almost certainly unrepresentative. People tend to focus on the interesting, the unusual, or the powerfully emotive events while ignoring the mundane. After all, no one ever sold a newspaper where the front-page headline read "girl receives new vaccine; suffers zero complications and is now alive and well". People who believe in alternative medicine may, due to confirmation bias, selectively remember stories that support their belief (which they will, of course, tell everybody about), and ignore those that don't.

Selection bias can have an impressive effect in terms of PR. Let's assume 1 million people (1,000,000) decide to take some ineffective remedy to cure their cancer. Let's further assume (for the sake of argument) that only 0.1% of this million will experience spontaneous remission (the actual remission rate, for breast cancer and basal cell carcinoma at least, is closer to 20%), and that 0.3% were misdiagnosed and so do not actually have cancer. This makes for a total of 0.4%, or 4000 people. Now, 4000 people translates into a lot of testimonials, "CAM cured my cancer" blogs, Internet comments, newspaper articles, and real-life word-of-mouth, so this makes an extremely positive impression for the therapy. But the other 99.6% died and so are not around to leave any testimonials, positive or otherwise. Thus, even in a hypothetical scenario that assumes statistically very low false positive rates, the quantity of false positives is nevertheless numerically quite large. In reality, the number of past and present cancer patients relying on alternative medicine is much larger than one million, and the false positive rate is significantly higher than 0.4%, making the possibility of a given supposedly successful "cancer cure" anecdote being a coincidence even more likely.

Even if a given treatment were effective, anecdotal evidence would still be useless, because many of the cases would be the result of natural improvement, and it would be impossible to tell which anecdotes are cures and which ones are flukes. There is no way to separate the signal from the noise with only single data points suffering from selection bias.

Incompleteness and lack of follow-up

Another issue arises: anecdotes are usually not complete — they do not necessarily tell the full story. They often cover only the period of the improvement, and anything happening after the punchline or after the publication of the anecdote is not included. In addition, the anecdote-narrator may omit important factors relating to the "cure" (such as, for instance, a patient using conventional medicine together with an alternative remedy). Mark Twain's essay A Majestic Literary Fossil, which describes various forms of old-fashioned quackery, notes an anecdote:

Then follows a long account of how a dying woman, who had suffered nine hours a day with an ague during eight weeks, and who had been bled dry some dozens of times meantime without apparent benefit, was at last forced to swallow several wads of "Spiders-web," whereupon she straightway mended, and promptly got well. So the sage is full of enthusiasm over the spider-webs, and mentions only in the most casual way the discontinuance of the daily bleedings, plainly never suspecting that this had anything to do with the cure.

If, in this case, the fact that the bloodletting was discontinued gets "mentioned only in the most casual way", it is easy to imagine that many anecdotes leave out important facts like this altogether.

A cancer patient named Kim Tinkham went on The Oprah Winfrey Show with a proponent of the alkaline diet and claimed the diet had cured her, but she died of the cancer a few years later. Now, the eventual fate of Tinkham was revealed in news sources because she was famous, but if anonymous (or at least non-famous) authors of online testimonials die of a supposedly "cured" illness, it is not likely that this fact will get published on the website of the original testimonial, or possibly anywhere.

In a case-study involving spontaneous remission, a woman's diabetes went away completely, with no treatment, shortly after she woke up from a coma. In a CAM anecdote, that would be that — cure proven. But since this was a case-study rather than a testimonial, the person's post-recovery situation was observed, and two years later, a mild form of diabetes reappeared. Similarly, in an anecdotal case of a "cured" disease, the disease may later turn out to have been illusory.

Advantages of anecdotal evidence

My grandma heeded anecdotal evidence and she lived to 106.

“”When a cook in [Jean] Nicot's household nearly cut off his thumb with a chopping knife, the steward ran for the tobacco plant and bound the thumb back on; after five or six dressings of the same sort, the wound healed.

—Medicinal uses of tobacco in history

Also, Jeanne Calment, the person with the longest confirmed lifespan (122 years), smoked for almost her entire adult life and nothing bad happened to her. And besides Calment, there are many centenarians who are heavy smokers. The composer Jean Sibelius was an alcoholic and smoker for most of his life, and he lived to be 91, even saying: "All the doctors who wanted to forbid me to smoke and to drink are dead." By alternative medicine logic, this should be indisputable proof that tobacco is actually good for you. As noted by Dan King in his 1861 book Tobacco: What it is, and What it does:

“”Some are ready to contend that tobacco can have no tendency to shorten life because there are many old persons who use it. This is the drunkards' standing argument, and if it proved anything it would prove quite too much: if we are to conclude that tobacco is harmless because old men are found who use it, then by the same course of reasoning we must conclude that intoxicating liquors are harmless because drunkards sometimes live to old age; and by the same rule it might be shown that nobody had been slain or wounded in battle because some old soldiers still survive. Out of fifty persons bitten by a mad dog several may escape the hydrophobia, but such rare instances do not show that there is no danger from the bites of rabid animals. Peculiarity of temperament or idiosyncrasy may enable some constitutions to withstand for many years the pernicious effects of a virulent poison, but such exceptions to a general rule do not prove that poisons are harmless.

The power of anecdotes

Anecdotal evidence is not accepted as proper proof or scientific evidence for good reason, yet they still dominate people's thinking. This is because while facts and figures may be true, they are difficult for people to relate to. Anecdotes have convincing and strong narratives that resonate with people on a key individual level — someone telling an anecdote will lead the listener to think that it just could happen to them too. There are further biases at work. Anecdotes, even implausible ones, can contain a high level of detail. It is this detail that causes people to become involved in the story and to imagine it more clearly; a story about a young girl won't be nearly as widely spread and accepted as a story about a five year-old girl called Sally from Wisconsin who was cycling to school one day.

The biases at work in causing people to add disproportionate weight to anecdotes cause major problems during health scares. A major case in point is vaccine hysteria.

• Hard, scientific and proven (yet dry and difficult to grasp) evidence about the value of vaccines exists on the one side.
• Strong narrative stories told by concerned parents competes on the other side.

It is often advised in such circumstances that those on the side of science should also use their own anecdotes to take advantage of their persuasive power. For example, instead of giving statistics of illnesses increasing when vaccination uptake dropped, tell — in great detail — stories about specific deaths that have occurred from vaccine-preventable illnesses. Respectable scientists, however, may be against such manipulative tactics.

See also

• What's The Harm? — Anecdotal evidence against alternative medicine
• Essay:List of CAM-free anecdotes — "Miraculous" recoveries without alternative medicine.
• Pragmatic fallacy
• Magical thinking
• Exception that proves the rule
• Credulity
• Miracle
• Trepanation
• Shoehorning

External links

• Why Bogus Therapies Seem to Work, Barry Beyerstein
• Why We Need Science: “I saw it with my own eyes” Is Not Enough, Harriet Hall
• The Futility of Anecdotal Evidence, The "What's going on at RationalWiki" blog
• Homeopathy and Its Kindred Delusions, by Oliver Wendell Holmes Sr., which dwells at some length on the topic of anecdotal evidence
• Cancer: "Miraculous Recoveries", Quackwatch
• Understanding alternative medicine "testimonials" for cancer cures, Orac
• A homeopath lectures scientists about anecdotal evidence, Orac
• The key difference between alternative medicine and evidence, Orac
• Popular but Weird and Dangerous Cures, Paul Ingraham
• Your logical fallacy is anecdotal, YLFI
• Sociologist Considers Own Behavior Indicative Of Larger Trends, The Onion