Essay:The Futility of Anecdotal Evidence

There is a revised version of this essay here.

Jean Sibelius lived to 91 just by following two simple tips. Interested? Read on...

In the 19^th century, the French doctor Pierre Charles Alexandre Louis invented the "numerical method", a way of statistically analyzing the effects of medical treatments. It was the precursor to modern clinical trials. Louis applied this method to a certain therapy in use at the time, and found that the therapy, contrary to traditional wisdom, did not actually help. The therapy in question had been used for thousands of years, as a sort of panacea, and doctors at the time reacted negatively to Louis' research. How could it be wrong if it had thousands of years of clinical observation behind it? Not to mention that these doctors all had positive experiences with the treatment, and were not inclined to trust some abstract numbers over all the cures they witnessed.

A few decades before Louis published his findings, an American physician named Benjamin Rush published a book in support of the said therapy, in which he recounted various anecdotes. He personally saw that the therapy could cure coma, poisoning, and fever. For instance:

[The therapy] removes coma. Mr. Henry Clymer was suddenly relieved of this alarming symptom, in the fever of 1794, [after being treated].

I [treated] my eldest daughter when she was but six weeks old, for convulsions brought on by an excessive dose of laudanum given to her by her nurse; and I [treated] one of my sons twice, before he was two months old, for an acute fever which fell upon his lungs and bowels. In both cases, life appeared to be saved by this remedy. I could mention many more instances in which it has snatched from the grave, children under three and four months old, by being used from three to five times in the ordinary course of their acute disease.

A boy of Mr. John Carrol owes his recovery from the small-pox to [the treatment], at a time when nearly every pock on his arms and legs had a purple appearance. Louis XIV was [treated] five times in the small-pox, when he was but thirteen years of age, and thereby probably saved from the grave, to the great honour and emolument of the single physician who urged it against the advice of all other physicians of the court. Dr. Cleghorn mentions a single case of the success of [the therapy] in the petechial small-pox. His want of equal success afterwards, in similar cases, was probably occasioned by his [treating] too sparingly, that is, but three or four times.

In the month of June of 1795, I visited Dr. Say in a malignant fever, attended with pleuritic symptoms, in consultation with Dr. Physick. An acute pain in his head followed six successive [treatments]. After a seventh [one], he had no pain. His fever soon afterwards left him. In thus persevering in the use of a remedy, which, for several days, appeared to do harm, we were guided wholly by the state of his pulse, which uniformly indicated, by its force, the necessity of more [treatment]. [Note the similarity of this explanation to the "temporary worsening" and "detox symptoms" posited by homeopaths and naturopaths.]

The anecdotal objections of Louis' critics were no doubt similar. Eventually, though, Louis' work prevailed, medicine became science-based, and the therapy was mostly discarded, except for a very small number of conditions. Now, the therapy lives on in Ayurvedic medicine, traditional Chinese medicine, Unani, and Hildegard medicine (which was apparently revealed to St. Hildegard in visions from God). In these systems, the treatment is claimed to be good for a large number of conditions. According to Ayurvedic medicine, the therapy should be done seasonally in order to detoxify the body, though mainstream medicine still asserts it is harmful. What is the therapy?

Bloodletting.

There is a similar case relating to herbal medicine. A certain herb had been used for centuries in indigenous American medicine. The aboriginals used it for many, many, things, including wounds, colds, burns, asthma, pain, and teeth whitening, both internally and externally. When the Europeans came, they took a great liking to it, going so far as to call it the “Holy Herb” and "God's Remedy". It was widely thought to be a cure for many illnesses, including cancer, asthma, and the common cold.

Jeanne Calment, the person with the longest confirmed lifespan, used it for 98 years, and lived to be 122; she stopped taking it when she was 119, and only three years later, she passed away. Jean Sibelius took it, and he lived to be 91, despite the fact that he was an alcoholic (or because of it; you could use anecdotal evidence to argue either way). His doctors told him he would die if he didn't stop taking it; later in life he said that he outlived all the doctors who told him not to take it. Arthur Rubinstein, another famous musician, also took it, and he lived to be 95. The plant was Marcel Proust's favorite asthma remedy, and he felt relief after taking it.

The healing powers of this plant were common knowledge, and a very large number of the packages in which this herb was sold advertised the many things it could "cure". But then came the clinical trials. It turned out that it was not actually an effective remedy; in fact, it was quite dangerous. The government made herb sellers take down the medicinal claims. Not only that, it also forced them to say that it was harmful — right on the box! (How diabolical!)

Here you have all the makings of a CAM wonder-cure: Traditional use, anecdotal evidence, celebrity endorsement, and a conspiracy theory. So what was the plant?

Tobacco.

Alternative medicine: Quackery, or life-saving cure? Tonight at 11.

In the tobacco example above, I used CAM-style cherry picking to support the conclusion that tobacco is good for you. Anecdotes are useless for determining whether something is good for people, because they could very well be flukes. Sure, Jeanne Calment and Sibelius were long-lived, and you can find a very large number of smokers who live to a ripe old age. But though there may be a large quantity of long-lived smokers, the question is whether the percent of smokers who live to be, say, over 80, is larger than that of a non-smoker control group. Many people swear by goldenseal and echinacea for cold prevention, and say they haven't had colds for a number of years. But there are also many people who do not believe in CAM or goldenseal and yet do not get colds either; for instance, Harriet Hall, a noted opponent of alternative medicine, hasn't had a cold in 16 years. Cherry picking anecdotes from either group proves absolutely nothing. (Indeed, you could even use anecdotes to "prove" that smoking and drinking are actually good for you!) Only a randomized, double-blind, placebo-controlled study can determine whether goldenseal can actually prevent colds. Anecdotal evidence can be used to "support" anything, no matter how harmful or ridiculous, such as drinking, smoking, bloodletting, and all sorts of diets, including high-fat, low-fat, meat-free, high-meat, low-sodium, high-sodium, low-carb, and alkaline ones. Anecdotally minded proponents of these diets can argue back and forth about which one is the best, but without any scientific studies, such arguing is quite pointless.

Alternative medicine's uncritical acceptance of anecdotes and contempt for statistics and placebo controls is unreasonable, but perhaps understandable. People whose conditions improve or disappear after the use of a supplement ("after" being an arbitrary amount of time) will naturally attribute the improvement to the supplement. After all, it seems obvious that the treatment cured the illness, and personal experience seems much stronger than some soulless numbers. Unfortunately, common sense is wrong here. Many self-limited conditions naturally have high and low points; when one of these waves of improvement (known as regression to the mean) coincides with the use of an alternative treatment, it will appear as if the treatment actually did something, when it was actually just a coincidence. For example: someone had an awful stomach infection, and they took a certain supplement on the day they got sick. The next day, they felt absolutely great. This anecdote proves nothing because it is possible that the particular illness in their case was the 24-hour flu. Even if it wasn't, illnesses are naturally unpredictable and often go away suddenly. This anecdote says absolutely nothing about all the other people who tried it and did not get better, and so, like all testimonials, it's an example of selection bias. Even supposing you had some magic crystal ball and were able to find out exactly how many of the people who tried the herb got better after taking it, that information would not tell you if the improvement was due to the pill, or if those people would have gotten better anyway. Natural recovery and other factors such as misdiagnosis virtually guarantee that a favorable anecdote will be a false positive. Cases of alternative medicine allegedly curing cancer are usually due to misdiagnosis or spontaneous remission.

Not to mention that the bar for what constitutes a favorable anecdote is extremely low. For example, in the absence of any objective improvement, the patient will almost certainly feel better because of the placebo effect, which, because it causes the body to release analgesic substances such as endorphins, can be quite powerful. Marcel Proust may have felt relief from smoking tobacco, but it was the placebo effect that did it, not the tobacco itself. Indeed, the tobacco almost certainly made his asthma objectively worse. But in much of alternative medicine, any improvement is good enough. Even if the patient died, that is still an acceptable anecdote as long as the person felt somewhat better at some point during the illness — which, thanks to the placebo effect, is, again, almost guaranteed to happen. No need to actually compare the overall mortality to that of conventional treatment. Single cases are good enough. Not only are they good enough, but they are actually much better than those close-minded clinical trials invented by greedy, insensitive allopaths who just want to kill people with their expensive poisons.

It's this kind of thinking that keeps useless treatments like bloodletting alive in alternative medicine. "Oh, who cares what your trials say? I had a patient who came out of a coma after he lost 12 ounces of blood! How is that not evidence?" Well, for one thing, how did Rush know it wasn't just a fluke? How many coma patients did he treat in total before he wrote the book? How many of those came out of the coma after bloodletting, and how many didn't? How did the mortality rate of coma patients treated with bloodletting compare with that of a non-bled control group? Maybe that coma patient was the only one of Rush's who got better after being bled, meaning that the story was just an example of selection bias. Incidentally, selection bias is the reason why anecdotes about alternative medicine are skewed in favor of it — the people who died as a result of forgoing conventional therapies and using CAM are not around to tell their stories, and so, with the exception of sites like What's The Harm?, there is a relative lack of negative testimonials. For every person supposedly cured by alternative medicine, there are many others that died, because they chose CAM instead of scientifically proven treatments.

Because CAM clings to anecdotes and is almost totally immune to advances in science, many harmful remedies are currently in use in alternative medicine. For example, traditional Chinese medicine uses arsenic, lead, and mercury compounds: lead oxide is supposed to be good for a variety of conditions, including eczema, intestinal parasites, and, strangely enough, detoxification (whatever that means); cinnabar (mercurous sulfide) is supposed to be good for insomnia; realgar (arsenic sulfide) is used for parasites and sore throats, among many other things; and calomel is used for longevity. (The fact that the remedies contain lead and mercury doesn't necessarily make them useless — after all, the dose makes the poison. But the point is that they are completely unproven.) Body parts of endangered species are also used in TCM; rhino horn is thought to be a panacea that can cure pretty much everything, even though it's been shown to be useless. A significant part of rhino poaching is due to demand caused by TCM use of rhino horn. Ayurvedic medicine uses not only lead and mercury, but also leeching — which is claimed to be a cure for baldness!

In the world of alternative medicine, treatments are frequently considered panaceas; that is, they are thought to be good for a wide variety of unconnected conditions. This is because natural improvement, misdiagnosis, and the placebo effect occur in virtually all conditions (even cancer) and can give the appearance of a treatment (such as bloodletting) being a cure-all, thus leading to all sorts of inappropriate medical indications. Occasionally, traditional medicine stumbles onto something useful, and proponents of CAM will seize the opportunity to say, "Hey, traditional medicine brought us drug x, so CAM must be good." This is problematic for two reasons: a) it ignores all the medical claims for the herb that were wrong, and b) the herb may contain only a very small and ineffective amount of the medicinal substance, while the isolated drug will be 100% pure (think taxol). Because herbs are usually said to be good for a large number of diseases, the fact that just one of those claims happened to be real does not validate the many other uses which were completely wrong. Since anecdotal evidence can pretty much be used to support any and every conceivable medical claim, it's inevitable that some of those claims will happen to be right just by chance. It's a classic example of shoehorning.

But how could traditional healers not know about the risks? After all, you'd think it would be obvious that something causes kidney failure or some other serious problem. If something causes a serious side effect in only 5% (or some other relatively small percent) of people exposed to it, then it is impossible to tell, using only personal observation, that the remedy actually caused the side effect. Even if the herbalist had a suspicion that the medicine did it, then it would be impossible to narrow it down, as 95% of people taking it will be fine, and people usually take multiple herbal remedies. How do you know which one it is? Individual plants within the same species can vary considerably in their chemical makeup because of various environmental factors, such as the climate where they were grown, and the quality of the soil, among other things; one plant might cause kidney failure, and another might not. Not to mention that the toxicity may manifest itself only after several years. Cancer resulting from carcinogen exposure usually occurs only after a long time, and carcinogenic substances don't always cause cancer, so observation is useless here, too. Jeanne Calmet smoked for most of her life, and she didn't get cancer, but that doesn't mean tobacco is safe.

Science-based medicine evolves, gets rid of ineffective remedies, and constantly improves. Anecdote-based medicine does not. TCM continues to use lead, mercury, and arsenic for exactly the same reason people use homeopathy: the belief that personal experience trumps science. If a drug is shown to be useless, science discards it (the whole process takes a few years, at most). But no amount of evidence will convince a CAM true believer that their favorite remedy doesn't work. If it hadn't been for Pierre Charles Louis Alexandre, we would all still be using mercury, because of anecdotal evidence's complete inability to detect long-term toxic effects.

Humanity trusted anecdotal evidence for thousands of years, and what did we have to show for it? Bloodletting, lead, mercury, birthwort, medicinal tobacco, many other ineffective "remedies", and maybe a few better-than-placebo painkillers.

By contrast, during the small period of almost two centuries that statistics-based medicine has been around, we've eradicated smallpox, discovered insulin, dramatically improved our lifespans, achieved 90% cure rates for things like testicular cancer, and made measles, mumps, rubeolla, and polio practically a thing of the past. Sure, modern medicine is flawed — no one is saying it's perfect — but there is no comparison.