Why Extraordinary Claims Demand
Extraordinary Proof
Ed J. Gracely, Ph.D.
© 1998, PhACT
Skeptics are typically unwilling to accept paranormal claims?-such as claims of psychic powers, human energy fields involving energies unknown to science, detection methods involving unknown forces (like dowsing), and predicting the future with cards or dreams?-unless the evidence in support of those claims is of very high quality. "Extraordinary claims demand extraordinary proof," we say. Paranormal proponents might question the appropriateness of this logic. They observe skeptics accepting some claims, such as those made by orthodox medical research, on the basis of average, reasonably good evidence, while demanding flawless, near-perfect research before paranormal claims would be accepted. Intuitively, it seems as though evidence which is "good enough" for one claim should be acceptable for other claims as well. This article discusses the issue and provides a statistical basis for skeptics' favorite line, while also pointing out its limitations.
Basic Logic
First, it is important to understand that the strength of a conclusion is a function both of the quality of the evidence provided in its support and the a priori probability of the claim being supported. Thus there can never be a single standard of "acceptable evidence" that will suffice to render every claim equally plausible. Suppose, for example, that a reasonably reliable source tells me (a) that President Clinton has vetoed legislation that places restrictions on trade with China and (b) that Newt Gingrich has switched to the Democratic party. Most people would be much more confident of the truth of the first report than of the second, even though the source is identical. The difference lies in the a priori plausibility of the claims.
A more precise formulation requires us to cast the a priori probability of a claim into the form of "odds" in its favor. A proposition with 90% probability of being true has 90 chances of being true for every 10 of being false. Thus the odds are 90 to 10, which reduces to 9 to 1. A proposition with 20% probability of being true has 20 chances of being true for 80 of being false. The odds (in its favor) are 20 to 80 or 0.25 to 1. It is more natural to translate the latter case into odds of 4 to 1 against the proposition, but the calculations require us to work with odds "in favor of" a proposition, even if they are fractional.
Pieces of evidence alter the odds in favor of a proposition by a multiplicative factor in proportion to the quality of the evidence. A good source of evidence might multiply the odds by 200. A fair one, perhaps, by 10. A negative result might reduce the odds 10-fold. So, let's say that my reliable source is good enough that his or her support for a proposition increases its odds 100 fold. This would increase the odds that the veto took place from, say, 50-50 (1-1 odds) to 100-1 in favor.
On the other hand, the probability that Newt would switch parties is very small, perhaps 1 in 10,000 (Stranger things have happened!) This is odds of about 0.0001 to 1 in favor. After my source makes that claim, the odds rise to 0.01 to 1 in favor. This is still around 100 to 1 against its truth.
So the evidence provided by my neighbor has had the same effect in each case, but the conclusion is different because of the different a priori probabilities.
Practical Considerations
The principle is clear, the difficulty lies in the application. How likely, for example, is it that homeopathy or therapeutic touch really work? Proponents argue that we need to open our minds to new possibilities and grant these systems a fairly high a priori probability (say, 50-50 odds). Then, even modest-quality evidence would make the claims quite probably true. Skeptics argue that these systems violate known laws of physics and their validity should therefore be considered remotely improbable.
An alternative I have heard suggested is to drop the extraordinary proof argument and instead to hold paranormal and alternative medicine claims strictly to the ordinary requirements of replicability and good research. This approach sounds sensible but it has a serious flaw. Skeptics are not willing to accept the plausibility of most paranormal claims unless the evidence is extremely strong. We risk being perceived (correctly) as disingenuous if we call for solid quality research, then revert to the extraordinary claims argument should it in fact appear.
In some areas of paranormal investigation, such as extrasensory perception (ESP), the research is already often better done than much orthodox scientific research, with controls and double-checks most scientists would regard as overkill. Skeptics mostly still feel that the intrinsic implausibility is so great that nothing short of airtight and well-repeated research would be sufficient to support ESP. Little or none of the existing research rises to that level, so we remain skeptical. (Some recent work has been of high quality, see Ray Hyman's article, "The Evidence for Psychic Functioning: Claims vs. Reality", in the March/April 1996 Skeptical Inquirer, pp 24-26.) Had skeptics said some 40 years ago that all we wanted was reasonable quality replicated research, we might now be having to eat our words.
The logic of the extraordinary claims argument is independent of the possibility of cheating by paranormal proponents. Many paranormal claims have a very low a priori probability. This means that potential biases and research flaws are more plausible as explanations for pro-paranormal results than is the truth of the claims. The argument would still be valid even if cheating, one possible explanation, was completely ruled out. In situations where fraud is especially likely, such as a single individual self-promoting as a psychic, we don't need the extraordinary claims argument to require tight controls against cheating. Ordinary diligence and common sense suffice to mandate exquisite and painstaking caution.
Conclusion
The skeptics' line, "Extraordinary claims demand extraordinary proof," is justifiable on probabilistic grounds, but the difficulty of determining a priori probabilities is a serious drawback. This may prevent communication with non-skeptics unless they are willing to adopt our strict standards so as to achieve general acceptance. A strict but not "extraordinary" standard of ordinary good science and replicability is risky because most skeptics would not actually believe typical paranormal claims if evidence at that level were provided.
http://www.quackwatch.org/01QuackeryRelatedTopics/extraproof.html
