Feeling the Future

"Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect" is a paper submitted to and published in the Journal of Personality and Social Psychology by Daryl J. Bem of Cornell University that claims to offer strong evidence for the case of precognition in humans. It gained some notoriety during its "in press" phase in late 2010, with several pop science organisations and blog posts covering the experiment and attempting to understand it.

Methodology
The paper introduces some ingenious methodologies for rigorous study of certain kinds of psychic claims. The ingenious part comes from taking well-established psychological tests and reversing the order that the individual tests are administered. For instance, in one test participants are asked to type certain words, selected at random from a list, and then are asked to recall words from the full list. The result is that they are capable of recalling more of the words that they had previously typed. In Bem's research, this is turned on its head: students were asked to recall words from a list and then asked to type a random selection of them. The result being that these students recalled the words that they would later type &mdash; the ones selected at random that they had no prior knowledge of at the time of the recall test. This suggests that their minds were somehow aware of what words they would experience in the future and selected them from the full list in advance!

However, the statistical analysis and subject selection criteria are highly suspect and the statistically significant effects are likely a product of experimenter bias.

Specific criticisms
A response by E.J. Wagenmakers et al. highlights some of the major issues that call into question the validity of the analysis by Bem.


 * Bem has published his own research methodology and encourages the formulation of hypotheses after data analysis. This form of post-hoc analysis makes it very difficult to determine accurate statistical significance. It also explains why Bem offers specific hypotheses that seem odd a priori, such as erotic images having a greater precognitive effect. Constructing hypotheses from the same data range used to test those hypotheses is a classic example of the Texas sharpshooter fallacy
 * The paper uses the fallacy of the transposed conditional to make the case for psi powers. Essentially mixing up the difference between the probability of data given a hypothesis versus the probability of a hypothesis given data.
 * Wagenmakers' analysis of the data using a Bayesian t-test removes the significant effects claimed by Bem.

Any new result in any field of science requires extensive independent replication before it can be accepted as valid. This is no different for psychic powers, drug studies, or theories of physics. At least one replication of one of the tasks Bem used has failed to show significance, and hopefully others will attempt additional replications. Researcher Richard Wiseman is attempting to create a registry of replication attempts to control for publication bias.

In the end, the paper offers some promising methods but should not be taken as evidence for precognition without extensive independent replication of the results, particularly since there are significant questions about the introduction of bias into the analysis of the data.

An analysis published in Psychonomic Bulletin & Review found statistical evidence of publication bias in Bem's reported results.

In March of 2012 three independent failed replication attempts were reported in a PLoS ONE paper. The paper also noted a failed replication published in the Journal of the Society for Psychical Research. Bem tried to brush it off by suggesting that because the group attempting replication was skeptical this would negatively influence the results. According to the paper Bem presented at the Parapsychological Association 46th Annual Convention in 2003 he had actually already tested for this possibility and found that a researcher's skepticism had no such effect. The group had substantial problems trying to get their negative results published, several prominent journals refused to consider publishing a replication and another journal rejected the paper after a negative report from one referee was later confirmed to be Daryl Bem himself.

A test of Bem's "retroactive priming" published in Memory & Cognition found no evidence of any such effect.

Bem has published a detailed response to Wagenmakers et al. in Journal of Personality and Social Psychology. Bem et al respond to other critics and present a meta-analysis of 90 experiments in a paper currently under review.