Nature critique, rebuttal notes:
http://www.tcm.phy.cam.ac.uk/~bdj10/psi/doubtsregood.html
Bem vita:
http://psych.cornell.edu/sites/default/files/djb5_vita.pdf
Radin bio:
http://www.deanradin.com/NewWeb/bio.html
Dean Radin, PhD, is Chief
Scientist at the INSTITUTE OF NOETIC SCIENCES (IONS) and Volunteer Faculty in the Department
of Psychology at Sonoma State University. His original career track as a
concert violinist shifted into science after earning a BSEE degree in
electrical engineering, magna
cum laude with honors in
physics, from the University of Massachusetts, Amherst, and then an MS in
electrical engineering and a PhD in psychology from the University of Illinois,
Urbana-Champaign. For a decade he worked on advanced telecommunications R&D
at AT&T Bell Laboratories and GTE Laboratories. For over two decades he has
been engaged in frontiers research on the nature of consciousness. Before
joining the research staff at IONS in 2001, he held appointments at Princeton
University, University of Edinburgh, University of Nevada, Interval Research
Corporation, and SRI International.
He is author or coauthor of
over 200 technical and popular articles, a dozen book chapters, and three books
including the award-winning The
Conscious Universe (HarperOne,
1997), Entangled Minds (Simon & Schuster, 2006), and the
2014 Silver Nautilus Book Award winner, SUPERNORMAL (Random House, 2013).
http://en.wikipedia.org/wiki/Rational_Wiki#RationalWiki
In April 2007, Peter Lipson, a
doctor of internal
medicine, repeatedly attempted to edit Conservapedia's article on breast
cancer to include
evidence arguing against Conservapedia's claim that abortion was a major cause of the disease.
Conservapedia administrators "questioned his credentials and shut off
debate".[14] Several editors whose accounts were blocked
by Conservapedia administrators, including Lipson, started another website,
RationalWiki, a sometimes satirical and sometimes serious wiki website with
articles written from a secular, progressive perspective.[citation needed] RationalWiki's self-stated purpose is to
analyze and refute "pseudoscience", the "anti-science movement", and "crank
ideas", as well as to conduct "explorations of authoritarianism and fundamentalism"
and explore "how these subjects are handled in the media."[78]
According to an article
published in the Los
Angeles Times in
2007, RationalWiki members "monitor Conservapedia. And—by their own
admission—engage in acts of cyber-vandalism."[14]
http://rationalwiki.org/wiki/Feeling_the_Future
The paper introduces some
ingenious methodologies for rigorous study of certain kinds of psychic claims. The ingenious part comes from taking
well-established psychological tests and reversing the order that the individual tests
are administered. For instance, in one test participants are asked to type
certain words, selected at random from a list, and then are asked to recall
words from the full list. The result is that they are capable of recalling more
of the words that they had previously typed. In Bem's research, this is turned
on its head: students were asked to recall words from a list and then asked to type a random selection of
them. The result being that these students recalled the words that they would later type - the ones selected at random
that they had no prior knowledge of at the time of the recall test. This
suggests that their minds were somehow aware of what words they would
experience in the future and selected them from the full list in advance!
However, the statistical analysis
and subject selection criteria are highly suspect and the significant effects
are likely a product of experimenter
bias.
A response by E.J. Wagenmakers et al. highlights some of the major issues
that call into question the validity of the analysis by Bem.[2]
§ Bem has
published his own research methodology and encourages the formulation of
hypotheses after data analysis. This form of post-hoc analysis makes it very difficult to determine
accurate statistical significance. It also explains why Bem offers specific
hypotheses that seem odd a
priori, such as erotic images having a greater precognitive effect.
Constructing hypotheses from the same data range used to test those hypotheses
is a classic example of the Texas sharpshooter fallacy
§ The paper
uses the fallacy of the transposed
conditional to make
the case for psi powers. Essentially mixing up the difference between the
probability of data given a hypothesis versus the probability of a hypothesis
given data.
§ Wagenmakers'
analysis of the data using a Bayesian t-test removes the significant effects
claimed by Bem.
Any new result in any field of
science requires extensive independent replication before it can be accepted as
valid. This is no different for psychic powers, drug studies, or theories of physics.
At least one replication of one of the tasks Bem used has failed to show
significance,[1] and hopefully others will attempt additional
replications. Researcher Richard
Wiseman is
attempting to create a registry of replication attempts[3] to control for publication
bias.
In the end the paper offers some
promising methods but should not be taken as evidence for precognition without
extensive independent replication of the results, particularly since there are
significant questions about the introduction of bias into the analysis of the
data.
An analysis published in Psychonomic Bulletin & Review found statistical evidence of
publication bias in Bem's reported results.[4]
In March of 2012 three
independent failed replication attempts were reported in a PLoS ONE paper.[5] The paper also noted a failed replication
published in the Journal of the Society for Psychical Research. Bem tried to brush
it off by
suggesting that because the group attempting replication was skeptical this would negatively influence the results.[6] According to the paper Bem presented at the
Parapsychological Association 46th Annual Convention in 2003 he had actually
already tested for this possibility and found that a researcher's skepticism
had no such effect.[7] The group had substantial problems trying to
get their negative results published, several prominent journals refused to
consider publishing a replication and another journal rejected the paper after
a negative report from one referee later confirmed to be Daryl Bem himself.[8]
A test of Bem's "retroactive
priming" published in Memory
& Cognition found no
evidence of any such effect.[9]
§ Journal of Personality
and Social Psychology, 100, 407-425. (Published version, March 2011)
Unpublished failed replications:
§ Jeff
Galak and Leif D. Nelson (2010). A
Replication of the Procedures from Bem (2010, Study 8) and a Failure to
Replicate the Same Results
§ Steve
Snodgrass (2010). Examining
Retroactive Facilitation of Recall: an Adapted Replication of Bem (2011, Study
9) and Galak and Nelson (2010).
§ Galak,
LeBoeuf, Nelson and Simmons (2012). Correcting
the Past: Failures to Replicate Psi.
§ 2011
panel discussion with Daryl Bem, Jonathan Schooler and Samuel Moulton:
Moulton's presentation describes a failed replication of experiment 6 and
issues with Bem's reported results. Bem provided Moulton with the software and
stimuli to conduct a replication in 2004. Moulton used a greater sample size,
but found an insignificant effect in the opposite direction to Bem. (Moulton's
slides)
§ Wagenmakers
et al. preliminary results from failed replication of experiment 1. [1](Dutch)
Bem's 2003 and 2005 preliminary
reports on these experiments at the Parapsychological Association Convention:
§ Bem, D.
J. (2003). Precognitive
habituation: Replicable evidence for a process of anomalous cognition.
Proceedings of Presented Papers: The Parapsychological Association 46th Annual
Convention. 6-20.
§ Bem, D.
J. (2005). Precognitive
aversion. Proceedings of Presented Papers: The Parapsychological
Association 48th Annual Convention. 31–35.
Bias:
New Scientist:
“Extraordinary claims
don't come much more extraordinary than this: events that haven't yet happened
can influence our behaviour.
Parapsychologists have
made outlandish claims about precognition – knowledge of unpredictable future
events – for years. But the fringe phenomenon is about to get a mainstream
airing: a paper providing evidence for its existence has been accepted for
publication by the leading social psychology journal.
What's
more, sceptical psychologists who have pored over a preprint of the paper say they can't find any significant flaws.
"My personal view is that this is ridiculous and can't be true," says Joachim Krueger of Brown
University in Providence, Rhode Island, who has blogged about the
work on the Psychology Today website. "Going after the
methodology and the experimental design is the first line of attack. But
frankly, I didn't see anything. Everything seemed to be in good order."
Wagenmakers, et al.
Why Psychologists Must
Change the Way They Analyze
Their Data: The Case of Psi
http://dl.dropboxusercontent.com/u/1018886/Bem6.pdf
Abstract
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysis was partly exploratory, and that one-sided p-values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem’s data using a default Bayesian t-test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s p-values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
In other words: using a different style of Bayesian t-test shows weak to non-existent when being conservative.
One way to interpret this is: the results that Bem is claiming cannot be true, so the authors’ recommendation is that the entire field of psychological testing must change the way it runs its tests on data. Or, although different forms of Bayesian testing are disputed, because these results cannot be true we must use the most conservative form of disputed Bayesian analysis.
Here is good work, making a valid attempt to replicate one of Bem’s results, and failing to do so. Obviously these researchers take Bem’s work seriously.
A replication of the procedures from Bem (2010, Study 8) and a failure to replicate the same results.
Jeff Galak
Carnegie Mellon University
Leif D. Nelson
University of California, Berkeley
“Recently, Bem (2010) published an extremely thought-provoking article demonstrating the existence of precognition, “the conscious cognitive awareness… of a future event that could not otherwise be anticipated through any known inferential process.” Through nine meticulously constructed experiments, using a range of tasks, Bem finds consistent support for the idea that people have precognitive abilities. As Bem suggests, the purpose of the paper was not exclusively to simply report evidence relevant to precognition, but also to develop procedures “that can be replicated by independent investigators (p. 3)”. We sought out to replicate one of those procedures.”
Ritchie, Wiseman, and French, response to Bem’s criticism of their failed attempt to repeat his results:
Ritchie, Wiseman, and French, response to Bem’s criticism of their failed attempt to repeat his results:
Here is more good work, in this case responding to Bem’s comments. Note the clear statements, and also that the researchers took the time to attempt to repeat the results. As they state, it should not matter whether the researchers are “skeptics or believers” in a well-run experiment.
We thank Professor Bem for
his comments on our article, and for his kind assistance in our attempt to
replicate his experiment.
Bem is, of course, correct to note that our
study is not conclusive evidence that the 'Retroactive Facilitation of Recall'
(RFR) effect does not exist, and that our study is not the only attempt at
replicating this effect. However, Bem criticises us for not citing some of the
other attempts. In our article, we referenced all replications of the RFR
effect of which we were aware that had been published in peer-reviewed journals
(this included only one other replication attempt, which failed [1]). A
meta-analysis of the combined effect of the published and unpublished studies
is currently in progress. Bem does not mention other studies published online
at the Social Science Research Network [2, 3] which differ somewhat in their
methodology as they were carried out online, but nonetheless both fail to find
significant RFR effects. Since many other unpublished experiments may exist, we
feel our focus on the peer-reviewed literature was justified.
Next, Bem argues that we did not in fact carry
out three separate attempts to replicate his effect, since our sample size
equalled that of his Experiments 8 and 9 combined. Both of these experiments
focused on the RFR effect, but Experiment 9 had a critical methodological difference
from Experiment 8: more post-test practice of the words. The evidence for the
importance of this methodological change is that it boosted the effect size of
the procedure from d = .19 (Experiment 8) to d = .42 (Experiment 9). For this
reason, we do not believe it is correct to consider Experiments 8 and 9 as the
same experiment (indeed, Bem does not do so in his original paper).
Bem then points to the example of the 'mere
exposure effect', which across its history has had a mixture of failed and successful
replications. As Bem himself anticipates, the analogy here is flawed, since the
experiments on the mere exposure effect have employed widely-differing
procedures, outcomes, and samples, while our replication of Bem's procedure was
exact.
We do not doubt the importance of the
'experimenter effect' in psychology. However, we do not believe Bem gives a
persuasive rationale for its existence in our replication attempts. Firstly,
since the random selection of 'practice' and 'control' words in this particular
experiment occurs after the participant's completion of the memory test, it is
not clear how expectations in one direction or another would influence the
results. Bem gives no potential mechanism for this effect, and we cannot
imagine one (except one involving anomalous psi-related processes, which would
of course beg the question).
Secondly, Bem makes reference to the
investigations of the parapsychological experimenter effect, co-authored by one
of us (RW). While he is correct in his summary of the results of these
experiments, he does not make reference to the quality of each study. The final
study, in which neither believer nor skeptic found significant results [4], was
by far the largest and best-controlled of the experiments. Given this fact, we
leave it to the reader to decide whether these experiments provide convincing
evidence of strong skeptic-believer experimenter effects in parapsychology.
However, even if the Wiseman-Schlitz experiments
were conclusive evidence of experimenter effects, there is still no analogy
from them to Bem's experiments. This is because, in the Wiseman-Schlitz
experiments - which investigated 'the sense of being stared at' - the
experimenter was heavily involved in the procedure, being the 'starer' who
directed their gaze at participants, who were to report if/when they sensed
that gaze. Bem's experiments, on the other hand, are run and scored entirely by
the computer program, with experimenters only greeting and debriefing
participants. In addition, the staring experiments measured an
emotional/perceptual outcome, which is quite different from the basic word
memory test involved in Bem's experiment.
Finally, it is worth pointing out, as we do in
our paper, that only one of us (SJR) personally ran their replication attempt;
two of our replication attempts were run by research assistants. Thus, the
participants in those samples did not encounter either of the two of us who
are, to quote Bem, 'well known as psi skeptics' (RW and CCF).
Bem's study was viewed as a wiping clean of the
slate in parapsychology; his ingenious experimental paradigms were based on
well-known psychological effects, were simple to administer, and did not
involve the experimenter in the procedure in any substantial sense. We believe
that, if the effects are real, they should be easy to replicate either by
believers or skeptics, and we do not find Bem's arguments to the contrary to be
compelling.
Stuart J. Ritchie
Richard Wiseman
Christopher C. French
Tressoldi and Di Risio
controversy / Bayesian testing:
http://deanradin.com/evidence/Rouder2013Bayes.pdf
Psychol Bull. 2013 Jan;139(1):241-7.
doi: 10.1037/a0029008.
Rouder JN1, Morey RD, Province JM.
Psi phenomena, such as mental telepathy,
precognition, and clairvoyance, have garnered much recent attention. We
reassess the evidence for psi effects from Storm, Tressoldi, and Di Risio's
(2010) meta-analysis. Our analysis differs from Storm et al.'s in that we rely
on Bayes factors, a Bayesian approach for stating the evidence from data for
competing theoretical positions. In contrast to more conventional analyses,
inference by Bayes factors allows the analyst to state evidence for the
no-psi-effect null as well as for a psi-effect alternative. We find that the
evidence from Storm et al.'s presented data set favors the existence of psi by
a factor of about 6 billion to 1, which is noteworthy even for a skeptical
reader. Much of this effect, however, may reflect difficulties in
randomization: Studies with computerized randomization have smaller psi effects
than those with manual randomization. When the manually randomized studies are
excluded and omitted studies included, the Bayes factor evidence is at most 330
to 1, a greatly attenuated value. We argue that this value is unpersuasive in
the context of psi because there is no plausible mechanism and because there
are almost certainly omitted replication failures.
(PsycINFO Database Record (c) 2013 APA, all
rights reserved).
·
Testing the Storm et al. (2010) meta-analysis using Bayesian and
frequentist approaches: reply to Rouder et al. (2013). [Psychol Bull. 2013]
·
Meta-analysis of free-response studies, 1992-2008: assessing the noise
reduction model in parapsychology. [Psychol Bull. 2010]
Response to (above) Rouder, et al.:
http://deanradin.com/evidence/Storm2013reply.pdf
Psychological
Bulletin © 2013 American Psychological Association
2013, Vol. 139, No. 1, 248–254 0033-2909/13/$12.00 DOI: 10.1037/a0029506
Testing
the Storm et al. (2010) Meta-Analysis Using Bayesian and
Frequentist
Approaches: Reply to Rouder et al. (2013)
Lance Storm
University
of Adelaide
Patrizio E. Tressoldi
Universita`
di Padova
Jessica Utts
University
of California, Irvine
Rouder,
Morey, and Province (2013) stated that (a) the evidence-based case for psi in
Storm, Tressoldi,
and
Di Risio’s (2010) meta-analysis is supported only by a number of studies that
used manual
randomization,
and (b) when these studies are excluded so that only investigations using
automatic
randomization
are evaluated (and some additional studies previously omitted by Storm et al.,
2010, are
included),
the evidence for psi is “unpersuasive.” Rouder et al. used a Bayesian approach,
and we adopted
the
same methodology, finding that our case is upheld. Because of recent updates
and corrections, we
reassessed
the free-response databases of Storm et al. using a frequentist approach. We
discuss and
critique
the assumptions and findings of Rouder et al.
Keywords:
Bayesian analysis, ESP,
ganzfeld, meta-analysis, null hypothesis significance testing, parapsychology
Rebuttal of
Good:
http://www.tcm.phy.cam.ac.uk/~bdj10/psi/doubtsregood.html