http://www.tcm.phy.cam.ac.uk/~bdj10/psi/doubtsregood.html
Unfounded
criticism of a parapsychology book in Nature
Highly regarded journal
refuses to publish correction of error in review: a case of censorship?
"Any time a reputable news organisation gives its readers or viewers details that
later turn out not to be true, they are obligated to tell the truth". -- Don Hewitt of 60
Minutes, CBS.
If your browser cannot display exponents as superscripts, click here for an alternative
version.
(illustration by David Newton, from a
recent issue of Nature)
In the highly regarded journal Nature (Oct. 23rd. 1997,
p.806), I.J. Good gives a strongly critical review (now available on the Web, courtesy of Nick
Herbert) of Dean Radin's The Conscious Universe. The Journal refused to
publish the rebuttal of the criticisms
printed below. It has also since that time refused to publish a request from the author of the book for a correction
to be noted (see also Apr. 9th. 1998 update: a grossly inadequate
'correction' is now published).
Update of Aug. 14, 1998: In its
correspondence pages (Nature 394, 413 (30 July 1998)), the
journal has now belatedly published Radin's letter
(with the omission, no doubt not surprising given the Journal's record, of his
closing remark 'I hope this note motivates readers to study the evidence for themselves'). A comeback by Good, still conforming to Rossman's strictures of 'obsolescent critique', is appended to Radin's letter. This is more than six months after the
author originally requested of the journal publication of a note concerning the
error, and more then eight months subsequent to my
own similar request. The charge of censorship on the journal's part is hardly
affected by this belated response, perhaps made only in response to widespread
complaints.
See also comment by Michael Rossman.
Should the Journal offer a response to the criticisms contained on
this web page, such a response will be posted here.
The gist of the situation: Good's review involved three main lines
of criticism:
- Effectively ignoring current
practices in the field designed to counter such criticism, Good suggests
that positive results in parapsychological research can be explained in
terms of fraud and self-deception.
- A calculation of the
'file-drawer factor' (see below) giving a result greatly
different from Radin's, implying both faulty
statistical analysis on the part of the latter and the possibility that
the existence of a sufficiently large collection of unpublished negative
results could explain away the deviations from chance found by the
analysis. But investigation of this matter reveals that Good used an
incorrect number in his own analysis. His negative conclusion is
therefore invalid.
- Quotation of a misapprehension
concerning the method of combination of results from more than one
experiment. In his book Radin quoted, without
comment, an extract from a book by Hansel containing this misapprehension.
Good mentions this in his review in order to back up his claim that Radin's statistical abilities are of a low
standard. This criticism is also misconceived. Radin comments in the letter reprinted below: 'I am well aware [of
the problem with the criticised statement] but
did not think it necessary to correct Hansel's mistake in a book aimed
towards a general readership.' The Journal increased the
confusion by adding an editorial 'clarification' that was itself mistaken,
viz. attributing some of the erroneous extract to Radin
rather than Hansel.
The journal's response to requests for a correction to be
published has been (by ordinary standards) bizarre. It has published a
correction of its own editorial error but has failed to acknowledge the far
more serious difficulty with the review that the file-drawer analysis, central
to Good's criticisms, is based on an incorrect assumption.
For a journal not to correct a known error of any significance,
for whatever reason, discredits more than just that journal. The reputation of
science is also at stake, because the integrity and validity of the scientific
enterprise demand that individual journals adhere to the accepted practices of
science. Indeed, it is hard to draw a clear line between a journal's
remaining silent about the fact that a criticism to which it has given the
authority that goes with publication in its pages is unfounded, and an
individual submitting for publication results that he or she is aware are
probably invalid.
1.Original letter, sent to Nature
Correspondence by Brian Josephson (Nov. 13th., 1997):
I.J. Good's review of Radin's survey of
the evidence for paranormal phenomena, The Conscious Universe [1], misleads by its
selective approach to parapsychological research, combined with claims of error
on the author's part that are invalid. As the book indicates, possibilities for
fraud and unintentional error are much reduced by present day techniques so
that what may or may not have happened in the case of Soal
is essentially irrelevant (unless one believes in extensive collusive cheating
among apparently reputable individuals, a hypothesis I find implausible). For
example, readings are nowadays normally not written down by the experimenter,
but recorded and analysed automatically. Such
improvements have not made the effects go away, giving one some reason to
consider that they are real.
Regarding the claims of error, Good may have been misled by
certain simplifications in the book that Radin has
explained as having been necessitated by the requirement that it be attractive
to the general reader as well as informative to the interested scientist, on
which grounds he omitted comment on the error of Hansel that Good highlights in
his review. Again, for the general reader's benefit, he described a P-value
which was actually of order 1 in 102000 simply as 'odds of more
than a billion trillion to one against chance', so that the latter number does
not approximate to the actual P-value as Good
assumed. clarification
Thus investigation shows Good's claims of mistakes on the author's
part to be unfounded. Radin has, as Good admits,
provided a well-written account of the arguments supporting the existence of
ESP, while the very frequently misconceived nature of the arguments of sceptics
may have justified giving less space to them than Good would have liked.
[1] Good, I.J., Nature 389, 806-7, 1997.
Brian D. Josephson, Cavendish Laboratory, Madingley Road, Cambridge CB3 0HE, U.K.
2. There follows the letter sent
subsequently by Radin (17 Jan 1998):
Dear Editor,
I have heard that numerous people have gained an unfavorable
impression of my book, "The Conscious Universe" (1997, HarperEdge), as a result of a review published in your
journal on Oct. 23rd, p.806. Prof. I. J. Good's review involved factual errors,
not merely differences in opinion, and I would like the journal to publish a
correction in view of this.
---------
In his review (Oct. 23rd, p.806) of my book The
Conscious Universe (1997, HarperEdge), I.J. Good
suggests, based on certain misunderstandings on his part, that something
"must be" wrong with my statistical arguments in favor of the reality
of psychic phenomena. His inability to reproduce my estimate that 3,300
unpublished, unsuccessful experiments would be required for each published ESP
card experiment (to nullify the cumulative outcome) follows from his incorrect
assumption that my words "more than", used in connection with the
cumulative odds against chance for the 186 experiments listed in Pratt et al
(1966, "ESP after 60 years", as noted in my book), could safely be
replaced by "approximately equal to." The actual p-value is
approximately 10-2000, to which application of standard methods
(Rosenthal, 1991, "Meta-analytic procedures for social research")
gives my reported figure. Secondly, Good attributed to me an inaccurate
statement actually made by the skeptical psychologist Mark Hansel. I am well
aware that combining the statistical outcomes of experiments is not the same as
combining the probabilities of independent events, but did not think it
necessary to correct Hansel's mistake in a book aimed towards a general
readership.
I am encouraged by the fact that a statistician of Good's repute
did not discover any genuine flaws in my comprehensive analysis of the empirical
evidence for psi phenomena, and I hope this note motivates readers to study the
evidence for themselves.
Dean Radin, Ph.D.
Once again, the Journal did not see fit to inform its readers that
there was a problem with its critical review.
Clarification: the P-value discussed enters into the
calculation of the so-called 'file-drawer factor', which indicates how
many unpublished studies there would need to be for each one
that was published for the significance of the results to fall
to chance. With Good's incorrect P-value, the file-drawer factor is around 16
(low enough for it to be not totally unreasonable to argue that this accounts
for the P-value calculated on the basis of the published results); with the
correct value, derived from the reference that Radin
quotes, it is around 3300, which is much harder to argue away, especially in
view of the vast total amount of experimentation that would have to be
postulated for such an explanation to work. Note that Good chose 1% as the
point where a result is significant while Radin uses
5%, which difference accounts for only a small part of the difference between
the two quoted file drawer factors.
Update
of Dec. 23rd., 1997: In response to a
request to clarify its position regarding publication of a rebuttal to the
erroneous review, the journal now argues that there is really no need for a
rejoinder to be published, since the review"was
anyway quite supportive of the book".
This characterisation may surprise those
who have read the review concerned, which after a discussion of the way people
misjudge coincidences, and a famous case of alleged fraud, goes on to discuss
some apparent 'mistakes' on the author's part (I remind the reader that, as noted
above, proper investigation shows such claims of mistakes to be unfounded),
concluding with the comment:
"So Radin's method for evaluating
the file drawer effect, whatever that method may be, must be misguided. This
conclusion largely undermines Radin's meta-analysis
which is central to his case for ESP"
True, the above quote was immediately followed by the assertion
that the book is "well written and provides a good summary of the
arguments supporting the existence of ESP". But still, if someone
writes of a thesis or paper "it is well-written, though unfortunately the
arguments in it are faulty" (and I find it rather difficult to see the
above quote as meaning anything other than 'the arguments in the book are
faulty'), what does one make of that? Is it being characterised
as a good thesis or paper because it is well-written -- or the reverse, because
the arguments are faulty? I tend to assume the latter myself but perhaps
(taking account of the reponse by Naturein this particular case) I should be less certain
than I am that the same would be the general view.
But if I am right in seeing the statement that the review was
supportive of the book as mere ad hoc and post hoc justification
for a decision that it would be hard to justify on rational and scientific
grounds, what does this mean? Is there after all, despite the intense claims
that have been made to the effect that science is concerned only with matters
of truth, a social and political dimension to science?
Update of Apr. 9th., 1998 Following an exchange of letters, the journal eventually agreed it would publish a correction. However, this correction, published in its April 9th. issue (p. 564), turned out to be merely an acknowlegement of an error introduced during the editing process (leading to a misconception of Hansel's being attributed to Radin, see Radin's letter
above). The journal is remaining silent in regard to Radin's first complaint, correction of which radically alters the validity of the whole review.See also further comment on scepticism, and letter to Nature by Michael Rossman.