Peer review

From RationalWiki
(Difference between revisions)
Jump to: navigation, search
(Allegations of Bias in Peer Review)
(Possible problems)
Line 16: Line 16:
 
#The peer-review process doesn't involve ''replicating'' experiments or studies in order to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work, as it is presented, is suitable for publication.  Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
 
#The peer-review process doesn't involve ''replicating'' experiments or studies in order to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work, as it is presented, is suitable for publication.  Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
  
In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the editors of the scientific journal a work has been submitted to. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with publication, as the article or paper remains available for all interested parties to view - if someone tries and fails to replicate the results, or finds that they contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review  process remains the most objective and qualified way to assess scientific work that has ever been developed.
+
In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the editors of the scientific journal a work has been submitted to. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with publication, as the article or paper remains available for all interested parties to view - if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review  process remains the most objective and qualified way to assess scientific work that has ever been developed.
  
 
==Peer review as a legal benchmark==
 
==Peer review as a legal benchmark==

Revision as of 16:59, 17 September 2009

Information icon.svg Cover Story
This article is, among others, randomly included on the Main Page.
Please keep this in mind and be sure that your edits are of the quality that this implies.
Its front-page abstract can be found here.

Peer review is the process of subjecting scientific work to review by other experts in the field before publication. The process is designed to ensure that the work meets the standards of the field in question and of science in general. It is obviously not foolproof, but it generally ensures that pseudoscience does not manage to masquerade as science, and that low quality work gets filtered out.

The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers", i.e. other experts in the same field, to assess the quality of the work. A similar approach is also generally taken to evaluate research proposals submitted to Science Foundations, such as the American NSF, where the proposals are sent out to qualified scientists to assess whether the proposed projects merit funding.

Contents

Relevance to the scientific community

As practised in modern universities, peer review is a key part of the scientific method, where the goal of the system is to ensure that work is stripped of biases, assumptions, and other errors, through the scrupulous review by one's equals in the field, acting as learned professionals and mediators. Accordingly, peer in this context implies equals: i.e. the reviewers or judges should have the same or reasonably similar qualifications that the author of the work has - or claims to have. No ideology, other than the commitment to "rigorous empiricism... without which no man is a scientist," matters in asking whether a person is a scientific "peer".[1]

Thomas Kuhn, in the process of seeking to define the nature of science, and the nature of "scientific progress", writes in his work, The Structure of Scientific Revolutions, that the unique composition of the scientific community, as composed of disinterested (read: unbiased) intellectual equals, is alone capable of generating scientific "progress."[2] Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life... the prohibition of appeals to heads of state or to the populace at large in matters scientific."[3] Simply put, politics has no place in science, and the goal of peer review is to strip politics out of the equation.

Possible problems

Like all human activities, peer review can be subject to biases in certain situations or if insufficient care is taken in the selection of reviewers:

  1. In very competitive areas of sciences, it is possible that reviewers are tempted to gain an unfair advantage when they receive a rival's work for review long before publication, and may even try to scoop the results.
  2. In the case of competing schools of thought, it cannot be ruled out that some reviewers might be influenced by their adherence to a certain point of view in giving negative reviews to colleagues from an opposing school of thought, although most intelligent editors would be able to allow for this issue.
  3. If the field of experts for a given topic is small, there is a certain likelihood that the reviewer may have a relationship of animosity, rivalry, or perhaps friendship with the author, and this could lead to a biased review process.
  4. The peer-review process doesn't involve replicating experiments or studies in order to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work, as it is presented, is suitable for publication. Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).

In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the editors of the scientific journal a work has been submitted to. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with publication, as the article or paper remains available for all interested parties to view - if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review process remains the most objective and qualified way to assess scientific work that has ever been developed.

Peer review as a legal benchmark

For more on this topic, see Evidence.

Without doubt, the twentieth century has seen an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life, before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove science to win his case, the court must have a way of judging the science.

Courts have met this challenge by adopting Federal Rule of Evidence 702, and the Daubert test, which require that "expert testimony" - a category which compasses scientific testimony - be methodologically sound.[4] In law student's shorthand, the Court in a Daubert inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the Daubert court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science, and also agreed that the only cure for bad science is vigorous "cross examination."[5]

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review, before science can be science.

Creationists and peer review

Allegations of bias in peer review

Creationists, whose "theories" have repeatedly been rejected by peer-reviewed journals,[6] claim that the system is rigged against them, since theistic explanations for natural phenomena are rejected out of hand because of scientists' assumption of methodological naturalism when working. The argument runs something like this: institutions of peer review assume that theistic explanations are invalid, since a built-in requirement of naturalism constitutes atheistic bias.

While prima facie compelling, and probably enough to dupe a few uninformed bystanders, this argument does not hold up to logic. The purpose of science (and rationalism) is to determine explanations of natural phenomena, with reference only to objective natural indicia, so as to create a body of knowledge which all of humanity can use and apply, if possible, in engineering. The purposes of science - to determine objective truth, free of a precondition of religious belief, and to create usable knowledge - are incompatible with theistic explanations.

Of course naturalism has its limits - if no objective explanation of a phenomenon exists, perhaps some theistic agency is at work. Thus, naturalism does not displace theistic explanations; rather, it pushes back theistic explanations where they are incomplete and inaccurate. To argue that theism should be a start for science, then, completely discounts the nature of science: accusations of bias fall apart accordingly.

Attempts by creationists to intelligently design peer review

A common criticism of creationism and its intellectual kindred (baraminology, intelligent design, and creation science) is that these "sciences" can never pass peer review, as they fail to conform to scientific methodology. In addition to accusations of bias, disproven above, creationists have attempted to create their own "peer review publications," in a vain effort to create their own body of "scientific" knowledge, and render neutral the "no peer review" criticism.

First as tragedy: the creation science "mainstream"

Of course, the "symptom" that creationism lacks peer review is indicative of the "condition" that it, and its brethren, lack real scientific merit: thus, by creating new "peer review" periodicals, "cdesign proponentsists" merely cure the symptom, not the underlying condition, i.e., the unscientific nature of their theology. In short, creationism does not gain merit by virtue of having an empty label ("Now Peer Reviewed!") attached to it.

However, several creationist groups have not gotten the memo: they continue to attempt to create sham peer review journals, each of which in turn betrays the ultimate failing of their own discipline, by failing to conform to the methodology of peer review, and merely usurping the label. Several examples come to mind: Answers in Genesis, CreationWiki, JPANDS, and more "mainstream" creation scientists.

As one commentator has noted, creation "science" under its infantile peer review system approximates the level of depth of inquiry of 18th-century science.[7] Since creation science cannot be tested without failing horribly, its intellectual discourse resembles, essentially, the creation of wild guesses, the discussion of said guesses around a fireplace, the assembly of these guesses in journals, the commitment to investigate them at some point, and then, the break for dinner. The process then repeats. The system entails no accountability because by its nature it cannot be accountable.

Then as farce: CreationWiki

For more information on CreationWiki, see its mainspace article.

CreationWiki, "a free encyclopedia of apologetics that is being assembled by an international team of missionaries,"[8] has attempted to create a peer review system[9] to vet CreationWiki articles, with the assumption that the result is something of "peer reviewed" quality. The assumption is either deliberate deceit or unconscious misapplication of a scientific term of art; since science depends upon a group of "peers," those being individuals versed in high sciences, not apologetics, nothing produced by CreationWiki could ever approximate peer review, unless its membership base changed by nearly 100%.

Note - parts of the following section refer to private conversations, on matters of public import. If you are the quoted private author, please comment on the Talk page to request removal of your words, but bear in mind that they are unattributed and public in scope.

Evidence suggests that the conflation of "peer review" with "casual editing" by CreationWiki indicates a larger confusion, within creationist circles, about the meaning of "peer review." CreationWiki administrators in particular seem to believe that "peer review" is meant to be deliberately slanted: in their own words, "that is the goal of peer reviews in general - to uphold the consensus position."[10] Apparently, creationists reconcile their own exclusion from academic journals not by conceding their own lack of merit, but asserting that peer review, in general, is meant to "censor," and responding in kind: in their own words, "peer review is done by professionals who hold to a POV and censor other views."[11] In short, creationists see denial of publication for lack of scientific merit as censorship for point of view: a rhetorical trick often used in creationist literature, as in Expelled: No Intelligence Allowed, to make the cause out as a pariah.

Latecomers: Answers in Genesis

In 2008, Answers in Genesis attempted to found its own "academic" journal, Answers Research Journal.[12] Hilarity continues to accrue at a fairly rapid rate, but it's too soon to have much to report.

Footnotes

  1. Thomas S. Kuhn, The Structure of Scientific Revolutions 42 (3d ed. 1996).
  2. Thomas S. Kuhn, The Structure of Scientific Revolutions 168 (3d ed. 1996).
  3. Id.
  4. See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993) (noting that the guarantee of “scientific knowledge” is one of “more than subjective belief” based in “the methods and procedures of science."
  5. See id. at 597 n.13 (“This is not to say that judicial interpretation, as opposed to adjudicative fact-finding, does not share basic characteristics of the scientific endeavor.”).
  6. See Intelligent design and academic freedom.
  7. Hanna Rosin, God's Harvard.
  8. http://creationwiki.org.
  9. http://creationwiki.org/CreationWiki:Peer_review
  10. Full quote:"Creationist disagree with each other on many issues. There are a variety of creationist views or opinions regarding almost every aspect of the cosmos. It is the goal of the CreationWiki to represent the main view held by creationists, and describe fringe views as such. Peer reviews are just what the phrase describes - reviews by peers. Atheists and creationists are not peers regarding theories formed from these worldviews. Only creationists can provide peer reviews of creationist views. Nevertheless, we allow for noncreationist reviews as well." CreationWiki Administrator, discussion between AmesG and CreationWiki Administrator, 10/20/2007.
  11. CreationWiki Administrator, Discussion between AmesG and CreationWiki Administrator, 10/21/2007.
  12. http://scienceblogs.com/pharyngula/2008/01/a_new_source_for_fake_science.php
Personal tools
Namespaces

Variants
Actions
Navigation
Community
Tools
support