Cover story articlePeer review

From RationalWiki
Jump to: navigation, search
Simple stuff, in theory.

Peer review is the process of subjecting scholarly work to review by other experts in the field.

The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers" (i.e., other experts in the same field) to assess the quality of the work. A similar approach is also generally taken to evaluate research proposals submitted to agencies for funding, such as the National Science Foundation (U.S.) or Natural Sciences and Engineering Research Council (Canada), where the proposals are sent out to qualified scientists to assess whether the proposed projects merit funding.

Contents

[edit] Initial and ongoing peer review

The first, and most notable, step in peer review is when a piece of scientific work is looked at by reviewers (sometimes called "referees") for approval prior to publication. Typically each paper is evaluated by three reviewers. The journal's editor then evaluates the reviews and decides whether the paper should be published or rejected. Acceptance usually is provisional on the authors making revisions to take the reviewers' comments into account.

Approval by peer review is a necessary but not sufficient criterion for quality work. It mostly judges basic competence, such as minimising the chances that pseudoscience can masquerade as science, and filters out trivial or low quality work which would not contribute anything of value.

Following publication, peer review is an on-going process where a work is open to scrutiny by the scientific community at large - this is what publication allows a scientist to do. The process is designed to ensure that the work meets the standards of the field in question and of science in general. A paper which survives initial peer review may be shown to be rubbish when examined more widely after publication.

[edit] Relevance to the scientific community

Peer review is a key part of the scientific method, where the goal of the system is to ensure that work is stripped of biases, unjustified assumptions and other errors, through the review by one's professional colleagues. Accordingly, peer in this context implies equals: i.e., the reviewers or judges should have the same or reasonably similar qualifications that the author of the work has - or claims to have. No ideology, other than the commitment to "rigorous empiricism... without which no man is a scientist," matters in asking whether a person is a scientific "peer".[1]:42

Thomas Kuhn, in the process of seeking to define the nature of science, and the nature of "scientific progress", says in The Structure of Scientific Revolutions that the unique composition of the scientific community, as composed of disinterested (read: unbiased) intellectual equals, is alone capable of generating scientific "progress."[1]:168 Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life... the prohibition of appeals to heads of state or to the populace at large in matters scientific."[1] Politics has no place in science, and the goal of peer review is to strip politics out of the equation.

More prestigious journals have a more thorough peer-review process and have tighter criteria for the level of work they can publish - indeed this is why they are prestigious, as only the most groundbreaking work is published in these journals. The two top journals in the world for science are Science and Nature and getting a paper into them is considered a great achievement.[2] The vetting process for these two journals is intense, with less than 10% of submitted manuscripts going forward for publication[3] - that's 10% of the manuscripts scientists think are good enough. These top publications also tend to "coach" their submissions, improving the readability, conciseness, and clarity of the paper (again, this is why they're considered the best).[4] This isn't to belittle "lesser" journals at all; the publishers of Science, for example, openly acknowledge that articles may be rejected "not because the science isn't spectacular but because the area is no longer 'hot.'"[5] More typical journals in scientific fields ultimately accept perhaps 50-70% of submissions, though some revisions will almost always be required before publication.

[edit] Possible problems

Like all human activities, peer review can be subject to biases in certain situations or if insufficient care is taken in the selection of reviewers:

  1. In very competitive areas of sciences, it is possible that reviewers are tempted to gain an unfair advantage when they receive a rival's work for review long before publication, and may even try to scoop the results. A good editor will be aware of the potential for such conflicts and take them into account when making his final judgment on acceptance of a paper.
  2. In the case of competing schools of thought, some reviewers might be influenced by their adherence to a certain point of view and give negative reviews to colleagues from an opposing school of thought. Again, the journal's editor should be able to judge this issue.
  3. If the field of experts for a given topic is small, there is a certain likelihood that the reviewer may have a relationship of animosity, rivalry, or friendship with the author that biases the review process.
  4. The peer-review process doesn't involve replicating experiments or studies in order to test their truth value. The reviews of submitted papers are only to detect glaring errors in methodology and to determine if the work, as it is presented, is suitable for publication. Therefore the process won't detect outright fraud immediately (unless it's blatantly obvious).
  5. In journals lacking real oversight from publishers, editorial board members may be able to publish their own articles by either bypassing the process or hand-picking their own peer reviewers. This created a small scandal in the mathematics community when Chaos, Solitons and Fractals was found to have carried hundreds of articles by its editor-in-chief, Mohamed El Naschie.
  6. The editors of many journals are able to reject papers before they reach the peer review process.[6][7] While this can be a good thing in that it avoids wasting reviewers' time on obvious junk, depending on the rules in place this could lead to the unjustified dismissal of good science.
  7. In 2013 and 2014, the editors of Journal of Vibration and Control and SAGE uncovered a peer-review ring involving assumed and fabricated identities and led to the retraction of 60 published articles.[8]

In all these cases, the responsibility to choose unbiased reviewers and to recognize a biased review rests on the shoulders of the journal's editors. Some journals allow the authors to suggest that certain colleagues not be used as reviewers. In the case of fraud prevention, the peer-review process does not end with publication, as the article or paper remains available for all interested parties to view - if someone tries and fails to replicate the results, or finds that it contradicts research that they have done, then criticisms can be published and investigations made. Even in the face of these possible problems, the peer review process remains the most objective and qualified way to assess scientific work that has ever been developed.

[edit] Peer review as a legal benchmark

See the main article on this topic: Evidence

Without doubt, the twentieth century saw an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life, before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove a scientific argument to win his case, the court must have a way of judging the science.

Courts have met this challenge by adopting Federal Rule of Evidence 702, and the Daubert test, which require that "expert testimony" - a category which encompasses scientific testimony - be methodologically sound.[9] In law student's shorthand, the Court in a Daubert inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the Daubert court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science, and also agreed that the only cure for bad science is vigorous "cross examination."[10]

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review, before science can be science.

[edit] What peer review is not

Many people think that the process of peer review is meant to settle the actual validity of the work, and that in any paper that has passed peer review, the science is entirely correct. This is not the case. Peer review is an "entry level" sort of test that weeds out the pseudoscience and obviously bad work, but is not intended to be a catch-all for outright fraud or experimental error - reviewers simply challenge the rigour by which scientists are reporting their own work or challenge their conclusions if they haven't successfully eliminated competing hypotheses. Often enough, the demand for the right data and better conclusions made by reviewers is more than enough to ensure the work is valid enough, as the process is about making sure everything is submitted and out in the open with nothing being hidden. Due to this, direct replication and validation isn't usually a priority or even a necessity for peer review.

There are a few exceptions. For example, the American Chemistry Society won't accept computational chemistry papers unless the results have been verified. This is because taking parameters from a paper's supplemental material and running it on a computer for a few hours is practical; rigorously replicating experiments that may have taken months to get right and require specialised and bespoke equipment, is not.

Passing peer review and publication is indicative that (by the standards of the journal in question) the science is thorough, there are no glaring omissions, and the interpretation of the results presented are at least plausible, but this does not cement the science. Further publications and research can then use the data contained in the paper and its conclusions can be amended (in worst-case scenarios, retracted) in later publications.[11]

To make a legal analogy, if it is erroneously assumed that the peer review process is like a trial (the case either proven true or dismissed), the actual process is more like an arraignment, only verifying that the case has enough merit to be heard. Indeed the "trial" part of a scientific work is a very much on-going and continuous process that happens as other scientists cite the paper, or attempt to replicate or use it in their own work.

It is also worth noting whom those "peers" may be, as practitioners of pseudoscience might form a circle of pseudoscientists who start a pseudoscientific journal. It isn't the support of a claim that makes it true, it's the honest attempts to disprove a claim through experimentation that solidifies it.

[edit] Creationists and peer review

[edit] Allegations of bias in peer review

Creationists, whose "theories" have repeatedly been rejected by peer-reviewed journals,[12] claim that the system is rigged against them, since theistic explanations for natural phenomena are rejected out of hand because of scientists' assumption of methodological naturalism when working. The argument runs something like this: institutions of peer review assume that theistic explanations are invalid, since a built-in requirement of naturalism constitutes atheistic bias.[13]

While prima facie compelling, and probably enough to dupe a few uninformed bystanders, this argument does not hold up to logic. The purpose of science (and rationalism) is to determine explanations of natural phenomena, with reference only to objective natural indicia, so as to create a body of knowledge which all of humanity can use and apply, if possible, in engineering and medicine. The purposes of science - to determine objective truth, free of a precondition of religious belief, and to create usable knowledge - are incompatible with theistic explanations.

Of course naturalism has its limits - if no objective explanation of a phenomenon exists, perhaps some theistic agency is at work. Thus, naturalism does not displace theistic explanations; rather, it pushes back theistic explanations where they are incomplete and inaccurate. To argue that theism should be a start for science, then, completely discounts the nature of science: accusations of bias fall apart accordingly.

[edit] Attempts by creationists to intelligently design peer review

A common criticism of creationism and its intellectual kindred (baraminology, intelligent design, and creation science) is that these "sciences" can never pass peer review, as they fail to conform to scientific methodology. In addition to accusations of bias, disproven above, creationists have attempted to create their own "peer review publications," in a vain effort to create their own body of "scientific" knowledge, and render neutral the "no peer review" criticism.

[edit] First as tragedy: the creation science "mainstream"

Of course, the "symptom" that creationism lacks peer review is indicative of the "condition" that it, and its brethren, lack real scientific merit: thus, by creating new "peer review" periodicals, "cdesign proponentsists" merely cure the symptom, not the underlying condition, i.e., the unscientific nature of their theology. In short, creationism does not gain merit by virtue of having an empty label ("Now Peer Reviewed!") attached to it.

However, several creationist groups have not gotten the memo: they continue to attempt to create sham peer review journals, each of which in turn betrays the ultimate failing of their own discipline, by failing to conform to the methodology of peer review, and merely usurping the label. Several examples come to mind: Answers in Genesis, CreationWiki, JPANDS, and more "mainstream" creation scientists.

As one commentator has noted, creation "science" under its infantile peer review system approximates the level of depth of inquiry of 18th-century science.[14] Since creation science cannot be tested without failing horribly (when it can even be tested), its intellectual discourse resembles, essentially, the creation of wild guesses, the discussion of said guesses around a fireplace, the assembly of these guesses in journals, the commitment to investigate them at some point, and then, the break for dinner. The process then repeats. The system entails no accountability because by its nature it cannot be accountable.

[edit] Then as farce: CreationWiki

See the main article on this topic: CreationWiki

CreationWiki, "a free encyclopedia of apologetics that is being assembled by an international team of missionaries,"[15] has attempted to create a peer review system[16] to vet CreationWiki articles, with the assumption that the result is something of "peer reviewed" quality. The assumption is either deliberate deceit or unconscious misapplication of a scientific term of art; since science depends upon a group of "peers," those being individuals versed in high sciences, not apologetics, nothing produced by CreationWiki could ever approximate peer review, unless its membership base changed by nearly 100%.

Note: Parts of the following section refer to private conversations, on matters of public import. If you are the quoted private author, please comment on the Talk page to request removal of your words, but bear in mind that they are unattributed and public in scope.

Evidence suggests that the conflation of "peer review" with "casual editing" by CreationWiki indicates a larger confusion, within creationist circles, about the meaning of "peer review." CreationWiki administrators in particular seem to believe that "peer review" is meant to be deliberately slanted: in their own words, "that is the goal of peer reviews in general — to uphold the consensus position."[17] Apparently, creationists reconcile their own exclusion from academic journals not by conceding their own lack of merit, but asserting that peer review, in general, is meant to "censor," and responding in kind: in their own words, "peer review is done by professionals who hold to a POV and censor other views."[18] In short, creationists see denial of publication for lack of scientific merit as censorship for point of view: a rhetorical trick often used in creationist literature, as in Expelled: No Intelligence Allowed, to make the cause out as a pariah.

[edit] Latecomers: Answers in Genesis

In 2008, Answers in Genesis attempted to found its own "academic" journal, Answers Research Journal.[19] Hilarity continues to accrue at a fairly rapid rate.

[edit] The humanities

Fields in the humanities, such as history, literary criticism, philosophy, art history, cultural anthropology, psychology, women's studies, African-American studies, classics, and poetics differ from the hard sciences in terms of methodology and scholarly practice. In these cases, peer review exists to see if they hold to the academic and methodological standards of the specific disciplines, if they are internally consistent, if they engage with the established scholarship on the topic, if they account for long-existing problems, and if they make a meaningful contribution to the scholarship.

[edit] See also

[edit] External links

[edit] Footnotes

  1. 1.0 1.1 1.2 Thomas S. Kuhn, The Structure of Scientific Revolutions (3d ed. 1996).
  2. An achievement worthy of 3 cases of champagne, a slap-up steak dinner at a fancy eatery and an all night drinking session at the pub later. All on departmental expenses, actually.
  3. Getting published in Nature
  4. PhD Comics - Tales from the Road, Science vs Nature
  5. "Academic Scientists at Work: Publishing at the Top of the Heap"
  6. Editorial peer review: its strengths and weaknesses
  7. Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System
  8. http://jvc.sagepub.com/content/20/10/1601.abstract Retraction notice
  9. See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993) (noting that the guarantee of “scientific knowledge” is one of “more than subjective belief” based in “the methods and procedures of science."
  10. See id. at 597 n.13 (“This is not to say that judicial interpretation, as opposed to adjudicative fact-finding, does not share basic characteristics of the scientific endeavor.”).
  11. "[S]cientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth." Charles Jennings, Quality and Value: The True Purpose of Peer Review, NATURE.COM (2006), http://www.nature.com/nature/peerreview/debate/nature05032.html; see also Lynn S. McCarty et al., Information Quality in Regulatory Decision Making: Peer Review versus Good Laboratory Practice, 120 ENVTL. HEALTH PERSP. 927, 930 (2012) ("It is difficult to extract from the extensive body of work and commentary published over the last 25-30 years that scientific journal peer review is a coherent, consistent, reliable, evaluative procedure. . . . [T]he opposite conclusion may be more accurate.").
  12. See Intelligent design and academic freedom.
  13. Creationism, Science and Peer Review, Andrew Kulikovsky, Creation Ministries International
  14. Hanna Rosin, God's Harvard.
  15. http://creationwiki.org.
  16. http://creationwiki.org/CreationWiki:Peer_review
  17. Full quote:"Creationist disagree with each other on many issues. There are a variety of creationist views or opinions regarding almost every aspect of the cosmos. It is the goal of the CreationWiki to represent the main view held by creationists, and describe fringe views as such. Peer reviews are just what the phrase describes — reviews by peers. Atheists and creationists are not peers regarding theories formed from these worldviews. Only creationists can provide peer reviews of creationist views. Nevertheless, we allow for noncreationist reviews as well." CreationWiki Administrator, discussion between AmesG and CreationWiki Administrator, 10/20/2007.
  18. CreationWiki Administrator, Discussion between AmesG and CreationWiki Administrator, 10/21/2007.
  19. http://scienceblogs.com/pharyngula/2008/01/a_new_source_for_fake_science.php
Personal tools
Namespaces

Variants
Actions
Navigation
Community
Tools
support