Last week a friend sent me a link to a website that had some pretty scary statements on it and asked, “IS THIS TRUE?????” Most of us know that when a friend sends us an all-caps message with an uneven number of question marks, the issue requires immediate attention.

I reviewed the post which claimed to have references, however, all of the references linked back to other comments or blog posts on the same site. It also didn’t reference any peer-reviewed sources and a quick review of the literature showed that nothing supported their claims.

When I told my friend this, she had only one response, “What does peer-reviewed mean?”

In short, peer-review is professional hell.

Despite its flaws, peer-review is a core pillar of science and academia. The phrase peer-review is literal, and means that your work is reviewed by a body of your peers and is deemed valid (or legitimate) for publication. Once your study has been published, it becomes a part of the larger academic body of knowledge and other researchers are able to repeat, cite and build on your work.

Sounds simple, right?

Oh, but let me expound upon the prestige of peer-review that I have just glossed over: Peer-review is the glamorous process in which your colleagues and leaders in your field, tear apart years of your work and point out everything you did wrong. When you submit your paper the journal may accept your paper as-is, reject it, or they may send it back with numerous recommendations and questions written with all the wit and snark of the late Joan Rivers.

Thus when you submit your study to a journal you want to make sure it’s pretty darn good and you have a thick skin. In the sage words of Dr. Rosalyn Cooperman: “never read the reviewers’ comments until you’ve had at least one glass of wine.”

Like anything, peer-review is not perfect. Each scholarly journal has their own process for peer-review and a lot of the time that process isn’t very transparent. Sometimes bad studies slip through because the reviewers aren’t familiar with that specific topic, or they become star-struck with a big name.

When bad studies do slip through, the informal peer-review process typically catches them, and quickly. One of the foundations of science is that a study needs to be repeatable in order for the results to be considered valid. Kind of like a recipe, if everyone follows the instructions exactly, we should all get the same result (within a margin of error). If Jessica follows my instructions exactly and gets a completely different result, that raises a red flag.

If enough people find that they can’t repeat the study’s results, the scholarly journal may retract or remove the study. Articles aren’t often retracted, so when they are it’s a very big statement on the validity of the study. When an article is retracted a brief statement addressing why it was retracted is added to the table of contents and to the study itself, and online a watermark reading “retracted” is added to the article. By the way, as career moves go, having an article retracted isn’t the best one.

Peer-review is painful and exacting because it needs to be. Researchers, academics and governing boards base their recommendations, policies and procedures on the studies published in scholarly journals. Reviewers have an obligation to protect the public from the impact of sloppy science and the author’s personal ego is the necessary minimal collateral damage in that pursuit.  So when a friend sends me a link to a website that uses only itself as a reference, I’m appropriately skeptical.

 


Resources:
University of California, Berkeley. Scrutinizing Science: Peer Review. Retrieved February 12, 2015.

Smith, R. Peer Review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine. 99:4. 178-182.

Nature Materials. Editorial: Transparency in Peer Review. 10:81. January 24, 2014. Retrieved February 12, 2015. doi:10.1038/nmat2952

Wing, JM. Why Peer Review Matters.Communications of the ACM. September 7, 2010. Retrieved February 12, 2015

Tags: , ,
Categories: Science 101 + Mythbusting