Low quality peer review

I think that I’ve mentioned before that I do a large amount of manuscript reviewing. Most of the reviewing I do is for a particular journal in my field and I get almost all of these reviewing opportunities from one Editor. While some may look at reviewing as a burden, I do not look at it this way and the job is made more pleasurable because this particular Editor tries to send me manuscripts for review that she is confident that I would read anyway. She does a very good job of judging this and I rather enjoy getting to take an early look at work that I am interested in and I also enjoy having the opportunity to help authors improve on their manuscripts (many of which are excellent to begin with).

As a reviewer you eventually get to see what your fellow reviewers have to say about a manuscript (generally right after you submit your reviews). I usually try to take some time to check these other reviews out and as a general rule those other reviews are very similar to those that I submit. On occasion; however, they are quite different. Sometimes another reviewer will point out something that I overlooked or they will have a problem with a manuscript that is likely based on a difference in our specializations. All too often, though, the other review is completely worthless. What do I mean by this?

There are occasions where a review that essentially says that a paper is pure crap in kinder terms is warranted. Sometimes you get a paper that is written so poorly that you have no idea what is going on. There is really no way to suggest to improve on such a paper other than saying that the authors really need to work on the quality of the written manuscript. I have also received manuscripts that completely lack controls (in which case you can reject and simply state the controls that need to be done) or manuscripts that are based on a false premise (in which case you can simply tell them where they went wrong). Sometimes a stock-critique (like the paper lacks focus or there is no hypothesis) is warranted. In this case evidence must be given by the reviewer to support such a claim. If the stock-critique is warranted it should not be hard to elucidate to the Editor evidence that this is true.

The above instances do not describe two recent reviews submitted for papers that I thought were quality manuscripts. While I am not going to quote the actual reviews, they both boiled down to a few sentences laden with the usual stock-critiques. The paper lacks focus, there is no clear hypothesis, the images are of low quality and the findings are not clinically relevant. Not a word of support for such statements (and not even a word to indicate that the reviewer actually read the manuscript), just pure regurgitation of keywords geared to move the Editor toward the direction of rejection. Except that is never what happens.

I’ve been through this enough times with this particular Editor at this particular journal to know what is going to happen next. A new reviewer is going to be invited to look at the manuscript and the time to decision is going to be delayed. This creates extra work for another reviewer, it creates more work for the Editor and it delays the progress of the authors of the manuscript.

So what is the lesson here? Do a good job when you are invited to review. Always give evidence of your claims and try to improve the quality of the papers that you review. Sure, you’ll never get credit for making a manuscript better but there is some satisfaction in seeing a final product that is quite polished based on some suggestions that you made to the authors. Finally, remember that as a reviewer you are not the gatekeeper for publication. This is the responsibility of the Editors. Your job is to provide advice to the Editor and to the authors to help everyone add quality studies to the archive that is the published scientific literature.


8 responses to “Low quality peer review

  1. preach on. Also, don’t accept an invite to review if you are going to boot on it.

  2. Ugh, having received a useless review like you describe, and also seen it from fellow reviewers, I can relate to the frustration with it all. I think it flows from two things:

    Although reviewing is generally considered important and worth doing, the incentive to do it well is somewhat fuzzy, meaning that the big things that all scientists want, promotion and grants, are not directly related to performance in this aspect. So sure, your reputation as a crappy reviewer might get around and be a negative, but that probably won’t directly impact you at the next evaluation point. Some way to firm this up, and to really recognize the added scholarship inherent in a good review would help, but it’s not clear how to put something like this into practice.

    Secondly, this is further enabled by anonymity of reviewers. If your name was attached to a crap review, then you wouldn’t make a crap review. Still though, there are times when anonymity is necessary, again leaving us with how to change things in practice.

    When I’ve done reviewing (by direct request to me, surprising as it may be), I always force myself to: back up any assertion I make with some reference to the actual manuscript, and to write as if I were signing my name.

  3. the incentive to do it well is somewhat fuzzy

    In the short view this may be true, but the long view is quite different. The road to Editorial positions (I now have two in very decent journals) is a history of quality reviewing. Editorial positions are an important aspect of your career development and they reflect the stature of your reputation (that all important international reputation) for those all important tenure and promotion packages.

    by direct request to me, surprising as it may be

    Not surprising at all. Editors have a hard time finding reviewers and authors of quality papers with high citation indexes are going to get invited to review regardless of their career stage. You’re lucky, this gives you an opportunity (similar to my situation) to get some nice feathers in your hat at a relatively early stage of your career. The sooner you get on a Editor list the sooner you become a section editor and so on and so on…

  4. I think reviewing is like holding a door for someone: Respect. If you don’t give a crap about other people, you’re not going to bother to hold the door for them, you probably don’t care how much work went into their experiments and the good of science is obviously not of primary importance either. Also, lets not forget the element of competition…

  5. Juniorprof-

    I’m so with you on this one… and DM- useless reviews that are a few sentences and purely generalities with no details are not useful to ANYONE, not the author, not the editor, not anyone.

    If one is too busy to do a thorough review- one should decline the invitation to review the manuscript. You (and I don’t mean you personally juniorprof) aren’t doing anyone a favor by writing a 1/2 ass backwards unhelpful and uninformative review.

    Ok, done venting now.

  6. And one more thing Nat-

    I couldn’t agree with you more (I’m just all agreeable today) – people wouldn’t write such crap if they knew that their names would be attached to it.

  7. Before I get into specifics when reviewing a manuscript, I like to write a ‘General Comments’ section in which I basically summarize the manuscript in my own words while also offering, well, general comments. I really value when other reviewers do this for my own manuscripts, because I can then be confident that they really read and understood the main points of the paper. It enables an author to take a reviewer’s comments more seriously.

  8. バッテリー販売。 セルモーター修理。 オルタネーター修理。リビルト在庫多数。電装品販売。リンク品在庫多数。ウイングモーター修理・販売・在庫多数。パワーゲートモーター修理・販売・在庫多数。

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s