Friday, August 11, 2006

Happy hour: Peer review accountability

Peer review still remains the best available way for journals to review and accept papers, though this system has some limitations and problems. There have been some exciting changes though in how journals are viewing their reviewing process. Nature is beginning to debate about it, and PLoS now has a new, community peer review system, with PLoS ONE. All very cool trends.

But there was something else that the significant other and I ended up discussing about, and it was what we thought was a rather simple but required step. The present system of peer review is anonymous peer review, where you submit your paper to a journal, and then it is sent off to a couple of experts in your area, who review the paper. But neither you nor any one else in the world (apart from the editors and the reviewers) know who these people are. Though a majority of the reviewers take their job of reviewing papers very seriously (the only two papers I’ve reviewed thus far, I spent hours analyzing. That we can dismiss as early enthusiasm, but still), many do not. They end up doing a rather sloppy job of reviewing papers (usually due to reminders or phone calls from editors). They sometimes pass on their decision without reading all the details of the paper, sometimes basing it on the author reputations, or where the paper’s from (sad, but sometimes true). Or else, they base it on how “cool” the story seems to be (without looking out for appropriate controls in the experiments). These papers slip through the cracks, and get published, even in so called “premier” journals.

So, and idea we had a couple of nights ago was to make the reviewers and editor responsible for the publication of a paper a little more accountable. For this, an obvious solution could be to have anonymous peer review during the process of review, but after a paper has been accepted for publication, the reviewers names should be mentioned in print when the paper is published. In addition, it could be easily possible to include a paragraph at the end of each paper published, where the reviewer(s) can say what they liked about the paper, and what else they would have liked to see. That way, it seems to me that reviewers will strive to do a better job with reviews, since their reputations will (in a way) be at stake. This step should go a long way in ensuring fair but rigorous reviews of papers, with a little more responsibility for the reviewer to do a good job.

(Interestingly, I mentioned this to my advisor, who thought it wasn’t a bad idea. So perhaps it may have some support amongst academics after all (there’s an n of one here, I need two more people supporting this, and this idea becomes practically publishable in itself J)).

Update: More from PLoS One "9. Anonymity
Although reviewers may remain anonymous during the review process, we strongly urge them to relinquish this anonymity to promote open and transparent decision-making."
, and "If published, papers will be accompanied by comments from the handling Editorial Board member and will be made available for community-based open peer review involving online annotation, discussion, and rating.". Read about it here. All more than excellent. In addition though, I would like to see the reviewers comments too in the published paper, and their names being made pubic, just like the handling Editorial Board member's comments. Adds incentive to review well.

5 comments:

Anonymous said...

Sunil,

I would prefer if the peer review process was a bit more anonymous - the reviewers should not be given the name/affiliation of the authors. This makes the review process more transparent and helps the authors avoid murky politics.

Secondly, many scientists have a rather parochial attitude. They usually stick to their ideas and do not encourage alternate/novel interpretation on the same topic. This could be avoided if all journals adopt the Nature's policy.

Unfortunately, the editors themselves are scientists in many cases and hence have their own view on things. The paper assingned to such an editor is obviously doomed as he will choose refrees who have similar inclination as him.

Honestly, I doubt this problem can be ever solved or minimized to say the least.

Sunil said...

Patrix...yes.....peer review is not currently double blind (at least in the sciences). It is blind for the person submitting the paper, s/he does not know who is reviewing it, but the reviewers know who has written the paper, and what their affiliations are. I have always liked the idea of a double blind review (though you can figure out who has done the work, it atleast adds a layer of anonymity). Here what i'm suggesting is that it remain double blind during the review process, but the reviewers are mentioned after the paper is accepted for publication, in print. At this stage, they also get to write a short paragraph (which appears in print) on what they thought the best features were about the paper, and what is missing/should be done in the future. That way, the reviewer cannot hide in anonymity after the review process is done, and the paper is found to be shoddy or incomplete, or badly reviewed.

Anonymous.......that's a good suggestion, and I think some journals are beginning to debate it. I second that idea. That way there's less of a chance that work from "smaller institutions" etc will not be taken seriously, or "big name labs" getting papers published on the weight of their names. But all your other points remain very true. But these are exciting times, with the changes happening. PLoS seems to be really leading the way towards journal reform here.

Anonymous said...

It is my understanding that it is exactly what the new PLoS journal is providing.

Sunil said...

Anonymous.....almost there, but no one is really pushing for making the reviewers directly accountable. PLoS is really a nice initiative though, and is encouraging reviewers to give up anonymity. I've added an update, thanks.

Anonymous said...

The medical journals in the BMC series published by BioMed Central use a system of open peer review. The reviewers are still selected by the editors, but the authors know who the reviewers are and the reviewers who are the authors are. If the article is published, the reports are available as part of the "pre-publication history". This last part was what Sunil suggested, and it has been running for the past 6 years.

Declaration of competing interests: I work for BioMed Central, and I have run the peer review process under both anonymous peer review on the biology journals and open peer review on the medical journals.