What’s the best (or worst) review you’ve received?

It’s Peer Review Week and we’d like to hear your thoughts

Sep 13, 2019
0
1

Can you believe this is the fifth year of Peer Review Week?!

There’s a great post about the history of PRW on the Scholarly Kitchen if you’re keen to learn more about how it started and what it’s grown into. Most years have had a theme, from transparency to diversity, and the topic chosen for 2019 is…

‘Quality in Peer Review’

So we’re asking:

What’s the best (or worst) review you’ve ever received? And why

We’d love to hear from you so please do take the opportunity to share your thoughts and experiences on Peer Review! Simply use the comment box at the bottom of this post to share your answers.

One thing – as with any contribution to the Community, do keep in mind our Community Policy when sharing your comments (basically, be nice and stay on topic).

Got a question? Get in touch with us here.  

Ruth Milne

Community Manager, Springer Nature

1 Comments

Cédric Aria 1 day ago

I find this call to be an excellent idea, because this topic is in urgent need of being discussed. There are a number of peer review panels out there, but a wider, open and transparent forum should take place.


I’ll put it strongly: I think the peer review system is broken. Or, perhaps more accurately, that its organization and evaluation are primitive and, hence, dysfunctional. More and more commonly do I collect my manuscript after one or more rounds of reviews just to resubmit it elsewhere, with barely any change but the formatting for the new journal.


Don’t get me wrong: I am not about to complain about the harshness of reviews, nor am I that stubborn scientist crusading against the world. Well, perhaps a little bit, to the extent that we all have to when trying to put forward new ideas. But what I am denouncing here is much more serious. It is the abundance of “reviews” that entirely miss the task they are meant to fulfill. And way too often this is left unchecked – but we’ll go back to that.


A reviewer can be more or less a promoter of their own views, and their review can be more or less emotional, but, as a basis, we expect that, in its main lines, the review will:
- Discuss the evidence at hand and its analysis;
- Try to contribute to the betterment of the work by providing constructive content – unless the study is flawed or notoriously below quality standards;
- Consider the value of the evidence presented for the community, regardless of interpretational differences.
This is how the peer review system could be really powerful and efficient, by contributing more minds to the problem at hand, and in so doing, widening perspectives and providing avenues of investigation that the original authors may have missed.


Instead, about two thirds of the reviews I receive either (or often combinations of the following):
- Are openly hostile, generally implying that everything written is wrong because the authors should not be allowed to publish, regardless of what is actually said;
- Try to unsubtly block the publication by asking for absurd complementary analyses or by simply saying that the interpretations are not justified or incorrect, while, in fact, they clearly are to all parties. Working in palaeontology, I have had reviewers recommending rejection based on a lack of a detailed description of every specimen documented and justification that they belonged to the correct morphotype (which, of course, is not how we proceed) or hypocritically claiming that specimens were wrongly assigned to a given morphotype – when they actually were;
- Attack alleged weaknesses of the manuscript during a first round of review and, when proven to be incorrect by the authors, ignore these points and invoke other supposed caveats during subsequent rounds, up to, when running short of arguments, the originality of the study;
- Discuss at length the opinion of the reviewer on the topic, and how the incomplete study should be rejected, while that reviewer seemingly has not even read the manuscript, for most of the points were already evocated in the submitted text;
- Initiate debates irrelevant to the objectives of the paper and the evidence presented, and present these debates as major arguments for rejection;
- Provide a two-sentence review stating that they “do not believe” in the conclusions of the paper;
- Reject the validity of the Linnaean nomenclatural system and the concept of macroevolution.
The consequence of this is not only to hamper the advancement of science, but also to over-protect paradigms defended by colleagues with influence and their followers, and, after a while, to lay the ground for dogmas that are more comfortable than really sound and solid.


It is more than obvious to me that a system of evaluation of reviews is lacking. The idea of Peerage of Science is very interesting in that regard, as they propose to cope with this problem by having the reviews themselves evaluated through peer-review. However, (1) this could become time consuming, (2) reviewers should be registered with Peerage of Science to participate, (3) a wide range of journals should abide by this system to make it worth, and (4) such fundamental process should not be left to a single company, which has journals pay to be able to browse through reviewed manuscripts (at least with options). 


Perhaps easier would be to provide editors with a sharable database in which reviews could be secondarily evaluated, for instance by the editors themselves, or at least those of the same subject area. Although extraneous to the editorial community, I do not doubt that editors already exchange about the suitability of different reviewers, but, given the above, something much more systematic and objective is needed.


And this makes me reach my next, and perhaps most important point, that the responsibility does not stop at the circle of authors and reviewers. Editors have a large and direct involvement in this issue. This starts of course with the apparent unwillingness of many editors to judge past blatantly inadequate reviews, or at least the reluctancy to put them into question, as long as they are disguised as professional reviews or that they come from alleged experts. It seems impossible that this stems from an unawareness of editors about what reviews are adequate, and I suspect that timeliness and other journal constraints come into play as well. But the problem also goes much further than that. For instance, cases such as:
- Editors replying that “although they think the authors could provide counter-arguments to the reviewers, they decide not to send the manuscript for further review because they think the reviewers will never concede ground.” If the editors implicitly acknowledge that there are arguments in favour of the authors’ work, they should assume their role as additional party and decide whether the position of the reviewers warrant rejection (because revisions are another matter entirely). Or call for additional opinions. Otherwise, this is simply stating that new and controversial ideas are fated to never be published in scientific journals, regardless of their strengths. That is, killing science;
- Editors having preferential relationships with some authors, either as such or as reviewers. While this needs not be, this can certainly lead to conflicts of interest. Authors that do not have the favours of these other researchers close to the editors of a given journal are in a way cut off from that journal, and know that there is no point submitting there. That may be common, disavowed knowledge, but as an early career scientist, I find it unacceptable, especially when some colleagues occupy several of such editorial positions and can block access to a wide range of journals;
- Editors that tend to consider status before content, and would give more credit to reviewers because their position / institution would be higher or more “prestigious”. This seems perhaps too simple-minded or too openly unethical to be true, but such a bias can express itself at different levels, and can be more or less conscious.


On the other side of the bench, as reviewer, the behaviour of certain editors can be also disconcerting. This happened to me for instance when an editor decided to tell the author in his letter to ignore my review, because this review dared to criticize some of the work the editor had been involved with. Such partiality has very serious consequences for the representativity of visible research and the health of a given field as a whole.


While there are measures to establish in the long term, editors have the power and ability to make improvements in the short term. Asking for more fairness in review through appeals is not really a solution, because authors could also indiscriminately appeal even when a rejection is reasonable. Asking for additional opinions when original reviews are questionable is something that some editors are already doing, and I have also seen editors taking the initiative of having the reviewers assessing each other’s reviews. But this also implies that the editor, following this, has to be able to make a judgment on the quality of the final reports, and not simply stand back and take a decision based on the average outcome. 


Of course, the perspective on peer review will depend on the field, the thematic, the research group, and the individual researcher involved. But this is also part of the problem: Complete impartiality in peer review is impossible (and perhaps conflicting with the idea of peer review itself), but we should strive for more equality in fairness and objectivity across publications.


C.A.