A surprisingly satisfying way to push back against Bad Reviews, is to engage in reflection upon one's own reviewing skills and strive to improve them. This course of action is not going to have an immediately measurable effect of improving the system as a whole, but it does restore a sense of balance. Especially if you interact with a lot of students, you have an amazing opportunity to teach them to review. There's something cheering about knowing that scientists that you have mentored are not going to be the ones generating the Bad Reviews of the next generation.
In order to be able to tell people quickly about my own reviewing values and my campaign to constantly improve my own reviews, I have packed the points I consider while reviewing into a scheme that I call IF THEN:
- I is for Issue: Does the paper motivate the issue that it addresses and then close the loop in the end, convincing the reader that it has accomplished what it set out to do?
- F is for Fit: Does the paper fit with the call for papers of the conference or scope of the journal to which it was submitted?
- T is for Technical soundness: Do the authors apply solid, state-of-the-art experimental and/or analytical method?
- H is for Historical context: Do the authors present the context of their work? (Including both the related work and the outlook onto future work.)
- E is for Exposition: Is the paper clearly written and a pleasure to read? Is the information it contains complete and comprehensive?
- N is for Novelty: Is the idea new in the field? Is it the sort of innovation that is destined to make an impact?
But what if you are already a world class reviewer? What then? Like musicians we must remember that excellence is not static. Moshe Vardi introduced a rule for reviewing an editorial in the current edition of Communications of the ACM. The rule reads, "Write a review as if you are writing it to yourself." He calls it The Golden Rule of Reviewing. Most people are still working on putting in to practiced the Golden Rule they learned in their childhoods. The Bad Reviews will keep on coming, and about the only thing we have control over is how we review back.