How to write a good scientific review

by Martin Monperrus

A review is a critical evaluation of a paper. In a usual peer-review process, a review is meant to be read: by the editor (or the PC chair) and by the authors. The editor/chair uses it to accept/reject the paper, the authors get a wealth of information to improve their paper.

There are good reviews and bad reviews. A good review helps the editor to make an informed and just decision. A good review helps the authors to improve their work in terms of method, interpretation and writing. In the long term, a good review contributes to building better knowledge. A bad review is wasted energy for everybody.

There are no unique recipe to write good reviews. The tips below are just meant to help nascent reviewers writing their first reviews.

Reviews Parts

In a nutshell, the two first parts are for the editor (and read by the authors as well), the rest is for the authors only (but may be read by the editor as well). The review flows from high level considerations (e.g., soundness, novelty) to details (disagreement on sentence formulation) and typos.

Then, one discusses the usual scientific points:

A deep discussion on those points may take many pages. An other important point for knowledge dissemination is clarity.

This corresponds more or less to Offutt's questions: "What are the major results, are they correct, are they new, are they clearly presented, and are they worth publishing?" [3].

Verifiability: in addition, for empirical research, to what extent sufficient information is available to support the full or partial independent verification or replication of the claimed contributions.

A third part gives high-level comments on the presentation: Are the title, abstract, and keywords appropriate? How well are the ideas presented, is the paper structure appropriate? What is the overall quality of the writing? Is the language acceptable? Should anything be deleted from or condensed in the paper?

The last part of the review lists remarks and suggestions and follows the order of paper.

Constructive reviews

One often disagrees with many points. But it is important to focus on the positive! For the negative parts, a very important aspect of reviewing it to provide helpful comments, I.e. to turn "turning negative feedback into productive feedback" [2]. Here are some good examples from Boi Faltings [1] and Seri Lowell [2].

Rather than: “This problem has been solved by many people years ago.” Helpful Comment: “This problem has been solved by A. Smith (AI Journal, 1992), with improvements by C. Miller (ECAI, 1999).”

Rather than: “I don’t think this solution works.” Helpful Comment: “On the following example, the method produces the wrong result: ...” or “The proof of Theorem 3 is wrong, and here is a counterexample...”

Rather than: “The description is unclear.” Helpful Comment: “The terms “gizmo” and “babble” are not defined anywhere...” or “The term “globber” is used before it is defined...”

Rather than: "This section needs work." Helpful Comment: Combine the related actions into a single sentence in Methods, eg, "Flies were assigned randomly to 5 treatment groups of 25, and were weighed, sexed, and marked with non-toxic paint before behavioral trials began"

Rather than: "Disorganized!" Helpful Comment: "This section discusses both animal-rearing conditions and experimental methods, but the two are mixed together. Could you separate each into its own paragraph?"

Rather than: "How are these references relevant?" Helpful Comment: "The background and references given in paragraph 2 don't seem directly relevant to our hypothesis. I think we need references on how light has been shown to affect flowering (in sunflower or any species), and less on other factors that promote or inhibit flowering."

Rather than: "Unclear." Helpful Comment: "I'm not sure what your interpretation is after these two paragraphs: does the experiment show that mung beans cure cancer, or not? Which are we concluding? If the sample size is too small, we need to discuss that when we suggest future research, but that does not change our results here."

Bibliography

[1] "How to write a review", Boi Faltings, http://liawww.epfl.ch/~faltings/STAIRS-2004-talk.pdf

[2] "Helpful Hints for Effective Peer Reviewing", Seri Lowell, http://abacus.bates.edu/~ganderso/biology/resources/writing/HTW_Guide_Effective_PR_9-30-08.pdf

[3] "Standards for reviewing papers", Jeff Offutt, http://cs.gmu.edu/~offutt/stvr/17-3-july2007.html

Tagged as: