How to be a good reviewer and a good reviewee

[In this module, we will discuss the steps involved in reviewing a paper in an academic journal]


By the time you finish graduate school, you may very likely have been invited to review a paper by the editor of an academic journal.  And you almost certainly will have had to respond to reviewer comments on papers you published during your degree if you were first author.

Reviewing Part I: background

Peer review involves vetting of research by colleagues of similar competence in order to maintain standards of quality in scientific work, and the descriptions of that work.  Most of the time one to four reviewers are asked by the editor of a journal to review a manuscript that has been submitted to the journal.  Usually two or three, and not very often one or four.

Most journals do not reveal the identity of the reviewers to the reviewees (single blind). This is meant to encourage reviewers to give forthright opinions, without fear of future retribution.  However, it does have the drawback that as a reviewee you never know who just recommended rejection of your work, perhaps for malicious or unethical reasons; for instance, they are perhaps working on a similar analysis,  and they feel it is in their best interests to delay publications of competing analyses.  The latter is highly unethical, and isn’t supposed to happen, but reality is that it sometimes does.

Some journals additionally do not reveal the identities of the reviewees to the reviewers (double blind).  The latter helps to further ensure opinions in the review that are unbiased by previous associations (“old boy networks“) or biases related to things like gender, race, or seniority level.

Best of all, in my opinion, is open review, where everyone knows who everyone else is.  Not too many journals do it, but I think it is a nice way to do things because it tends to keep things polite on both sides. Plus if you know who reviewed it, and they recommend rejection but turn around and publish something similar in the near future, the jig is up.

Sometimes shocking to some people in the arts and humanities is that scientists do not get paid to peer review the work of others.  It is an exchange of social capital that is expected in the field.

Reviewing Part II: receiving and accepting the invitation to review

Once you receive an email inviting you to review, if you are unfamiliar with the name of the journal Google the name, and look at what kind of papers they publish and determine whether or not the journal does indeed publish things within your area of expertise.

Within reason, you should agree to peer review publications within your areas of expertise when asked.  Being a statistician in addition to other things, I get called upon a bit too often by various journals to review a wide variety of papers.  I’ll do at most one a month, and decline to review excess requests above that limit.  If you decline to review, you are usually asked to give a reason, and suggest another reviewer.

Reviewing Part III: reading the paper

Expect to devote three to six hours to reviewing a paper.  Always keep in mind that your job as a reviewer is to assess the relevance and novelty of the work, assess the robustness and veracity of the results, and provide suggestions to improve the written and graphical presentation (if necessary).

Jot down notes as you read through the paper.

The first things to read are the Abstract, Introduction, and then flip to the Figures and Tables.  In a well written paper, this should be enough to get a good idea of the merits of the paper.  In the Abstract, they should have concisely summarized the conclusions of the paper.  If not, they need to.

Even if the Abstract is not explicitly in structured format (Background, Methods, Results, Conclusions), it should read like it is, with the information in that order.

Now turn more careful attention to the Introduction. Did the authors begin by properly motivating their work?  Do they provide references to the literature for every single statement-of-prior-fact that they make?  If not, they need to.

Did they clearly state the objective of their work, and state why it is novel?  If not, they need to.  If the novelty of the work is unclear to you, don’t be afraid to speak up as a reviewer… don’t be timid if you are a relative novice in the field and think that maybe the novelty of the work might be obvious to a more senior person.  It isn’t necessarily, and it is a poorly written paper that assumes it.

Is the outline of prior work on the subject comprehensive?  If unsure, take one or two of their references to previous work and search for them in Google Scholar, then look at the papers that cite those papers.  If there are relevant publications that should have been cited in the paper, they need to be added.

Do they have their paper in the usual format of Introduction, Methods and Materials (with subsections, as relevant, Data, Mathematical Modelling Methods, Statistical Modelling Methods), Results, Discussion, Summary (note that some journals have slightly different section names, but most follow this forma)t.  If the paper is not in this format, do the authors give a sentence or two at the end of the Introduction telling you what the stages of information flow will be?  If not, they need to add it.  Around 50% of papers I review are not in this format, and they are invariably a tedious mess to have to wade through.  If you are finding that you are getting lost as you try to read through the paper, the problem is the paper, not you.

After the introduction should come Methods and Materials.  If data are used in the analysis, the description of the data sources is the first subsection.  The authors need to thoroughly describe their sources of data.  When, and where, and by whom was it collected?  Using what methods or protocols?  The paper should give sample sizes and other relevant information.

If the paper involves a mathematical model, that is usually the next subsection (or the first subsection, if no data were used in the study).  Do the authors give the equations of their model?  If relevant, the initial conditions?  If relevant, a compartmental diagram? Do they provide a table describing all the parameters of the model (some authors put the information in the text, and I hate that when I’m trying to review… if you forget what one of the parameters is, you have to flip back through the paper and tediously re-read the text).  Do they describe the assumptions of their model?  Do they cite relevant work related to similar models?

If possible, code up their model in R or Matlab, and double check their results.  You’d be amazed at how often there are typos in the paper, or problems in the model formulation.

In the statistical modelling methods subsection (if relevant), they authors should have given a brief overview of standard statistical tests they applied, and longer descriptions of more esoteric methods (and they should have citations relevant to those methods).  If you don’t understand this section as a non-statistician, don’t feel afraid to comment that the methods require more explanation.

In the Results section, they should have just results, and no discussion (far too often the Results and Discussion sections are mixed together in a mess).  Do their tables and figures have informative, stand-alone captions?  Are the axes on all plots labelled?  Are the different lines on a plot clearly labelled?  Is the figure or table actually relevant to the description of the results?  (sometimes plots are just thrown in for no apparent reason other than to put a plot in the paper)

In the Discussion section do they have results that should have first appeared in the Results section?  Do they put their results in context of what has been done in the field before?  And explain why what they have done is novel?

In the Summary section, do they reiterate what is novel about their research?

Do all the methods and results mentioned in their Abstract actually appear in the paper?

Writing up the review

Remember, your job as a reviewer is to maintain the standards of the field, and provide suggestions to make a body of work better (if indeed it requires improvement).

Always try to find something good to say about the paper, even if you are recommending rejection.  Was the work well motivated, even if not novel (or some other problem)?

Even if the paper or analysis has problems, can it be salvaged with improvements

Lack of motivation and/or clearly stated objective=rejection.


Visits: 1221

Leave a Reply