I get asked to review a large number of submissions to second- and third-rate journals.
For a long time now, I’ve had a policy of agreeing to review the paper if it looks like a junk paper that’s an easy reject. If I do so, I usually try to review the paper immediately, which (if the paper is really junk) takes about 15 minutes, maximum. (If a paper looks interesting to me, I may agree to review the paper but I won’t get to it right away and it will take me longer. If the paper does not look like garbage but does not look interesting to me, I will decline to review it. If the paper was submitted to a top-tier journal, I will usually agree to review it assuming I am not over-committed already.)
I should be clear what I mean by “junk/garbage/crap”. There are really several categories here. There are the papers that are unreadable due to poor exposition, awful English, formatting, etc. There are the papers that prove trivial (or known) results, and papers that prove uninteresting results. Then there are papers that have the potential to be interesting, but do not provide any formal definitions or proofs. (In the latter case, depending on the result, I may encourage the authors to resubmit with proofs.) I’m really not talking about bad papers here; I’m talking about junk. (It’s amazing to me that there’s so much junk being submitted…)
The question is: is spending even 15-20 minutes performing this “service” worthwhile? Or should I not care so much about the fact that a junky paper might get published in a junky journal? What do other people do?
Random bits 
You are improving the signal-to-noise ratio in the literature. That seems like a worthwhile thing to spend 15 minutes on.
By: D. Eppstein on June 21, 2010
at 3:33 pm
Yeah, but it’s 15 minutes 20-30 times per year…
By: jonkatz on June 21, 2010
at 3:54 pm
You are also saving some other reviewer’s time, someone to whom it may be less obvious that it’s junk. 🙂
By: anon on June 21, 2010
at 4:32 pm
I feel that editors should do a better job of rejecting crappy papers. They should standardize reasons for rejecting crapping papers and one simply has to select the reasons. I feel that too much time is wasted in finding “reasonable” ways to reject papers when the conclusion is arrived at very quickly. Of course there should be mechanisms for redress in case of errors by editors/reviewers but I think it will be a net win.
By: Anonymous on June 21, 2010
at 4:35 pm
I should, I suppose, mention that I am an editor for a second-tier journal (maybe that merits another post…) and I always do a quick check of papers to see whether they can be rejected as garbage before sending them out for review.
I have a couple of “standard” responses about lack of formal proofs, or lack of interest in the result.
By: jonkatz on June 21, 2010
at 4:38 pm
If the 2nd or 3rd tier journal is a journal from a publisher that has been a bad actor (e.g. Elsevier) then it is less clear that performing the service is of benefit to the community..
By: Paul Beame on June 21, 2010
at 5:58 pm
I think we should find ways to reduce the number of junk submissions. One simple way to accomplish this is to make all submissions public. Has this ever been tried/discussed?
By: Arvind on June 21, 2010
at 9:45 pm
I think the answer you’ll get depends on how you phrase the question. Observe the difference:
“Should I review these papers?”
“Should I review these papers or write a paper / prepare for a class / write a blog post / etc.?”
Your time is not an infinite resource, and you will be constantly asked to give it away to other people for free. I think that getting an answer to your specific question is much less important than making a habit of thinking about your time in terms of the second question form exclusively.
By: Jonathan Herzog on June 21, 2010
at 11:09 pm
Hi Jon, this is somewhat tangential to the question you were asking, but your post made me think about the following issue.
Your DBLP entry indicates that you have published almost 100 papers in the past 10 years. Assuming that a paper is reviewed by 3 people on average this means that 300 experts have spent time on reviewing your papers. Moreover, since your papers are not crappy, one is safe to assume that the reviewers have spent more than the 15 minutes one would have spent on a crappy paper.
It only seems fair (and in fact may be necessary for the peer review system to function) that the time people spend on reviewing is proportional to the time other spend on reviewing their work.
My question is what is the total number of (non crappy) papers that you think you should have gotten to review in the past 10 years (on average)? Is this number comparable to the number of papers you actually got to review?
In coming up with a number I would of course take other factors into consideration, such as the number of co-authors and their relative seniority/expertise. In this context, I am not at all sure that the right thing would be to simply divide things up by the number of co-authors, since some of them may have had short academic careers, and may have even reviewed more papers than they have written. I am also not sure how one should be counting papers you have read, say as a PC member, but not written a full review for.
By: Alon on June 22, 2010
at 6:01 am
My question is what is the total number of (non crappy) papers that you think you should have gotten to review in the past 10 years (on average)?
Good question, to which I don’t know the answer. Clearly reviewing papers is a part of community service, and one that I don’t mind doing (see below). My question was more specifically related to obviously bad papers, and I was most interested to know whether other people are getting all these bad submissions also (and, if so, how they handle them).
Is this number comparable to the number of papers you actually got to review?
I save almost every review I write. (I used to save every review; now I am more selective and, in particular, don’t save reviews of garbage papers.) When I am on a PC my default is to write a review for every paper I am assigned. I took a look, because I was curious myself, and found that I have reviewed at least 900 papers in the last 10 years. (It should go without saying that the time invested in a conference review is much less than the time invested in a journal review, and the overwhelming majority of my reviews have been for conference submissions.) Hmm…maybe I should cut back on reviewing the crappy papers after all…
By: jonkatz on June 22, 2010
at 7:32 am
900 papers?!
By: Alon on June 23, 2010
at 1:24 am
I have a general policy that I don’t review papers for journals that I don’t publish in (especially obscure journals that I would never even contemplate publishing in). I focus my reviewing on the conferences and journals that I publish in. I think that’s only fair.
By: AnonProf on June 23, 2010
at 6:50 am
Maybe I will refuse to be a reviewer for a crappy journal and reject the junky submission immediately.
By: Jaci on August 11, 2010
at 9:58 pm