Posted by: jonkatz | July 6, 2009

What’s wrong with FOCS/STOC (take 2)

I’ve been traveling, without regular Internet access(!), the past few days. But since so many others have already been posting about FOCS, I figured I should deliver on my promised post.

Below I discuss two concrete ways to improve STOC/FOCS. This post is motivated by the lame reviews I got for my FOCS submissions — 2 out of the 6 reviews were written by people who either didn’t understand or didn’t read the papers — but are not specific to that, and represent things I have thought about for a while.

  1. Increase the size of the program committee. This would actually serve two purposes:
    1. First, it would cut down on the insane number of papers assigned to each reviewer.
    2. Second (and something I have not seen anyone mention before), it could ensure that each area of TCS has at least two representatives on the committee. I can’t tell you how many times I have heard someone say: “I guess I won’t submit my paper to STOC/FOCS this time around, since there is only one (or even no!) cryptographer on the committee”. Is this really the way we want decisions to be made, or science to progress?

    The only argument I have heard for smaller committees is that large committees are problematic because committee members cannot submit papers. Personally I would get rid of that rule. (Plenty of other conferences allow PC-member submissions, and once conflicts of interest are being enforced anyway I don’t see the problem.) But to address this concern, here is a “radical” idea: have “heavy” and “light” PC members. Heavy PC members would essentially fulfill the same role as PC members do today: they would be expected to review lots of papers, possibly in areas outside their core expertise, and discuss all papers at the PC meeting. Light PC members would be area experts. Each paper could get assigned to, say, 2 heavy PC members and 1 light PC member with expertise in the paper’s area. Light PC members would read fewer papers overall, and would not be involved in paper discussions or the PC meeting. It is therefore feasible to have more light PC members, and without necessarily preventing them from submitting their own papers.

    I would still recommend that, for each area, there be at least one “heavy” PC member in that area.

    For those who are not aware, systems of “light” and “heavy” PC members are already used in other conferences, though without making light PC members area experts (as far as I know).

  2. Move to anonymous (or partially anonymous) submissions. This has been discussed ad nauseum many times before, so I won’t belabor the point. All I will say is that while there are some valid arguments in favor of non-anonymous submissions, I think they are outweighed by the arguments in favor of anonymity. Moreover, there is a compromise solution that I think would work very well (and has not received enough attention in the whole debate): have papers be reviewed anonymously, and then de-anonymize papers during the discussion/PC meeting. This can be easily done with conference review software.

One final comment: when will any of the various proposed changes to STOC/FOCS be discussed seriously, i.e., by anyone with actual power to change the way things are done? Can the issue be brought to SIGACT vote? (I have no idea how these things work.) If a majority of people prefer non-anonymity, so be it; can we at least take a poll/vote to find out?



  1. Is the idea of “light”/”heavy” PC members really all that different from the current system of PC members and sub-reviewers? When PC members get a paper from outside of their area, don’t they usually give it to an expert to sub-review (and isn’t that essentially the same as a “light” PC member)?

    If your reviews were really outrageous, consider posting them publicly on your blog (or on “”). It would at least shame the committee/PC chair for allowing bad reviews. Maybe, if the community saw how bad the review process is, more people would agree with you that there is a need for change.

  2. I fully agree that we should have anonymous submissions, and I also agree that in recent blog discussions, the reasons for anonymous were much more substantial than the “reasons” that emerged against anonymous submissions.

    But you suggest this partial anonymity and the question is, what is going to change during the discussion? You are having a discussion about a paper and you are still undecided and then you reveal the authors and this “helps” you make a decision. Why should it help? If you can make some evaluation anonymously, why introduce identities at all? There is something really wrong if we think that knowing the identity halfway through is going to help.

  3. Jon,

    Regarding your radical suggestion: you were on a two-layer Oakland PC last year, weren’t you? How did you find the experience?

  4. More regarding the two-layer PC:

    – one reason it is better than the current find-your-own-reviewer system is that this way the “light” PC members get explicit credit.

    – area chairs: WWW explicitly divides its committee into two layers, and both layers are divided by area. The area chairs then discuss the final set of papers. This is different than the system you are suggesting, but similar in spirit.

  5. @anon1:
    I think there are several differences between heavy/light and PC/subreviewer: first, some form of quality control. (How is a PC member outside the area going to choose a gd subreviewer in the area, other than always picking the top name in the field?) Second, a greater sense of responsibility for the light PC member. Third, as Adam notes, recognition. Fourth, the light PC member would see several submissions rather than just one.

    I think there are times when one can make the argument that knowing the authors should make a difference. (I’m not sure I buy the arguments, but they can’t be dismissed out-of-hand.) Anyway, it’s mainly suggested as a compromise solution.

    The heavy/light distinction on the Oakland PC was different. (Light PC members read fewer papers, and did not participate in the meeting but did participate in the discussions.) I thought it worked well.

  6. What is your feeling on author responses? In my personal experience, I have felt that they help improve review qualities by a great deal.

  7. @AA:
    I think author responses are a good idea in theory, but I don’t see how to implement them in any reasonable way.

  8. I like the idea of a larger PC. The heavy and light classification is an interesting way to keep the PC meeting manageable.

    I am not so sure about the anonymous submissions part. I do not think there is any noticeable bias in the community against people who are not well-known. Any (possible) bias against or for well-known people, in my opinion, will not be fixed by anonymization: it would be impossible for an area expert to not know. To rephrase, I think the primary benefit of anonymization is that if no one has ever heard your name, your paper will still get a fair reading. I think this bias-against-unknown is largely absent in our community.

    The cost of anonymization of course is non-trivial. When we send out subreferee requests, the author is often the most natural experts. So we would need some system in place to make sure I don’t get to review my own submission. It would probably translate to more work for the PC chair (Incidentally, such a system would often also lead to de-anonymization). There is one instance when X’s (anonymous) submission to a conference was rejected because a referee pointed out that most of the results in this paper are not new and will appear in X’s PhD thesis.

    I think to some extent, the difference in opinion arises from the question what the goals of a conference are. If you think the conference’s primary goal is to disseminate research fast to help in progress, then anonymity is a burden and it may be useful to take the author’s identity into account when deciding on the paper. If you think the goal is to signal what the best papers are, in a fair manner, then anonymity helps slightly with fairness.

    While I agree that there is a large prestige associated with FOCS/STOC, and hence fairness seems like a necessity, I think what we should be trying to fix is the large importance people give to a FOCS/STOC paper.

  9. I think anonymity helps in the cases that a result has not been announced: no talks, no arxiv, no webpage posts. These are usually borderline cases when having a bigname author makes a big difference, and anonymity would not allow someone to use their big name. In this case, an author would have to make a choice to announce the result before hand (which usually only occurs when they are confident about the quality of the result) or risk not obtaining the extra benefit accompanying the big name.

    I’m not sure why last anonymous categorically states that anonymity would not help anyone and that less-known authors are not hurt in our field. In my PC experience in conferences with non-anonymous submission, I have definitely seen it help big name people in instances when they have submitted mediocre papers that would have been much less likely to be accepted had at least one of the authors not been famous.

    In any case, Jon Katz, since you are in Crypto with anonymous submissions, what do you think about the complexities of anonymous submission?

    Also, to the last anonymous, perhaps there is a link between not having anonymous submissions and the prestige of Stoc/Focs? A possibility, at least? Big name people are more likely to publish there, since they are more likely to get even their mediocre results in and therefore it becomes more prestigious.

    Also, to JK, you say there are “some cases” in which anonymous submissions are necessary. Can you elaborate? Do you mean there is a black list on some people who always make errors? If so, hopefully those people have full information about their status. In any case, it means that the papers are not being read and actually understood and fairly evaluated if the author name is being used in the process.

  10. I’m not sure why last anonymous categorically states that anonymity would not help anyone and that less-known authors are not hurt in our field.
    I said that in my opinion, there isn’t a significant bias against unknown people. Let me elaborate on that: I am not claiming that given the same borderline submission one with a big name and one without, the outcome would be the same. What I am claiming is that given a high quality paper (say something in the top half of accepted papers), the outcome will very likely be the same.

    There is some randomness in the case of borderline papers, and there are biases. Anytime humans are making a subjective decision, there are bound to be biases of all kinds. The name of the authors is one of the causes of bias and a case could be made that it would help to make the submissions anonymous to alleviate this slightly. Though if one were to look at your example:
    I have definitely seen it help big name people in instances when they have submitted mediocre papers that would have been much less likely to be accepted had at least one of the authors not been famous.
    This would not be fixed by anonymous submissions since a big name would, if they needed to, announce that they wrote the paper.

    This occurred in the systems community. When they moved from non-anonymous to anonymous submissions, the fraction of big-univ papers dropped in the first year. In a few years, it was back to its previous levels. You could say it was because they figured out how to tell the committee about the result beforehand, or because they learnt to write their papers better so as to be more compelling. So I don’t quite buy that a big name would not make the result public beforehand: if they are not embarassed by the quality of the result when it’s published, why should they be embarassed to put it on their webpage when it’s in submission?

    But if we step back and ask what the goal of a conference is. It is not, in my opinion, a competition where the top 60 papers are awarded talk slots. The goal is to advance research at a fast pace. With this objective function, it is important to accept the top 30 papers. But if there is a large number of borderline submissions, any arbitrary rule to pick amongst them is as good an approximation as any other. There will always be biases (non-native speakers cannot write well; some areas are more popular than others; the committee members like of dislike a particular line of research). One could try to come up with a perfectly fair system for reviewing where none of these biases matter and the only thing that would determine acceptance is the quality of the result itself. If it didn’t cost anything, it would be good to do that. But I haven’t heard of good ways to even do anonymization at low cost when committee members have to depend on subreviewers. And if the only benefit is a little more fairness for borderline papers, is it worth the cost?

  11. From my experiences on various PCs, I do believe author-name bias exists but I am happy to say that I think it is small (and may not represent a serious concern). Nevertheless, I think even the appearance of fairness is important here, and one could make the case that removing (or reducing) even a small bias is worthwhile.

    I don’t understand any of the arguments against anonymized submissions based on the difficulty of implementing the process, given that many conferences already do it!

    Reasons given to have non-anonymous submissions usually include things like: if the author is known to make mistakes, the committee can read the paper more carefully; if the author is very careful, the committee doesn’t have to check the proofs; and if the author thinks a problem is important, then it must be. I don’t really find any of these compelling (I think a paper should stand on its own) but could imagine extreme cases where a result is important but difficult to verify, and one might want to take the author name into account. (I’m still not sure I agree here, but that goes to a larger debate about whether conference acceptance is supposed to indicate some degree of assurance that the result is correct.)

    I agree with the anon who said “I think what we should be trying to fix is the large importance people give to a FOCS/STOC paper.” It is not clear what is the inherent value of publishing most crypto papers in STOC/FOCS, other than the line on one’s CV.

  12. Hi Jonathan,

    I think you’re raising many interesting and important issues here, but I just want to comment on one for now.

    > It is not clear what is the inherent value of
    > publishing most crypto papers in STOC/FOCS,
    > other than the line on one’s CV.

    There are actually two issues here:

    * To what extent should we judge a crypto researcher based on the number of STOC/FOCS papers he/she has?

    * Should the “best” papers in foundations of cryptography be published at STOC/FOCS, and if so, what fraction of a STOC/FOCS should be dedicated to foundations of crypto?

    I don’t want to discuss the first issue here, but as for the second, I think the answer to the first part is a resounding “yes”, for two reasons:

    * Historically, many cryptographic concepts have rich impact on all of theory … think interactive proofs, pseudorandomness, zero knowledge, and I think it is important for any theoretician to have a rudimentary knowledge of basic cryptographic concepts, in addition to basic notions of approximation algorithms, complexity classes, etc, etc.

    * Good work in foundations of cryptography is good theory, in and of itself, even if there is no obvious connection to complexity or algorithms.

    At the end of the day, I think the fraction of the FOCS/STOC program that covers a certain topic reflects how much the theory community cares about – and should care about – that topic. Maybe I’m over-reading things, but implicit in a program that has twice as many game theory papers as crypto papers is a message either that crypto isn’t important in theory anymore, or there are no longer many interesting/exciting theoretical advances in foundations of cryptography, both of which are quite disturbing.

  13. Let me just clarify that as members of the crypto community, we all play a role — as subreviewers, in interacting with colleagues in other areas at conferences, workshops — in determining the representation of crypto papers in FOCS/STOC.

    This begs the question, are we being too harsh on ourselves?

  14. Many of the crypto papers in STOC/FOCS come from a small subset of people who publish the majority of their papers in STOC/FOCS. Do these people have extra power in that they are always asked to review papers? For them, it works to be harsh, if they want to keep their exclusive status. So maybe some people are being harsh, but if so, it is likely in someone’s best interest.

  15. By the way, serious discussions of changes in how PCs operate are very common between PC chairs and the conference organizing committee. There is a brand new SIGACT executive, which was voted on by more of the community than we find at any one conference and they may have ideas for changes. (Though, as with the comments on these blogs, I don’t see any unanimity among the new members on the merits of specific changes.) Obviously if we had any form of allowing PC members to submit, we would need to get lots of input at least at the level of a conference business meeting.

    I assume that for the two-tier PC that the top tier would meet face-to-face and the other tier would not. I also would suggest if this version were adopted that while the second tier PC members could submit papers, the top tier could not. (This variant of what you suggest gets around the problems of cost and unwieldiness of large committees that you don’t mention as an objection to larger PCs, as well as the ones you do.) There is a small difference between a second-tier PC and the external reviewers we use now: the second tier PC members would be able to participate in the online debate on the papers they are assigned, whereas external reviewers have write-only behavior unless their associated PC member asks them to comment again. However, I don’t see that this is such a large change in the final analysis. The major saving is precisely that some papers would end up being read (or shepherded) by 2 or fewer top tier PC members. (I am not sure that this is a good thing.)

    The logistics of a PC chair getting lots of people to serve on PCs are non-trivial. One advantage of the sub-refereeing process is that the PC members rather than the chair are making the requests and each request is a small one. If we had a tradition of 2nd tier PC membership then it would be much easier to say yes but that is still many more people to ask and there will be a potentially difficult transient.

    BTW: As a PC chair, one typically will send out papers to outside reviewers oneself if there is a clear lack of expertise on the committee.

    I must say that I am impressed that you got 6 reports on your paper, even if a couple seem to be off base. Given the default of 3 PC members per paper, This suggests that members of the PC went out of their way to try to get additional expertise .

  16. I think JK had two papers, hence 6 reports.

  17. I personally agree with all of Jon’s suggestions, though I am not very sure how the heavy/light PC idea would work in practice. I know however that it is implemented in other areas, like data mining. For example, their top conference KDD has a senior PC and quite large ordinary PC (

    As for anonymity, I feel very strongly about it, as do most of the cryptographers. We are used to have anonymous submissions in crypto conferences, and having non-anonymous submissions in FOCS/STOC cause a cultural clash which for quite a few of us is not easy to mend. A few years ago there were some trials to have non-anonymous submissions in Crypto/Eurocrypt and they failed – mostly because people just did not like them. See for my old blog post.

    Having non-anonymous submissions makes the whole process more fair, which is important for many reasons.

    As for the importance of FOCS/STOC in crypto – I always check the list of accepted papers in FOCS and STOC. Sometimes there are some important papers, sometimes there are not. In particular, for a cryptographer it is not always understandable why were *those* papers seen to be relevant enough and not some others. (May be there were no more submissions?) As an example, I think the best 2 papers from this Eurocrypt were by Kiltz & co about efficient public-key encryption. I am not sure if this topic would even be considered in FOCS/STOC community as “interesting”.

  18. I agree with Helger on that best crypto/eurocrypt papers of the year are not necessarily relevant to stoc/focs. After all cryptography as an area has both aspects: theory, and applied stuff (like breaking MD5) and it is not fair to ask stoc/focs to consider everything.

    So indeed, only those papers on cryptography that have got enough “theory” content in it (as Hoeteck pointed out), should be submitted and considered at stoc/focs. That being said, I do see cryptography papers appearing in stoc/focs even when they do not contain as much of “theory” content (I want to remain discreet and hence won’t call them out here).

    But since cryptography as an area involves many people from TCS, and many a times some technically weak papers are seen in crypto/eurocrypt fairly regularly compared to stoc/focs — there is a big +ve bias (in my opinion) towards a CV with more number of stoc/focs papers unless the person stands-out. A situation that needs to be corrected — as most pointed out.

    For anonymity, I personally always liked anonymous review system. But over the past few years, from personal experience, I can say that knowing who is the author of the paper helps in deciding a take on the paper. The help that I get is precisely what Hoeteck pointed out: it is not possible for the reviewers to sit down and verify all the details of the papers and their correctness. As a community, this responsibility is more of the authors responsibility than the reviewers’. A reviewer will typically be able to provide good review about how important the problem is, is the paper well written, do the proofs seem correct, does the paper mark an advance in state of the art, and so on. BUT in some cases, the proofs are just too complicated (either bad write-up, or just a plain involved-proof) — and its not worth sitting down and verify all the details of every damn paper that you get. It can become quite painful.

    At such a stage non-anonymity helps and reviewer would typically show more confidence in his/her review when it comes to comment on the technical correctness of the paper. Of-course this leads to some unfairness to non-famous authors since reviewers will tend to be unsure/negative about their paper regarding the technical correctness. But it seems to me that such a bias is not that significant in our community, and I would prefer to make the life of the reviewer somewhat easier by giving out the author information (specially considering the number of papers we produce every year these days as a community).

    Of-course it sucks to find out that the reviewers did not read (or were not even qualified to read) your paper to provide a useful review but I think this concern is reasonably orthogonal to the anon/non-anon submission process: how can we incentivize the reviewers to provide a better review (rather than the mere unsaid-understanding that its their responsibility)?

    So, personally I would not want the submissions to be completely anonymous. I think the “hybrid” solution that JK suggested may be a reasonable way to go about it.

  19. Since the comments having to do with anonymous submissions mainly refer to stoc/focs/soda (rather than crypto conferences since they are already anonymous), I could point out many well known authors who have made major errors in stoc/focs/soda conferences such that their results are completely invalid (but never publically admitted the error). Some time back someone pointed out the Fields medal winner who claimed to solve (i.e. improve on log n) sparsest cut in the early 90’s and was published in STOC. That’s just one example. Being famous does not make your results correct!

    So this reappearing argument that non-anonymous submissions are good because they gives confidence in correctness of certain papers is bogus. How many papers out there have never been read and contain undiscovered errors? Surely some of those papers have famous authors. The bottom line is that this is not a scientific review process if we simply assume a correlation between correctness and fame and don’t bother to verify the results.

  20. well said hoeteck. i do think part of the problem is that we are being too harsh on ourselves. my understanding is that if a paper gets all round positive reviews, then it would be accepted regardless of the area biases (if they exist).

  21. Some time back someone pointed out the Fields medal winner who claimed to solve (i.e. improve on log n) sparsest cut in the early 90’s and was published in STOC. That’s just one example. Being famous does not make your results correct!

    I think this is an excellent example. But I think so for reasons opposite to yours.

    Suppose we get a submission giving a constant factor approximation to the sparsest cut problem. Further the proof is complicated enough that it cannot be verified or shown to be wrong in the 45 days that the committee has. The committee tries hard to understand it, sends it to several subreviewers who might find a hole in it, and in the end concludes that a) there is no hole we could find, b) no one was able to certify this as correct, and c) the approach is not implausible.

    The question then is: what should the committee do. The committee could simply decide to outright reject any paper they could not verify. This is the fair policy since the committee does often find bugs in papers (and often even declares a correct paper buggy) and rejects them based on that, so why should my bugs be reject-worthy and somebody else’s bugs be accepted?

    But I think that is extreme and such a policy would result in several very good papers being rejected. It is in fact absurd to expect that every focs/stoc submission can be fully verified in 45 days. Moreover, if we wait for the author to come back with a more understandable proof in a year, we have just slowed down the pace of research. Let us analyze the ups and downs of accepting such a paper. If the paper turns out to be correct, it will be a major breakthrough that would spur lots of further work and fundamentally change our view of a large class of problems (this is what ARV ended up doing). If it turns out to be wrong, we would have rejected one borderline paper that we could have otherwise accepted. So if our belief abour correctness is above some threshold, I think the expected upside exceeds the expected downside. The threshold itself depends on the importance of the result.

    The belief itself is a function of a lot of things and I think the author’s identity is one of them. It may not be the fair, but I do not think fairness is the only (or even the primary) criteria that a conference should seek to optimize.

    So I think the decision the committee took at that point seems to me like the correct one given the information they had.

  22. I would be interested in knowing what YOU as a PC member would do in the following situation:

    1. There is a paper at hand, claiming a cool and important result.
    2. The paper is well written but very involved.
    3. You spent about 40 hours on this paper (spread it as you’d like over 15-20 days)
    4. In this time, you were able to convince yourself about half of the lemmas in the paper, but there are important lemmas left that you were unable to verify. (due to time, proof-complexity etc)
    5. You’re supposed to take a decision: ACCEPT or REJECT.

    What would your decision be in each of these cases:
    (a) You do not know who the author is.
    (b) You know that the author is someone who has written excellent papers in the past (sometimes singlehandedly). In addition, YOU do not know if he has ever written an incorrect paper.
    (c) You know that the author is some who writes reasonably good papers. You also know that he has written incorrect papers in the past.

  23. Jon said:

    :it could ensure that each area of TCS has at least two representatives on the committee.

    I think that that´s a wonderful idea. Of course, enforcing this requires having an exhaustive list of the “areas of TCS”. Jon, would you mind providing us with such a list?

  24. i think by each area jon means each major area of tcs.

  25. Some time back someone pointed out the Fields medal winner who claimed to solve (i.e. improve on log n) sparsest cut in the early 90’s and was published in STOC.

    I’ve never heard of this example before. Care to elaborate?

  26. The paper is called:
    A near optimal algorithm for edge separators (preliminary version)

    by Fan Chung and S.-T. Yau from stoc 1994

    Apparently many people from that era (e.g. attendees of stoc 1994) claim that they knew the paper was wrong because they knew the approach did not work but they were not asked to review the paper. Probably such a paper, if it were placed on arxiv, could have been refuted quickly since it claimed to solve an important problem that many people were trying to solve and therefore would have attracted a lot of attention and scrutiny.

  27. To those who think having a warm and fuzzy feeling about papers that have famous authors: this is exactly why I am against nonanonymous submissions. The acceptance should be based on the correctness of the paper only. If you cannot verify it, then reject it and let the authors to improve on the paper. This is done regularly in more journal-oriented areas. It would also not stall research since the authors can always publish the paper first on eprint.

  28. The acceptance should be based on the correctness of the paper only. If you cannot verify it, then reject it and let the authors to improve on the paper. This is done regularly in more journal-oriented areas. It would also not stall research since the authors can always publish the paper first on eprint.

    I do not think conferences are supposed to be correctness-driven. There are cases when even an expert cannot verify the paper in the given time frame, and here each PC member has a much larger responsibility. Larger committees can alleviate some of that, but I don’t think we can get to the correctness level of journals. In fact in many areas, conferences are for presenting approaches that may or may not pan out.

    Requiring that papers be verifiable-with-very-little-effort also puts a strong bias against areas where the machinery is complicated and verification is time consuming (e.g. PCPs, crypto). In fact I can think of many important papers that would not appear in focs/stoc if this were the case.

    Besides, as has been pointed out, non-anonymous submissions do not remove bias-towards-famous-people.
    In fact, there is an inherent contradiction in your comment. Either people would put papers on the eprint, in which case submissions don’t remain anonymous for any practical purposes (and I do not believe making submissions anonymous even removes the appearance of bias. If my paper gets rejected, and big-name’s paper gets accepted, I will still feel it was because big-name de-anonymized himself by putting the paper on the arxiv). Or they don’t in which case research is in fact stalled.

    Practically, I think people do not have a strong incentive to put papers on the eprint, but do have a strong incentive to submit to focs/stoc. I personally have been in situations where I would like to work on questions related to a submission I referee but have put it off since the work is not public yet. So yes, while in an ideal world, everyone will put results on the arxiv as soon as they have them, I do not think this happens in practice and the conferences do in fact help speed up research by publishing papers quickly.

  29. I think its possible to evaluate correctness separately from importance of the paper in most cases. In the review form, the reviewer could be required to give a confidence score in the correctness of the result (this is already being done). If it so happens that reviewers think that the result is important enough to be accepted but they don’t have good confidence in correctness, the paper could be de-anonymized by the program chair and the author names could be taken into account.

  30. With “very important results”, authors should simply put them on arxiv or eprint so that they can be refereed by the community. These type of results (e.g. results that people immediately see to be important and that may be very technical and hard to verify) benefit very little from the pedigree of the conference in which they are published. If Braverman had not published his paper in CCC, but just left it on arxiv, would it have mattered for the importance of the result? Probably not.

  31. As I see it, there are two cases here: the first, where the result is too difficult to verify in a short time-frame; the second, where the result is so badly written that it cannot be verified in a short time-frame. I am not very favorably inclined to the latter — we have enough badly written papers as it is, and I don’t see what purpose an unreadable paper claiming a great result serves. I might be willing to give more leeway to the former, but how many papers really fall into this category?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: