I recently noticed this feature of Physics Overflow that there is a tab called "Submit a Paper". What it essentially means is that one can submit a paper that has been previously published (either on arXiv or otherwise) for review. People post their reviews as well as upvote or downvote the paper on two parameters separately - originality and accuracy. We can change the details, of course.

I find this feature attractive as I think that not only the reviews expressed there will enhance the experience of reading the paper and getting more physics out of it but it can also give junior researchers or students some insight as to how a seasoned eye reads a paper and how to write better papers.

I know it is not an aim of Physics SE to train people to write good papers but as I said, the major benefit is the enhanced experience of reading a paper and getting more physics out of it. The secondary benefit would be to learn how to analyze and write papers. A third benefit would be that it will create a very good repository of interesting papers. Can we introduce this feature in Physics SE?

  • $\begingroup$ This seems like if it should be a site, it should be separate - Area 51 takes new site suggestions. $\endgroup$
    – auden
    Commented May 25, 2017 at 21:28
  • 1
    $\begingroup$ I don't think this would work as a new site if we intend to do it for only Physics papers. I think it would be better as a feature of Physics SE itself. Do you think it would be derogatory to the main site in some way? $\endgroup$
    – user87745
    Commented May 25, 2017 at 21:32
  • $\begingroup$ I feel like this has been discussed on here before. I'm sure QMechanic will magically dig up the links in about 10 seconds. But, 1) There is no way to implement the feature without programming from the network side, and 2) The focus of all Stack Exchange sites is high quality Q&A from professionals -- nothing more. It's not for reviews, or for conversations, or for anything else. Paper reviewing might be a great idea, but it's not what the site is intended for. $\endgroup$
    – tpg2114
    Commented May 25, 2017 at 21:39
  • 2
    $\begingroup$ Ah, easier than I thought to find: physics.meta.stackexchange.com/q/7516 $\endgroup$
    – tpg2114
    Commented May 25, 2017 at 21:41
  • 1
    $\begingroup$ Can't we expand the domain provided we are doing great at what we intended to do originally? $\endgroup$
    – user87745
    Commented May 25, 2017 at 21:41
  • 1
    $\begingroup$ We cannot, no. Stack Exchange is a business, with a plan and a model. You would have to convince them to be something more/different -- which you could try by bringing it up on meta.stackexchange.com . There's no reason this site should be a one-stop-shop for all things physics. There isn't a Q&A or paper review system on Wikipedia. There shouldn't be a paper review system here. $\endgroup$
    – tpg2114
    Commented May 25, 2017 at 21:43
  • 1
    $\begingroup$ An attempt at something similar was made, and ended up fizzling out: Journal Club $\endgroup$
    – tpg2114
    Commented May 25, 2017 at 21:46
  • 1
    $\begingroup$ Why downvote a proposal like that? Comment/answer why you would/wouldn't introduce such a feature, but downvoting a well-asked question surely leads to nothing. $\endgroup$ Commented May 26, 2017 at 17:41
  • 3
    $\begingroup$ @JannikPitt On meta, downvoting means "I disagree," which is slightly different from what happens on the main site. It also doesn't help/hurt/count in any way. That said, it's also possible who disagree (and downvoted) did not leave a comment or an answer because it was already covered by the comments above or the answer below. $\endgroup$
    – tpg2114
    Commented May 26, 2017 at 19:21
  • $\begingroup$ @tpg2114 I didn't consider that, I totally agree! $\endgroup$ Commented May 26, 2017 at 19:30
  • $\begingroup$ @JannikPitt No worries -- it's always a little strange the first few times somebody new jumps into the Meta side of things. It's a lot more... loose with the rules to facilitate discussion. Hope you stick around! $\endgroup$
    – tpg2114
    Commented May 26, 2017 at 19:34
  • $\begingroup$ Aside from the technical issues, the philosophical issues, and the issues with number of users for this; if all those problems were solved, we still couldn't do this for one reason. Votes. We have a system where any user can upvote and (to some degree) downvote for whatever reason. That means interesting but awful papers would be upvoted, regardless of our voting policy, while good but controversial papers would be downvoted. Furthermore, it opens up the judging of a review to many who, quite frankly, are unqualified to judge a review. How could we trust if a review/paper is good? $\endgroup$
    – Jim
    Commented May 30, 2017 at 12:00

1 Answer 1


To be blunt: no. We don't have the numbers to do this effectively, not by an order of magnitude at the very least, we would be actively harming other efforts to do similar things, there's no guarantee that it can even work to begin with, and the proposal is moot unless you can come up with someone to write the code for it.

Let me start with the more prosaic side: there's just simply no one to write the code. The kind of feature you're proposing entails, at the very least, many months of dev-team time, and you can pretty much take it as a given that Stack Exchange won't provide it. From one perspective, they are a business and they can only invest in things with a revenue stream associated with it, but to be fair they have been really awesome at building high-quality software mostly in the spirit of making the internet better, and finding non-intrusive ways of monetizing it a good while after it's been built.

However, SE has a very clear mindset and anything that takes away time and energy from the core emphasis on Q&A (beyond the jobs site, which does come with a healthy revenue stream) has been politely but firmly declined. So: unless you have a workable proposal of how that software is going to come into existence, the point is a non-starter to begin with. But let's ignore this and focus on all the other reasons this cannot work.

As I said, and as I've said before, this community simply does not have the numbers to implement this at any kind of scale where it would begin to become self-sustaining. Reviewing a paper takes a lot more work than bashing out an answer on the Q&A main - I can think of maybe two or three answers that have taken longer than a referee report. Writing a review is not a trivial task: it requires that you thoroughly understand the paper, a clear vision of the field, and a commitment to doing it fairly. In my experience doing one in less than one full workday is a pipe dream.

In addition, reviews also have a much reduced field of potential answerers: the proportion of site users that can write useful reviews is small to begin with, and moreover the scope of topics that each user can review in is smaller than the topics they can answer in. This leaves you with very few people, whom you want to ask for a lot of work - and it's just not going to come together in scale.

You mentioned PhysicsOverflow, and indeed they have gone out and built the software for that platform. Do they have the numbers? That's something for people to judge for themselves, but I don't think they do - at the moment I count 22 reviews out of 120+ requests, and I would argue that to the extent that they have managed to build a reviewing community, it's because they have followed in the footsteps of the old Theoretical Physics SE site by focusing exclusively on a rather narrow stretch of QFT and related topics.

More importantly, though: doing a half-hearted attempt at this kind of thing is actively harmful, and it detracts energy and attention from other projects that are trying to do pretty much the same thing. What you're talking about is called open peer review, and it's been a huge part of the conversations on open access and reforming the peer review system over the past decade or two. It's not a new idea, and the reason it hasn't come together is that it's a hard problem to crack.

There's a relatively large number of initiatives regarding open peer review, though most of them are tied to specific journals. However, the platform you propose is pretty much precisely (part of) the platforms over at SciRate and at arXiv Analytics (though the latter doesn't look very healthy at the moment, unfortunately). Those platforms put the peer review at the place where it's most natural - next to the papers themselves. Doing it on SE would have a strong disconnect from that infrastructure, making the reviews less discoverable, which would in turn make the platform less useful.

We've previously run community ads for both arXiv Analytics and for SciRate (see here and here for their 2016 incarnations). I'm not sure why there wasn't a 2017 version of the SciRate one but it's now up; you can re-post the arXiv Analytics one but frankly the site does not look healthy right now. If people find similar platforms I'm always happy to cook up community ads for them.

Seriously, though: this is a tough space to build platforms in, and this site is pretty much guaranteed to fail: moreover, if it does, it just adds one more ghost town to the mix, one more precedent of how it's just not doable and how we should just stick to the good old closed peer review, and one bit less enthusiasm in reforming scientific institutions that really are in very dire need of reform.

And, finally - would this kind of site really be as useful as you think it would be? In large part, what makes this Q&A work well (and also what makes traditional peer review work, too) is that there is a relatively clear audience that one is writing for, but that's a lot more hazy with the section you propose, which means that the reviews get harder to write and they become less useful to read.

Is the goal a review-on-demand kind of marketplace where Bob needs an expert point-of-view on paper X and requests it? Even if you could get that to work, I strongly suspect that if Alice comes along and writes that review, by the time Charlie comes around with questions on that same paper, they will find Alice's review to be suboptimal.

Alternatively, is the goal to build out a repository of good papers? Frankly, that's looking at it backwards - if you want a good collection of papers, find an antology or a review or a book. Or are you building a repository of nice reviews of nice papers? If the papers were already nice, why are you reading the reviews and not the papers? Or are you looking for a good collection of reviews that take apart the section of the literature that's incorrect and patch it up, thereby fixing science? This might sound hyperbolic, but I frankly don't understand what sort of value would accrue to that collection of reviews, who would go in to browse that collection, and what they would get out of it.

And, if what you want is just to get a nice analysis of how a paper is written and lessons to learn from them to write your own papers later - then frankly the first port of call is a critical reading of the scientific literature you yourself consume, but more generally what you're looking for is a blog post, from the wide variety that are already out in the physics blogosphere, so go out there and look for them!


You must log in to answer this question.