10
$\begingroup$

What peer review style questions are acceptable to ask on Physics SE and what actions, if any, should be taken against unacceptable questions?

When I originally encountered this question and its accompanying comments (note the question has since been edited), the asker appeared to be making clear the only acceptable answers for the question were ones that would require a full peer review of an older article, a second paper linked in the post, or both. The linked question may have other issues that I do not intend to address with this meta question.

Originally of the belief that there was likely a rule against such a request, I searched the Help Center in vain for something that mentions it. I asked an active user in Physics SE chat for an opinion, and the response suggested ambiguity in the actions that would be appropriate in general.

The linked question was eventually put on hold as "unclear what you're asking." A comment by one of the closers states it is because the question should be self-contained, a problem any peer review style question would almost certainly share. A later comment on the same question states that Physics SE is not a substitute for peer review.

I will produce some example fictional questions with commentary here to illustrate what I'm asking about and the variety of questions that could elicit varying responses.

Note that it seems likely questions of this type will overlap with non-mainstream physics questions, which are deemed inappropriate according to the Help Center. In these example questions I will assume they are asking about mainstream physics, but it is possibly worth describing how they would be dealt with differently if this is not the case.


  1. I recently encountered [this article] online, is it correct?

This question could be everything from a misunderstanding of the situation to a literal request that we review the article.


  1. I recently encountered [this article] online but I do not fully understand [specific part of article], is it correct?

This question improves upon the previous one by asking about a specific part of the paper, but it's unclear whether or not it is appropriate to expect site members to go through the entire paper to address the concern.


  1. I believe [this article] is incorrect because I found [another article] which appears to contradict it. Who is correct?

This question could simply be an underlying misunderstanding of the asker, but the asker insists answerers address the specific paper claims.


  1. I believe [this article] is incorrect because [outline of reasoning addressing a specific section of the paper]. Am I right?

This question could simply be an underlying misunderstanding of the asker, but the asker insists answerers address the specific paper claims.


  1. I recently submitted [a paper] to arXiv and was wondering if the community could comment on my claims.

  1. I recently submitted [a paper] to arXiv and was wondering if the community could comment on my claim about [topic]. Here is a brief overview of it: [long post detailing claim]

This is certainly not an exhaustive list of what types of questions could be asked, but is merely intended to be a small list of examples illustrating variety.

What is the opinion of the community for how peer review questions should be dealt with and why? How should I report the ones deemed inappropriate? Should there be any uniform approach here?

$\endgroup$
20
$\begingroup$

Generally speaking: no.

  • This site is very definitely not a replacement of traditional peer review. We simply do not have a community that is broad and varied enough to connect a given paper with an expert that is sufficiently qualified, interested enough, and with enough available time, to review it.

  • This site is also not a very good venue for peer review, generally because it leads to a much broader discussion than the format can really accommodate.

    For one, there's a mechanics issue: peer review requieres back-and-forth discussion over a long list of specific issues, which the Q&A format does not handle well at all, and the discussion will be left as a sorry mess of interlinking questions, answers, comments, edits to the question, edits to the answers, comments pointing to revision histories, more edits to the answers, and so on. This is completely opposite to the purpose of this site, which is (roughly) providing a useful resource for future visitors which can be parsed immediately without going through ten pages of forum discussions.

    There are also a bunch of actual problems, which depend on the specific way the request is phrased. So let's go over your examples.


  1. I recently encountered [this article] online, is it correct?

This is much too broad. It can depend on the specifics of the question (particularly if it's halfway between here and type 2) but in general these should be flagged/close-voted as Too Broad.

There is a reason why we close things as Too Broad, and it is that they are particularly unrewarding to answer. A question that is Too Broad as posed tends to require enormous amounts of work to answer (more commensurate with a full paper or even textbook than a simple ~blog post) and it is beyond what you really should be asking of strangers on the internet.

More importantly, though, if someone does try to answer, there is a definite danger that the OP will not be satisfied: "you did all of that work over on that side, but I was actually interested on that bit over there", so that huge amount of work comes with little reward.

This is why we require that questions be posed in a way that an answer can be recognized as such, and not as soliciting essays on a given topic.


Moreover, depending on the tone, this is a very tricky question to ask, because we're not here to engage in long-winded discussions. In particular, if it appears that the OP already has made up their mind about the topic and they are only here to stir up controversy (as was definitely the case in the posts you link to, but with several milder variants also possible) then the thread is a non-starter.

We are here to respond to specific, answerable questions, not to engage in long-winded back-and-forth discussions which can be incredibly draining of time, energy, and passion for engaging with random strangers' curiosity on the internet. Questions which are a bait-and-switch, pretending to be the former but obviously hiding a huge chunk of the latter, are very identifiable as a drain of experts' time and energy which should be spent elsewhere on the site.

In these cases, the close/flag reason Primarily Opinion Based applies.

  1. I recently encountered [this article] online but I do not fully understand [specific part of article], is it correct?

This is much more workable. Depending on how specific the post is it could still be too broad, but in general that just describes a bona fide question about a paper in the literature. Prospective answerers know what are the concepts that need clarifying, and if there are obvious issues with the work they can be brought out.

  1. I believe [this article] is incorrect because I found [another article] which appears to contradict it. Who is correct?

As posed, this can be a legitimate request for an evaluation of contradicting literature. However, there is very little we can do here: depending on exactly how this is phrased, it amounts to a request of full peer review of not one but two papers, and this is really beyond our scope.

In particular, there's very little chance that we'll be able to produce anything near definitive - our community is simply not big enough - and providing inconclusive answers for the consumption of the internet does very little to set the record straight and much to cloud it up.

Unless it is very obvious that there is an objective answer (e.g. the second paper makes a very compelling case, with only a very weak response from the authors of the first one), this is probably over the Primarily Objection Based line.

  1. I believe [this article] is incorrect because [outline of reasoning addressing a specific section of the paper]. Am I right?

This reads as a bona fide, self-contained question: is the argument the OP laid down correct, or not? (Unless, that is, the OP outsources the [outline] to some other journal paper, possibly of their authorship, in which case 'self-contained' obviously does not apply.) Prospective answerers are given a specific thing to address, with sufficient context, so this is probably a good use of the site and its resources.

  1. I recently submitted [a paper] to arXiv and was wondering if the community could comment on my claims.

This is not a question. The post solicits essays and opinions, and there is no criterion at all for when an 'answer' is 'correct' or not. This is textbook flag/closevote as Too Broad.

Moreover, with questions of this type, it is generally unclear what it is the OP actually wants from the post. Do they want community acceptance to rain down on them? Are they soliciting expert opinions about the correctness or significance of the paper? If so, exactly which experts, and which sections of the community, are they hoping to engage with, and what guarantees do they have that they frequent the site and will see the paper? (Again, we're not big enough to undertake this sort of thing.) Are they expecting problems with their paper to actually surface? (And, if so, will they react in a civil way once they do? Is that something prospective answerers can count on just from reading the question?)

  1. I recently submitted [a paper] to arXiv and was wondering if the community could comment on my claim about [topic]. Here is a brief overview of it: [long post detailing claim]

If the post does actually detail the claim, then this passes the self-containment bar, but it's still not really a question. What is it the OP wants? cf. the above, and again Too Broad for our format.


A few extra comments:

  • The business of bringing together manuscripts that need review with referees with enough time and expertise to review them is a Hard Thing. Academic publishing has sort of kind of got it down to a reasonable standard (but even they still fail often enough at it). At least, they've got it to a standard that has a kind-of acceptable rate of letting bad stuff through, and they've reached some form of equilibrium between the spectrum of (always take the literature with a grain of salt) / (if it's published it at least had a qualified set of eyeballs on it) versus how much energy the community can spend refereeing work.

    There are all sorts of problems with the current peer review model in academic publishing (though physics seems luckily to be less troubled than medicinal chemistry and psychology) and there are all sorts of problems with academic publishing in general. Non-commercial (or small enterprise) academic publishing has done a good bit over the past decade pushing this process to a place where the peer review does only require a talented and well-connected volunteer editor (without the overhead of an editorial office), but even then it's not perfect.

    Hoping that just putting a random collection of internet strangers from all over physics next to a random collection of manuscripts about all sorts of topics will somehow manage to work, and work better than academic publishing, has always seemed very strange to me.

  • On another track, we're not alone on the attitudes I detailed above. MathOverflow, in particular, is much better posed as a site for professional mathematicians, and they make it pretty clear that they too are no substitute for peer review, cf. Using Math Overflow to check whether or not a proof is correct, On discussion of published papers at MO, and Appropriate or not: “To what extent are the results in Paper X correct?”.

  • On the other hand, PhysicsOverflow does have an open paper refereeing section, which you can check out and see how it's working. If you're looking to solicit feedback on a manuscript, that's one place to turn.

$\endgroup$
  • 3
    $\begingroup$ Let's not forget the non-mainstream close reason, which was instituted to cover some of these cases. $\endgroup$ – David Z Feb 14 '16 at 11:13
  • $\begingroup$ @DavidZ I never understood what that close reason is supposed to mean. Is LIGO "mainstream"? $\endgroup$ – DanielSank Feb 21 '16 at 5:35
  • 1
    $\begingroup$ @DanielSank imo mainstream physics is the one in standard textbooks on the subject and addressed in peer reviewed publications. $\endgroup$ – anna v Feb 21 '16 at 7:12
  • $\begingroup$ @DanielSank What anna said is basically the way we've defined it on this site. Mainstream physics can be ambiguous, sure, and we've tried (with partial success, I would say) to remove that ambiguity by defining it in terms of peer reviewed publications and textbooks. But there are some things that are obviously mainstream (including LIGO, I would say) and some things that are obviously not. $\endgroup$ – David Z Feb 21 '16 at 10:04
  • 3
    $\begingroup$ Is this really the place for that discussion? (just saying) $\endgroup$ – Emilio Pisanty Feb 21 '16 at 15:57
2
$\begingroup$

I first want to state that, in general, "Am I right" and "Is this correct" type questions are usually terrible questions to ask here because the answer, "yes" or "no," is not long enough to be a valid answer on this site.

I think that many questions that are closed as off-topic can be easily modified (by OP) to be reopened, peer-review type questions are probably among these. The presentation of the question should be clear and specific, asking the very broad "can you review this n-page-long paper" is a quick way to lose rep (downvotes) and get it closed. However, beating around the bush can also be viewed as duplicitous and get some downvotes and a close vote.

  1. Off-topic: it's definitively too broad to be answered here.
  2. Can be On-topic: If the question were can you explain why X is the case here? would be alright. One could do better with, I'd expect Y to be the case, but they argue X the case; why is that?
  3. Off topic: we are not a peer review system.
  4. Can be On-topic: it could be too broad if OP is critiquing too much of the article, but if it's a small enough section I'd think it okay (though, for similar reasons as 2., it could use some work).
  5. Off-topic: it's basically spam and hardly about physics
  6. Off-topic: as written, it's not a question and what question could be there is the terrible type of "am I right" questions mentioned at the top.

So I'd say it's probably safe to say it's likely that peer review type questions will get closed as off-topic, but there may be ways to get around that. It definitely will require the OP to present the arguments the paper gives in detail and not require a potential answerer to download & read a paper. And it will require asking a specific physics concept about that section (e.g., "How is this claim made?").

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .