14
$\begingroup$

Abstract: Analyzing the data from the last replace-the-homework-policy question was inconclusive. So back to the drawing board, or really back to our starting point: what kinds of questions do we really want to see closed? In particular, what are better ways to characterize our actual reasons for closing questions where we currently use the "homework-like" close reason?


Important note, because I guess this hasn't been clear in the past: the purpose of this series of posts is to come up with a brand new set of guidelines for question closure. We are not reformulating or tweaking or improving our current homework policy. We are replacing it entirely - that means we are getting rid of it and developing a new set of policies and close reasons, which will presumably make no mention of "homework" or "homework-like". Once this process is done, there will be no homework policy and no "homework-like" close reason.


As some/many/most people are aware, we are in the midst of a long-term project to replace our homework policy with a new set of guidelines that better reflect what people actually consider on topic and off topic. In the first phase of this project, we collected some example questions and voted on whether they should be on or off topic, to help clarify the new guidelines we want to come up with. I also collected some opinions on various qualities of these questions in a separate survey. The goal was to try to find some correlation between the topicality of the questions, as represented by the score, and one or a few of the attributes, and that would tell us what are the main factors people consider while voting to close.

As it turns out... that doesn't work very well. I spent a long time tinkering with the data and basically what I found is that people's opinions vary widely, making it hard to draw meaningful conclusions. The first thing I checked was whether any individual factor from the survey correlated well to people's impression of whether a question was on topic. There were some correlations with physical context and interest, and to a lesser extent level and effort, and an anticorrelation with tediousness (this means that within our data set, people are more likely to consider questions which concern tedious calculations on topic), but none of these are very strong.

correlation heatmap between factors and scores

I also looked at whether a combination of factors could provide a good discriminator between on-topic and off-topic questions, in a couple different ways. First, a partial least squares regression to test whether some linear combination of ratings would be able to accurately predict the scores of the questions, but that didn't work at all.

plot of predicted vs measured scores

The other approach, which I think best mirrors how the actual close-voting logic works, is to find a set of factors such that the on-topic questions rate highly on all factors but each off-topic question rates low on one or more of them. This plot shows the mean minus one standard deviation1 of the distribution of ratings for each of the highest-score (most clearly on-topic) sample questions:

mean minus standard deviation of on-topic question ratings

and this is the equivalent, showing mean plus one standard deviation, for the lowest-score (most clearly off topic) sample questions:

mean plus standard deviation of off-topic question ratings

I was looking for a group of one or a few rating factors such that the corresponding columns in the top plot are all red or white, and the columns in the bottom plot between them contain one blue cell for each question. Unfortunately, there is no clear candidate. I even had the computer go through all possible combinations of rating factors to test them out, and all the combinations in which the on-topic questions rank high have large uncertainties in the off-topic questions, and vice-versa. The one factor that does keep popping up is "check-my-work-ness", which may be an indicator that people agree that questions just asking us to check work are off topic and that we do okay at identifying which questions those are.

Anyway, the point seems to be that none of the factors in the survey are a particularly good proxy for how we decide whether questions are on or off topic. (And remember, we're mostly talking about questions which we currently close or might consider closing using the homework-like reason.) So before we proceed, I want to throw the question back to the community in a revisit of the question where I first tried to collect possible close reasons. In light of all the discussion and analysis we've done since then, when we choose to vote to close questions as homework-like, what reasons do we actually have in mind? Or, how could we do more research to work this out?


1The idea is to identify clearly on-topic questions as those for which something approximating the 68% confidence interval of the score is entirely positive, indicating a high score and clear agreement on that score. It's basically the same idea underlying e.g. the reddit scoring algorithm.

$\endgroup$
  • 1
    $\begingroup$ 1) Grat, good work. 2) Yes, people seem to randomly pick out questions and close them. It is not surprising. :-) 3) Even this has an obviously positive effect to the site quality, it does this on the high price of the content/user loss. 4) Thus, it should be done on a more rational way, where an optimal compromiss between lenience and QA is found. Instead of measuring the behavior of the voters (what we can see, it is mostly random), maybe the stats of the returning new users with HQ content could/should be also measured. Although it is more hard to measure, it is still possible. $\endgroup$ – user259412 Aug 27 '16 at 21:28
  • $\begingroup$ @peterh: Perhaps you should write your own answer. $\endgroup$ – heather Aug 29 '16 at 11:16
  • 3
    $\begingroup$ @peterh "Thus it should be done on a more rational way" - not to be all Bill Clinton on you here, but when you say "it" what do you mean? Also, I have to flat out disagree about lenience on allowing non-QA questions. We have to maintain a high standard for questions in all cases. People may find the format troublesome at first, but they will soon find it to be useful in all aspects of their life (and that's not an exaggeration! Knowing how to present your problem in a way that others can help is a life lesson!) $\endgroup$ – corsiKa Aug 30 '16 at 15:03
  • $\begingroup$ What about having a policy like MSE. If a homework question shows effort or the question is challenging then answer it else ditch it. $\endgroup$ – A---B Aug 30 '16 at 19:09
  • 2
    $\begingroup$ @A---B If by MSE you mean Mathematics, that's what our current homework policy is based on. It doesn't work for us. $\endgroup$ – David Z Aug 31 '16 at 6:48
  • 1
    $\begingroup$ @DavidZ Why doesn't it work ? what is the problem ? $\endgroup$ – A---B Aug 31 '16 at 8:35
  • 1
    $\begingroup$ @DavidZ The homework policy here is not like MSE.Here even if the user shows considerable effort several times the question is closed for reasons like-it involves calculation,off-topic etc...I agree with A---B...atleast check-my-work problems should not be closed if user shows effort. $\endgroup$ – user74370 Aug 31 '16 at 15:28
  • $\begingroup$ @SanchayanDutta Our policy is based on their policy, or at least what their policy was at the time - there's a link right in our meta post giving the source on Math Meta. Maybe that policy is not what they actually implement, but it was written. Anyway, you can make a proposal that check-my-work questions with effort be on topic. You can post it below, but I think we might also have further discussion of each of the individual proposed close reasons, so you should bring it up again at that time. $\endgroup$ – David Z Aug 31 '16 at 15:37
  • $\begingroup$ @A---B (3 comments up) Because it's not consistent with the goals of this site. Specifically, one of the goals is for this not to be a place where people who just want help on homework problems are encouraged to come. The Stack Exchange model is not well suited to offering homework help; we want our questions to offer educational value to others who come afterwards who have similar conceptual difficulties, not only people working on the same problems. $\endgroup$ – David Z Aug 31 '16 at 15:40
  • $\begingroup$ @DavidZ I agree with you "partially" on your reply to A---B.However several times incidentally people work on similar "type" of problems.So learning from another person's conceptual mistake they too can learn.A major part of learning physics is problem solving.I do not understand why some people simply refuse to understand this.So I vouch for keeping homework problems involving conceptual doubts open(provided the user has shown their incorrect approach).Morever an elaborate title often helps future users to spot similar type of problems. $\endgroup$ – user74370 Aug 31 '16 at 15:50
  • $\begingroup$ @DavidZ Thanks for the advice btw.I'll make my proposal by writing an answer when I'm free. $\endgroup$ – user74370 Aug 31 '16 at 15:52
  • 1
    $\begingroup$ @DavidZ (and everyone else) I started a discussion based off of Jim's suggestion here. $\endgroup$ – heather Sep 5 '16 at 23:19
7
$\begingroup$

Determining how to redefine this policy is a tricky subject. Not only because of the varying and contradictory opinions about it, but also because we seem to be tackling the issue from the middle. Admittedly, in football, that is a sound strategy, but in politics (and that's just what this is), it's often best to start from the beginning.

Before we get ahead of ourselves, we should step back and officially answer some questions that I'm sure many of you will consider already answered. But until we know everyone agrees on them and knows our stance, we can't properly move forward.

This first question we need to ask ourselves is simply "What are the goals of this site?" What is it we want this site to represent? What are the ideals to which we should hold all of the content on this site? Should it be by physicists and for physicists? Should it be a place where people can learn physics concepts? Should it be a homework help site? Again, many of you may think this an unnecessary step, but when there exists so much disagreement concerning a policy, it makes sense to retreat to a position where we can all agree on something and build from there.

The second question we should ask ourselves is "How much freedom are we willing to grant users in deviating from the goals and ideals of this site?" If the site's goals are to make it a place for professional physicists to ask/answer about research-level physics, how much leeway should we give for users to ask more basic questions? This answer needs to be weighed with how much any amount of deviation will detract from the ability of our site to live up to the goals we agree on. It also serves to provide necessary background and understanding so that everyone can be on the same logical page moving forward.

Now we get into some of the more contentious questions. Most close reasons are easy to determine; the ones we need to decide on are the off-topic sub-reasons. So the next thing we must ask ourselves is "What types/topics/formats of questions would be actively harmful towards meeting the goals/ideals of the site?" This should be applied to questions at all levels. By determining what is actively harmful towards the proposed site goals, we can more easily determine what is off-topic. We should also take this opportunity to realize that the freedoms we established in question 2 may indicate sub-goals of the site and that it may be the case that some questions, which do not directly contribute to the primary goals, may be beneficial or detrimental to the sub-goals.

The last question we should be asking is "What commonalities in these harmful questions can be isolated as reasons to close a question as off-topic?" Here is the important step. This is a place to identify the important patterns in the questions we found to harm the site's objectives. We also should use this step to recognize any biases we have. For example, if we find that everyone is more lenient towards interesting questions, then we need to decide if policy should reflect this or not; allowing exceptions based on general interest or requiring that decisions be made with no regard to the popularity of the question. By approaching this from a standpoint of "Why are these questions harmful to the goals of the site?", we can more easily divine the set of rules that would prevent such questions. But, again, in order to do that properly, we need to have the answers to all the previous questions agreed on and made explicit. I would expect that reasons like the non-mainstream one will remain, but we may find that our homework-like policy never enters into it. We may also find that much of the misuse of some close reasons spawn from a general disinterest of users, and not from any rational source. That would, of course, be a worst case scenario; I'm just giving examples. At any rate, this step also needs to keep in mind how enforceable any close-reason can be. As we once discovered, closing a question purely because it is taken from a homework assignment is not very enforceable. All one needs to do is slightly reword it and claim it is a genuine curiosity. Furthermore, something like closing because a question is too tedious is too subjective to be adequate. Obviously, reasons should be clear, defined, and effective.

If I may jump back to the beginning of my post, we've been approaching this starting from the third question (David Z's thorough research). While I definitely see the appeal in this strategy and advocate it being a good first attempt, once this fails (as we noticed it did), we should go back and truly start from the beginning. It's a longer process that should help to create a more permanent solution.

Now, some of you may point out that I haven't actually done anything towards saying how we could answer these questions, nor have I suggested any way the home-work policy should be changed or researched. You're right. At the moment, I don't see much point in trying to directly address the homework policy, nor can I imagine a situation where we could conduct research and find a clear solution. And if I had included my own answers to the questions I presented, you'd all have voted on whether or not you agree with my answers to those questions as opposed to agreeing with the issue of whether or not the questions need to be addressed at all. Additionally, as demonstrated, people don't seem to have any one thing in mind when they use the homework reason, so the only way I know to fix that is to establish a basis that gets everyone on the same page.

We seem to be building a house on sand and every time the tide comes in, we question why part of this house gets washed away and every time, we try to rebuild that section newer and better. Instead, I merely suggest we tear it all down and lay a proper foundation before attempting to build it back up. The tide may eventually erode the foundation, but at least it'll last a good while longer.

$\endgroup$
  • $\begingroup$ I think these may be better posed as individual meta questions. After all, this whole process goes nowhere if we don't come up with consensus-based answers to the questions. It might be a good idea to just start by reexamining the goals of the site. $\endgroup$ – David Z Sep 2 '16 at 13:31
  • $\begingroup$ @DavidZ I agree. I'm considering making several meta posts over time to address this. But I felt it necessary to respond with this here to judge whether or not the community felt it worthwhile to invest time into this process in the first place. If there's no objection from anyone, I'll soon make a post about determining the goals $\endgroup$ – Jim Sep 2 '16 at 13:46
  • $\begingroup$ I'm not sure if I should upvote because what you've posted here isn't really an answer to my question, but I do support that idea. $\endgroup$ – David Z Sep 2 '16 at 14:04
  • $\begingroup$ A written declaration of support is just as meaningful to me as an upvote. Consider your opinion counted. I suppose, those who feel the same way you do can now also simply upvote your comment to show support for the idea. $\endgroup$ – Jim Sep 2 '16 at 14:09
  • 1
    $\begingroup$ @Jim, I support this idea. I think the meta posts you suggest would be an important step toward building the homework policy, and I'd be glad to see how the results affect the answers already here. Finally, I think it might be a good idea to post them sooner rather than later. DavidZ's goal seems to be not only to get the close reasons, but also to get them in a somewhat expedient manner. =) $\endgroup$ – heather Sep 3 '16 at 12:39
  • $\begingroup$ @Jim, I started a discussion for the first question here. $\endgroup$ – heather Sep 5 '16 at 23:19
9
$\begingroup$

The current situation and its problems (yes, again)

In light of all the discussion and analysis we've done since then, when we choose to vote to close questions as homework-like, what reasons do we actually have in mind?

Taking your data, and indeed the continual disagreement in most past meta posts about this topic, at face value it appears that there is no single reason people close something as homework-like, showing that the current close reason is also probably more used as a proxy for a host of different reasons than what's actually written in the policy, or whatever would be actually written in the policy once we'd chosen one or more "main" reason to close something as homework-like.

This is an uncomfortable situations, and it also explains why this policy continues to be discussed (other than, say, the non-mainstream policy, about which there seems to be general agreement, and only dissatisfaction about its application in isolated specific cases). However, it also means that "replacing" the homework policy by the things it's used as proxy for is infeasible, because there's no agreement what it is used as proxy for.

I suspect this is because the policy was initially conceived to ban actual homework to stave off using the site as a cheating resource. This has two issues: How to tell whether something is "actual" homework, and what about good questions that someone could conceivably encounter otherwise, but that are assigned as homework in the case of the asker? So the homework policy mutated into the "homework-like" policy, looking at the content rather than the origin of the question. However, what exactly is "homework-like content" was, at least from some points of view, insufficiently (and somewhat contradictorily) codified in the policy post as such, which is argued e.g. in this meta answer.

That therefore the policy as written is actually not what motivates many closures is what triggered this whole debate, cf. the opening statement of Generalizing the homework policy:

However, it's not always clear whether a question fits this description just from its content. And if you track the questions that we actually close these days using the homework-like close reason, quite a few of them are likely not of an educational nature. Instead, we've taken to using the homework-like close reason on questions that simply ask us to calculate something without the original poster making an attempt at it.

A radical solution

Get rid of the homework close reason.

Yes, I mean that. I don't mean to imply that we should leave any of the question open that we currently close as homework-like, but this whole debate shows that "homework-like" is really just a proxy for a bunch of more specific close reasons, which would be more helpful to the people asking the question and more helpful to avoid arguments between reviewers over whether something is "homework-like", when the arguing parties really mean two different things by that word.

Unfortunately, we don't have enough slots for pre-generated reasons left to cover all of the partial reasons for closing as "homework-like". Fortunately, the "other" close reason allows for an infinite variety of close reasons, so we only have to decide which reason deserves to be in the pre-generated spot and which ones the reviewers have to type out.

We already have discussions on insufficient effort and "check my work" as close reasons. We have broad support for closing "tedious" calculations without context. There are upvoted arguments that being a conceptual question and showing effort, as in the current "homework-like" close banner, should apply to all questions.

Each of these discussions about policy, each of which covers a certain subset of why people want to close homework-like questions, is worth having. These are specific topics on which people can take a stand for or against, and we can sort out where the general community trend lies.

This discussion about the "homework policy", however, and about a catch-all proxy reason for all the actual reasons it encodes is not worth having anymore. It's too confusing, people are talking past each other, and we have both historical and analytical data showing a consensus is nowhere in sight. So let's just axe this reason and start having constructive discussions about more specific close reasons.

$\endgroup$
  • $\begingroup$ Your radical solution is exactly the reason we're doing this whole thing. That's been the goal of the process all along. (I dare say that makes it not so radical?) What I'm hoping to get from this question is some sort of consensus on which other close reasons we should consider promoting to standard (e.g. insufficient effort, check-my-work, tedious). $\endgroup$ – David Z Aug 27 '16 at 14:32
  • 3
    $\begingroup$ @DavidZ: See, I thought we were just trying to reformulate the policy, i.e. identify how to better call the questions we close as "homework-like". Trying to correlate specific reasons with homework closure also seemed to me to be to that end. I'm saying: Don't worry about what "homework-like" may or may not stand for, but just discuss the specific close reasons without all this baggage. $\endgroup$ – ACuriousMind Aug 27 '16 at 14:39
  • 1
    $\begingroup$ I agree with this - +1. Not only would it help the askers of the closed questions, it would also help the close voters. $\endgroup$ – heather Aug 27 '16 at 14:45
  • $\begingroup$ Well... I would have said that we're trying to identify how to label the questions we close as homework-like, in the sense that we want to call them low-effort, check-my-work, tedious, non-conceptual, or whatever. In doing so, we would completely eliminate homework-like-ness (and its associated policy) from our vocabulary. Maybe some different wording is necessary. (Though I thought "replace the homework policy" was pretty clear that we are getting rid of it.) Perhaps I should edit my question to clarify that what you're proposing is what we're trying to do? If you don't mind.... $\endgroup$ – David Z Aug 27 '16 at 14:46
  • 1
    $\begingroup$ BTW the reason we are still thinking about homework is that we want to make sure whatever new close reasons we come up with still cover the set of questions that we currently close as homework-like. $\endgroup$ – David Z Aug 27 '16 at 14:47
  • 5
    $\begingroup$ @DavidZ: Well...but why do we want to make that sure? If we find that some questions closed as homework-like actually have no valid reason for closure, wouldn't that also be fine? That's why I say discuss actual close reasons, and not the murky and burdened topic that is "homework". $\endgroup$ – ACuriousMind Aug 27 '16 at 14:56
  • $\begingroup$ @DavidZ, I might also add that it is not clear we were trying to replace the homework close reason, I thought our purpose was to better define the homework close reason. As such, I agree with ACuriousMind's proposal (and think that your question should be clarified a tad if that is your intent). $\endgroup$ – heather Aug 27 '16 at 15:03
  • $\begingroup$ @ACuriousMind We want to make it sure because people want those questions closed. Evidence: people are closing them. If people are closing questions that have no valid reason for closure, we have bigger problems. $\endgroup$ – David Z Aug 27 '16 at 16:20
  • $\begingroup$ @heather (and ACM) Can you help me understand how "replacing the homework policy" is not clear? What more is needed to show that the current homework policy is going away? $\endgroup$ – David Z Aug 27 '16 at 16:22
  • 2
    $\begingroup$ @DavidZ, out of curiosity (asking about your earlier comment), would it be possible that people are closing these questions because the close reasons are unclear, and as such some of them are incorrectly closed? $\endgroup$ – heather Aug 27 '16 at 16:23
  • 1
    $\begingroup$ @heather Not in this case, because the definition of "correctly closed" is ultimately set by what people want to close (i.e. what they believe is not appropriate for the site). Every closure policy we have was developed to reflect the opinions of the site members at the time it was created. If a few questions are closed in a way that contradicts policy, sure, those are incorrect closures, but right now many questions are being closed in a way that's inconsistent with the homework policy, and at that point we have to consider the policy itself to be incorrect. $\endgroup$ – David Z Aug 27 '16 at 16:30
  • 3
    $\begingroup$ @DavidZ: Maybe we're misunderstanding each other. My main point is that we stop talking about what is meant by "homework", why we currently close "homework" question and whether or not "homework" questions should be closed, and start just talking about each of the more specific reasons you've tried to examine in the OP without the context of the homework debate. Once we've discussed all the specific reasons and decided whether or not we will close questions for those reasons, we just remove the homework close reason and its policy. $\endgroup$ – ACuriousMind Aug 27 '16 at 16:32
  • $\begingroup$ @DavidZ "Replacing" the homework policy technically means the right thing, but your entire question reads to me as if we're trying to figure out what "homework" actually means and then replace "homework" by that meaning. That's what I don't want to do anymore - just forget about "homework". $\endgroup$ – ACuriousMind Aug 27 '16 at 16:34
  • $\begingroup$ @ACuriousMind (previous 2 comments) It seems to me like we're talking about the exact same thing except that you want to do it without any mention of the current homework policy, and I don't see the point of that avoidance. We can certainly adjust the terminology to talk less about homework, but the current homework policy has a rich history which I think we should be considering in our efforts to come up with a new set of close reasons. $\endgroup$ – David Z Aug 27 '16 at 16:36
  • 2
    $\begingroup$ @DavidZ "We want to make it sure because people want those questions closed. Evidence: people are closing them." <-- I'm not sure this follows. At least in the old days, we were quite strongly asked to close questions according to the policy as decided on meta. So it could well be that people are closing them because they perceive there to be a rule that says they should be closed, and for no other reason. Because of this, I think a data based approach is of limited value anyway, and I'm very glad that you're moving this to a discussion-based approach instead. $\endgroup$ – Nathaniel Aug 28 '16 at 12:22
3
$\begingroup$

I agree completely with ACuriousMind's answer, so I won't go into that, but simply expand on what main reasons I see things being closed as (or should be closed as). These are of course just my opinions, and I would welcome explanation of disagreement in the comments.

Close Reasons

Not Enough Effort

I've seen much discussion about this one; however, I think it is important in a certain formulation. First, I think there are two situations where this is applicable:

  1. Questions that would normally be considered "problems" where no work is shown, e.g. https://physics.stackexchange.com/questions/276675/motion-in-a-projectile-plane, https://physics.stackexchange.com/questions/276640/pressure-cooker-at-a-hill-station (this can be used on non-math work questions; it is more about showing your thoughts).
  2. Questions whose answers are so blatantly obvious (I don't even mean Wikipedia-obvious, I mean even more blatantly obvious) that they are not worth answering, e.g., https://physics.stackexchange.com/questions/276508/what-is-internal-energy, or (hasn't been asked) "Why do rocks fall".

I do not think questions should be closed for this reason if they do not fall into one of those two categories. I mean, for example, that a question such as How to measure the inner diameter of thin tube (0.5mm to 2mm) should not be closed for not enough research effort. While it is a poor question, perhaps, and deserves a downvote, it should not be closed as "not enough effort".

The reason I think it is important that this reason is limited to these two categories is because of subjectivity. Most people can agree that questions falling into the two categories above should be closed, but there could be a lot of debate about another question that might have been otherwise closed with this reason.

Check My Work

This should definitely be a close reason. I have seen many questions that could (and should) be closed with this reason (for example, https://physics.stackexchange.com/questions/276576/line-of-charge-passing-through-centre-of-a-charged-semicircular-rod). This seems to be one of the main categories homework-type questions fall into.

It is also worth noting that the check my work close reason should only be used for trivial calculation errors; questions about a conceptual mistake (for example, whether a certain formula is appropriate in a given situation) are on topic.

Purely Calculation

First, I'd like to point out that just because a question asks about math or includes math does not mean it is off topic! I say this because I have seen multiple questions that I would consider good, on-topic questions closed under the idea that they are "not conceptual". I'd solidly disagree with this.

This close reason is more for questions that are all math, or all "do my math for me". Tedious do-my-work questions would probably fall under this close reason (aka, calculate this for me questions). An example of a question that is on-topic (even with this close reason) in my mind would be Mutual $E$ force due to charged coaxial rings (though I am specifically referencing question #3 as listed, #2 is more a mathematics.SE question and #1 is a check-my-work question).

Ones that should be closed under this reason would include, for example, the question https://physics.stackexchange.com/questions/275704/angular-trajectory-of-body-moving-on-rim-of-a-carousel.

Note that if there is a more specific close reason, then one should use that. For instance, you could say there's some overlap between the check my work close reason and the not conceptual close reason. But obviously, if both are valid, the one that is more specific to the situation is the one that should be used.

Formulation

The specific formulation of whatever close vote reasons we pick will probably need to be hashed out in their own meta post. But, I'll just give a few starter definitions here.

For "not enough effort":

This question does not show enough effort. It either does not show the thought process that lead to the question (mathematical or otherwise) or the answer can be very easily found on the internet. As such it does not carry any value for future readers. See our [meta site][] for further guidance.

For "check my work":

This question appears to be about checking your work, and a trivial calculation mistake, not a physics concept. See our [meta][] for more guidance. Please also make sure that if it is about how to calculate something to try math.SE.

For "Purely Calculation":

This question appears to be outsourcing a calculation to the community without any broader context. Please ask about a specific concept, show work and explain where you got stuck, or provide more context. For more information, see [this meta post][].

(Please note that where I put [some text here][] I meant for a link to the definitive meta post to go there.)

Hope this helps, and any feedback would be appreciated!

Edit: I originally worded the title to the third possible close reason rather poorly, and changed it. The third close reason is meant for purely calculation questions, as the title says, and I'll be adding more examples of questions that would be closed under this reason soon.

$\endgroup$
  • 1
    $\begingroup$ This is the kind of content I'm looking for in an answer. By the way, I didn't mention the tag to limit the scope of the discussion. Once we finish dealing with the close reasons, then we'll have a separate discussion about the tag. $\endgroup$ – David Z Aug 28 '16 at 3:57
  • $\begingroup$ @DavidZ, okay, so should I delete the "burninate" section? $\endgroup$ – heather Aug 28 '16 at 12:15
  • $\begingroup$ Well, it's up to you, but I think that would be a good idea. Save it for when we do have a discussion about the role of the tag. $\endgroup$ – David Z Aug 28 '16 at 12:18
  • $\begingroup$ @DavidZ, I have done so. What do you think about the close reasons? Is there anything unclear? $\endgroup$ – heather Aug 28 '16 at 12:20
  • $\begingroup$ I think there's a long discussion to be had about that. That's probably the next step of the process. $\endgroup$ – David Z Aug 28 '16 at 14:41
  • $\begingroup$ Check my work should be on-topic if the user has clearly shown his/her approach. $\endgroup$ – user74370 Aug 31 '16 at 15:30
  • $\begingroup$ @SanchayanDutta, no, I don't believe it should. This is a site for questions about physics. When questions that simply ask "hey, check my work" (even if they've shown all their math) that does not help the community in any way, shape, or form. The stack exchange model is such that the questions should also be useful for others later, and check my work questions wouldn't be. If you disagree with that, do you really think that people who come on this site will be helped by pointing out the error in some arbitrary problem? And, if it is a more complex question about how to do calculus (cont.) $\endgroup$ – heather Aug 31 '16 at 20:39
  • $\begingroup$ (cont.) or some such problem, that's a math.SE question. So check my work questions should not be on-topic. $\endgroup$ – heather Aug 31 '16 at 20:40
  • $\begingroup$ @heather Oh really?Check my work isn't useful for future users?Several times,rather many times my concepts became more strong after I saw the mistakes in problem solving approaches/conceptual faults usually made by people in their work.Obviously I'm not saying elementary questions like "What is the gravitational force between sun and earth?" should be allowed but atleast undergraduate/graduate level physics must be allowed.Problem solving ability is one of the MOST important part of learning physics.Anyone who asks a question along with clear signs of research and effort must be allowed here. $\endgroup$ – user74370 Aug 31 '16 at 21:08
  • $\begingroup$ @SanchayanDutta, please note that I am not saying problem solving questions should not be allowed, I am saying there should be bounds on those questions (as there are on all questions). A question asking about common mistakes would, I think, be on-topic, but a question asking one to go over work for a small error is off-topic. I really think that if you aren't careful, you can let in questions that you actually don't want. $\endgroup$ – heather Aug 31 '16 at 21:10
  • 1
    $\begingroup$ @heather BTW what's the problem in going through another's work to spot their mistake? In case it's a trivial calculation error we may just close it but in case the mistake is conceptual we must go forward and answer it!Future users can learn a LOT from what mistakes previous users had made.Learning from one's and other's mistakes is also a part of learning physics! $\endgroup$ – user74370 Aug 31 '16 at 21:14
  • $\begingroup$ @SanchayanDutta, the purpose of check my work is to keep questions from being about trivial errors. If it would make it clearer, I could add a sentence saying that it is for trivial errors, but I thought it was pretty clear. Even still, I'm not completely sure "conceptual" (what does that even mean?) check my work answers are on topic. $\endgroup$ – heather Aug 31 '16 at 21:17
  • $\begingroup$ @heather You do not understand what conceptual mistakes mean?Strange.For example someone asks "Is it correct to apply the method of effective gravity in this problem like I have applied below in my solution? " ---that is clearly a conceptual problem and we must help the user by saying "Yes/No" "You cannot/can apply that method here because your .... assumption was wrong because ...".In that case we should go through the work of the user and say if he/she has made any conceptual error in his/her assumptions. $\endgroup$ – user74370 Aug 31 '16 at 21:21
  • $\begingroup$ @SanchayanDutta, I did not understand what you meant by conceptual mistakes in that context! Anyway, I do not think that should fall under the check my work close reason; that is not what it is meant for. May I also suggest using spacing? Your comments can be a little hard to read. $\endgroup$ – heather Aug 31 '16 at 21:23
  • $\begingroup$ Several such questions have been closed on Physics SE which do not fall under trivial calculation mistake category.We must try to avoid it. @heather $\endgroup$ – user74370 Aug 31 '16 at 21:24
1
$\begingroup$

Here is my take on the matter. First, let's think about what Stack Exchange Q&A sites are good for and what they are not good for:

  • A Q&A site needs Questions, which can be answered in a concise fashion. There may be several good answers, but if a question needs books to be answered or many answers covering different topics (and thus many pages of answers), it's not a good fit.
  • Questions that are an especially good fit are questions where there can be different angles from which to approach it so that many good answers can emerge. This also gives meaning to question ranking.
  • Questions that spark discussions are a bad fit.
  • The specific idea of Stack Exchange Q&A sites is that the answers to questions and the questions themselves should be interesting to a broader audience.
  • If the questions are not interesting for very many people, they need to be of research quality.

In fact, as I see it, we have some tradeoff: If Q is the quality of the question in terms of how many people could answer it ("research quality") and A is the audience that can get some new understanding from the question, then the product QA should be large for good questions. This leaves room for conceptual questions about basic theories as well as high level and extremely specific questions about research.


If we agree on the above, here is my take on what should be closed and what shouldn't:

Check-my-work: Questions like "Where is my mistake?" or "Did I do it right?" come in two flavours: The first questions make one of a few very common mistakes. Those mistakes are common, because they are based on a fundamental misunderstanding of the theory. They should be closed but could be reworded to ask a conceptual question about the theory, which is interesting for many people. The other type of questions are just algebraic/mathematical mistakes. There are a myriad ways to make mistakes and thus the answer to the question is mostly only interesting to the asker and it usually also is of bad "research quality".

--> Close as check-my-work. If you think there is actually an interesting conceptual misunderstanding in the question, rewrite it and/or leave a comment. Then the question can be reopened.

Really low research effort: Since most of the audience arrives by search engines, those people will actually do a Google search. If this will directly lead them to a Wikipedia article, what do they gain by coming here? So aside from the fact that low effort questions might disparage researchers from staying on the site, they should be closed because they don't actually have an audience: Why should you come here to get a link to the Wikipedia page you could already see in your Google search results?

--> Close as "low effort", if the question could be commented on by "What about this [Wikipedia/other well-known source that pops up if you type the question into a search engine] is unclear".


Those are in my opinion the "easy" ones. Now come the "hard" ones:

(Low effort) problems: These are questions that essentially ask for the solution to a homework problem while providing or not providing own ideas. The problem with these questions is two-fold: a) if the asking party actually wants to understand the problem, you'd need to discuss with them --> that's not a good fit for Q&A, b) the audience for these questions is often only the asker him/herself (see "check-my-work" questions).

Defining a close reason such as "no sufficient audience" won't work and will only spark more discussion, as will "homework" ('oh, but that's not homework, I thought about this exercise myself'). One of the biggest problems is also that these questions are best answered by working with the person and asking questions and giving rough sketches and approaches - so they are just a bad fit anyway. This is particularly true for many questions where the author posts a lengthy and detailed list of her own failed effort. While those questions do signal that the author of the question doesn't just try to dump his problem on us, they are just generally a bad fit for Q&A sites.

--> One idea could be the following: Introduce a close reason "not conceptual" and apply it to any questions, asking for one solution to a problem where most variables come with numbers attached. This would allow questions asking about how to generally approach inclined plane problems where a set of parameters is given, but it would disallow questions asking about inclined plane problems with a specific set of parameters (for instance and angle of 45°). Having a rule like "A questions where a lot of variables are fixed is closed" would eliminate some of the ambiguity of the word "conceptual". It would also allow questions asking for different ways to approach some problem, which in my opinion is a valid conceptual question.

Pure calculations: Calculations are also difficult, because they can come at any level - and any difficulty. Most calculation questions are only interesting for the person asking them, so they should be off-topic.

But often when somebody reads a paper, he/she may stumble over some step in a calculation or some line of an argument that they don't understand. What about such questions? Now, often, we can refer those questions to math.stackexchange and be done with it, but often the questions may involve physical approximations and are better left here. Not understanding some equation in a paper is something extremely common - sometimes the equation is also just wrong.

Having a platform to ask these questions would be a valuable aid to research, because it could save a lot of time (the idea of course being that I invest some time to explain details in a paper that I understood easily, saving the asker hours or days of searching, while some other times, I am the one asking). In principle, they are also a good fit for Q&A and they should have a decent QA product. If we had arXiv-feedback, this could actually help other readers of the paper and enhance overall research.

However, we don't want to explain simple equivalence transformations in middle school textbooks, where the person was just too lazy to think about it for five minutes. And we don't want to do calculations in research papers that were left out, because they are tedious but foundational to the field and can be looked up in text books asking person is just too lazy to do them him/herself.

How to distinguish these two? If we want to have more people here who are doing research, it might be a good idea to allow paper questions and disallow all others (this would at least get rid of ambiguities). This is somewhat in line with the idea (Generalizing the homework policy)

Questions which attempt to outsource tedious calculations to the community, without any broader context, are off-topic.

but tries to be a bit more inclusive for research-level questions and a bit more exclusive to avoid ambiguities.

--> One suggestion (needs work): Close non-paper related questions, which want to do outsource calculations with a specific "tedious calculation" close-reason.

$\endgroup$
  • $\begingroup$ I mostly agree, but one objection is that IMO questions which have many different angles to allow different answers to emerge are not an especially good fit. The best questions in this respect are those that are fully handled by a single answer. The idea is that when someone else with the same question finds it on our site, we want to maximize the probability that the top answer helps them. For that purpose it doesn't matter whether a question has one answer or several, as long as one of them is well written, understandable, and correct. $\endgroup$ – David Z Sep 2 '16 at 11:40
  • $\begingroup$ I'm also not convinced by the argument that disallowing questions about tedious calculations from papers. Oftentimes such calculations are left out because the authors think they're clear, but they're actually not clear to readers. One of the most valuable uses of this site, I think, is to fill in the steps in these cases, or at least provide a roadmap to how the calculation works out, even if we don't necessarily give all the details. The idea of making "paper-related" a criterion for on-topicness is unconventional, but might actually be a good one. I'm intrigued. $\endgroup$ – David Z Sep 2 '16 at 11:44
  • $\begingroup$ @DavidZ: I agree with your take on papers, and that is precisely why I think those questions should be on topic, even if they are sometimes just "filling in the gaps" (mostly calculations, but often with a physical take). My argument concerns those rare questions where the filling in is really just "tedious and lengthy calculations" which can be looked up in introductory books (for instance, if a paper started with the linearised gravity and the question was "can you show me how to derive those equations?"). This should only concern foundations for the paper though, not the content itself. $\endgroup$ – Martin Sep 2 '16 at 14:03
  • $\begingroup$ I edited that sentence to better express what I meant there. $\endgroup$ – Martin Sep 2 '16 at 14:05
  • $\begingroup$ (2 comments up) Well, even that could be a reasonable question, I think. A good answer would give a quick justification of how Einstein's equations could be linearized, and refer the reader to a source for further details. $\endgroup$ – David Z Sep 2 '16 at 14:06
  • $\begingroup$ So, I understand and like your reasoning (maybe because our close reasons are very similar, though I combine two) but I would like some clarification on the last one. I think that might be a good direction to go, but what I don't understand is how to keep it from being arbitrary. I mean, sure, we might not want middle school textbook calculations, but, who knows - maybe we do! There could be a very interesting question that's middle school level, perhaps, but is an important concept, if that makes sense. (cont.) $\endgroup$ – heather Sep 2 '16 at 20:42
  • $\begingroup$ (cont.) So, I guess, I want to know what the fine line is between "every question that links to a paper/textbook/lecture is on topic" and "no question that is calculation based in on-topic". I think this kind of needs to be clarified in my own answer as well, so I'm interested in this method. $\endgroup$ – heather Sep 2 '16 at 20:43
  • $\begingroup$ @heather: I agree, we should think more about it (I guess this'd be another thread to flesh out the details for instance) - but in the end, we will have one of two problems anyway: Either the policy is ambiguous and hard to understand (status quo), or it is more or less unambiguous but a few valid questions will get closed... $\endgroup$ – Martin Sep 3 '16 at 8:45
  • $\begingroup$ > How to distinguish these two? ~~ There's no need to distinguish them at all - at least not in any general sense. Each user can decide for him/her self if they want to answer a given question or not. If nobody answers the question because everybody decides it's not worth answering, then no harm, no foul. If somebody decides to be gracious and answer it, then the OP gets the answer the sought and nothing is lost. $\endgroup$ – mindcrime Sep 7 '16 at 17:41
-2
$\begingroup$

The Problem as I see it

The impression I get is that the site is being flooded by a rising tide of poor quality questions which show little or no effort and often lack even a basic understanding of physics. I am not being elitist, I simply deplore lazy questions from those who are obviously not even students of anything, especially not physics.

Since reaching 3k rep, I spend more than half of my time on the site reviewing questions. Currently I vote to close around 7/10 in the "new unanswered questions" queue and a higher proportion in the "Close Votes" queue. This counts questions which I do not feel competent to judge, which I do not touch. Recently I am regularly running out of close votes within 2 hours of getting them, and have to make do with down-voting and commenting instead.

My expectation is that if this trend continues then in the near future the pool of active reviewers will not have enough votes or time to keep up, with the result that the average standard of questions will fall.

Closing questions is frequently pointless. By the time the crucial 5th vote has been cast, answers have often been posted so the question does not get deleted. A few new users respond by posting the same question again. Closure prevents good answers from being posted to bad questions.

Voting-to-close questions provokes resentment and animosity, not only from those posting questions but also other users. Explaining to the OP and others why I have VTC also takes up time - although this is not necessary.

Current policy is inconsistent. We routinely demand that "homework-like" questions must "show effort to work through the problem," yet we tolerate conceptual questions which show little or no effort and are ill-motivated ("I read somewhere... Is this true?" "While I was in the bath I had a thought..." "I was just wondering..."). The site is advertised as being "for students, researchers and academics of physics and astronomy" but we allow anyone, even those with no physics background whatsoever, to post any question. We cannot expect high standards if we adopt inconsistent policies.

Whatever policy is formulated, the most difficult part is implementing it - getting users (especially new ones) to read it, agree to it and follow it. Unless there is a requirement for users to do so, many will not bother. They come here to ask and answer questions, not to study and enforce site policies. It was only after I reached reviewer status that I accepted the closure criteria.

An Even More Radical Solution

I repeat my suggestions given in my answer to Generalizing the homework policy.

I see only two consistent options : either (1) abandon any attempt to control the posting or answering of questions, or (2) place questions 'on hold' as soon as they are posted, and do not open them until they have been approved by reviewers who understand and agree with site policy, rather than those who have their own interpretations of it.

I appreciate that both options are highly unpopular, but at least they are workable, efficient and self-consistent.

1. Abandoning Control

The only reason for closing questions should be that they are off topic, which should mean that they cannot be answered using mainstream physics. It should be open to any user to decide for him/herself whether or not a question meets his personal standards regarding quality or usefulness or research effort or clarity or broadness etc, and to answer or up/down-vote or comment accordingly. Lack of effort and usefulness are already addressed in the up/down-voting feature, so these should not be reasons for closing a question.

This is my preferred option.

Rationale :

Most users simply want to ask or answer questions. Those who ask usually want a quick response. Few use the site for research. Pre-occupation with question quality is inconsistent with allowing unrestricted posting of questions. Minimising reasons for closure will reduce the growing burden on reviewers, and avoid many bad experiences on both sides.

It is patronising of us to decide that some questions should not be answered, and to prevent users who have no objections to the question from doing so.

Abandoning control will allow more poor-quality questions onto the site, but there are plenty of those already which have been closed but not deleted. Voting already identifies good (and/or interesting) questions. If necessary, a periodic purge could remove questions with many down-votes, perhaps via a review queue.

2. Voting To Open (VTO) New Questions

This turns the VTC decision on its head by proposing that all new questions (or perhaps only those from new users with less than say 300 rep, or those with a poor history of closed questions) should be put on hold until approved by reviewers. Criteria for what is on topic would still have to be re-formulated (see below) : the purpose of this proposal is to ensure that policy is implemented effectively.

Rationale :

It is far more effective to get rid of poor questions, and deter poor question-setters, before they have gotten an answer rather than after. Those who need quick answers to trivial questions ("Is this right?" "Where have I gone wrong?" "What am I missing?") will turn away in frustration if it takes 24-48 hours before the question is approved. We would also need to be stricter about "answering in comments" - perhaps by increasing the rep required to comment and flagging such comments for immediate deletion.

It is currently impossible to prevent users from answering questions which are under review, making subsequent closure pointless. For the same reason VTO retains the option of migrating the question instead of closing it.

VTO is consistent with the avowed purpose of the site. Students, researchers and academics will have spent considerable time and effort already working on the problem, and are unlikely to expect immediate answers. Good quality answers to good quality questions often require further research or mathematical analysis, which cannot often be delivered within 24 hours.

It is also consistent with the policy of avoiding duplicates, because there is more time to identify duplicates before answers are posted to the new question.


Reformulation of Closure Policy

I agree with ACM and heather that the "homework-and-exercises" tag should be removed and the related policy scrapped. All questions should be subject to the same criteria, whatever those may be.

As stated above, my view is that the only reason for closing a question should be that it is off topic, meaning that it cannot be addressed by applying mainstream physics concepts and principles. The question itself need not be mainstream physics, since good science tolerates the unorthodox. I say addressable because for some questions it may not be possible to give an authoritative answer.

There is a lot of subjectivity in the other criteria, which I think are not worthwhile :

Duplicates : It is very difficult to avoid them, and checking is tedious. Even where they are identified before new answers are posted, existing answers may be unsatisfactory, or there may be a subtle difference. Reviving an old question is not easy for a new user (see Re-asking a Duplicate Question).

Unclear what you're asking : Easily dealt with by asking questions to identify exactly what the problem is. This is what comments are for. Very subjective anyway. It is rather contradictory for a question to be closed for this reason when an answer has been posted - and possibly accepted.

Too broad : Easily narrowed down through comments.

Primarily opinion-based : Many such questions (eg resource recommendations, career advice) would be off topic under the above definition (not addressable by applying mainstream physics).

Homework-like : No justification for discriminating based on style of question rather than its content. Questions asking how to apply physics are as much a part of physics as those asking for explanation of a concept or phenomenon - especially for students.

Mainstream physics : Questions about unorthodox physics (eg cold fusion, time travel, perpetual motion machines, warp drive, etc) are addressable using mainstream physics, so they are on topic.

Engineering : "The application of scientific knowledge to construct a solution to solve a specific problem" is what we do all the time as students and experimenters.

Insufficient Research Effort : Too subjective, difficult to judge, and already addressed by down-voting. The reason quickly becomes pointless when somebody provides an answer or even just points out a duplicate or makes a suggestion. If the OP's laziness disgusts you, ignore the question!

$\endgroup$
  • 1
    $\begingroup$ This is a reasonable thing to propose, but I don't agree. The whole purpose of this site is to give high quality answers to good questions (where the definition of "good questions" can be manipulated somewhat), and that requires retaining experts to answer questions. If we open the site to all questions that can be addressed using mainstream physics, that opens us up to a lot of bad questions that will frustrate experts and cause them to leave. And we just don't have the resources to pre-screen all questions before opening them. $\endgroup$ – David Z Aug 28 '16 at 5:37
  • $\begingroup$ I don't think this makes sense. You are giving two extremes. While it may be more difficult to keep the happy medium, and it may be more subjective, I think it is important to try. Yes, close vote reasons can be subjective, but we can try to formulate policy so they are not used subjectively. It is not a perfect system, but I think your ideal systems will actually result in madness. Oh, and yes, there is a deteriorating quality in questions. I think two things would help: 1. the little page before posting a question (which I suggested way back when) and 2. better formulated policy. $\endgroup$ – heather Aug 28 '16 at 12:23
  • $\begingroup$ Oh, and I guess under your definition of who is allowed on this site, I'm not (depending on how far you carry the student definition) allowed on. =) $\endgroup$ – heather Aug 28 '16 at 12:24
  • $\begingroup$ And one final thing: I've seen you vtc for insufficient research effort many times. $\endgroup$ – heather Aug 28 '16 at 12:50
  • $\begingroup$ @heather : I have not defined who is allowed on the site. The site has already done that at the start of the tour. However, that is inconsistent with the promise that "anyone can ask... anyone can answer". I am pointing out that we are implementing inconsistent policies : Insisting that everyone is welcome, but closing down questions when they come. Saying it is a site for physicists, but welcoming everyone. We need to decide exactly what our goal is, then match our policies and procedures to that goal. $\endgroup$ – sammy gerbil Aug 28 '16 at 12:57
  • $\begingroup$ @sammygerbil, I think the "anyone can ask...anyone can answer" is more of a fluffy statement...it is standard across all stack exchange sites. I'd say that's a poor example of a contradiction. Also, in the site definition, we say its a site for physicists, etc. You could close a question as a "layman's question" but that would be so subjective as to ruin the system (in my mind, anyway). $\endgroup$ – heather Aug 28 '16 at 12:59
  • $\begingroup$ @heather : I do not intend to exclude anybody, only poor questions. You are misinterpreting me. Even under the VTO option, the acceptability of questions depends on their quality, not the user's qualifications or rep ... re my VTC activity, I vote to implement site policy as I understand it, not to express my own personal preferences, even though the 2 goals often coincide. However, from today I have decided to abstain from VTC and leave the problem of enforcing policy to others. I shall restrict myself to down-voting and my usual caustic comments, on the basis of my personal judgement. $\endgroup$ – sammy gerbil Aug 28 '16 at 13:20
  • $\begingroup$ @sammygerbil, I understand; I apologize for the misinterpretation...however I think it is unfortunate that you are abstaining from close voting. $\endgroup$ – heather Aug 28 '16 at 13:21
  • $\begingroup$ @heather : If lay questions are ok, then this is not a site for "students, researchers and academics of P&A" - it is a site for everyone. $\endgroup$ – sammy gerbil Aug 28 '16 at 13:26
  • $\begingroup$ @DavidZ : 1. The purpose of VTO is to ensure only high quality questions get answered. All questions are currently screened by someone, so the resources are there. Pre-screening all questions is manageable eg (a) if the threshold rep or no. of votes for VTO is lowered, or (b) if only up-voted questions are moved to the VTO queue. 2. Abandoning Control : Bad questions are already flooding onto the site, posted without restriction. Is this frustrating experts? A simple strategy is for them to concentrate on highly-upvoted questions. Where there is a will there is a way (and converse). $\endgroup$ – sammy gerbil Sep 8 '16 at 5:06
  • $\begingroup$ How do you know all questions get screened? $\endgroup$ – David Z Sep 8 '16 at 7:13
  • $\begingroup$ @DavidZ : I mean that they all get "viewed", commented upon and voted upon. All users are involved in this. Not all are applying the same criteria as used in VTC, but all are judging the question. $\endgroup$ – sammy gerbil Sep 8 '16 at 10:55
-3
$\begingroup$

There are very few good reason to close questions at all. Even ones that are borderline off-topic. What should be closed is basically just spam, outright crankery, flames, etc. The rest can be dealt with through the normal upvote/downvote mechanism.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .