# Getting a better understanding of how and why off-topic homework-like questions get answered before they can be closed

Background

This is a follow-up to What happened with the experiment to lower the vote-to-close threshold?. (See that thread, and the initial proposal here, for background.) The experiment is currently running on a set of (other) Stack Exchange sites that need it more, and need it for other reasons. The SE Community team seems somewhat interested in running it for our use-case, but they don't seem convinced that the case is strong enough that it would do much of a difference.

In the meantime, the Community Manager in charge of the experiment, Catija, suggested (in the experiment announcement at MSE) that

there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer

and this is something we should consider carefully.

I would argue (contrary to this recent thread) that we do have a problem with off-topic homework problems getting answers before they can be closed by the close-vote review queue, which is currently too slow to deal with this. One way to handle this is to make the queue faster (by requiring fewer votes per closure), but that may not be the only way. As Catija suggests,

Maybe you need better guidance to prevent the questions from being asked such as a customized Ask modal or sidebar. Maybe better education for the people answering about when not to answer...

The modal in question currently looks like this, but it can be customized to look like e.g. this (at Code Review). This is something we should definitely look into, but I don't think this problem lies on the side of the askers $$-$$ I would argue that our problem (off-topic questions getting answers before they can be closed) is largely a problem of user education, i.e., we are not as effective as we should be at communicating site policies to new and relatively-new users.

As a way to address that education problem, I suggested to Catija that we could install

something like a modal, or ideally a warning box that appears above the answer text editor box (even better, something that pops in once the user clicks on the editor box), which reminds the user what kinds of h/w questions are on-topic and which ones shouldn't be answered, and which appears, say, on questions with the h/w tag or which have one or more pending close votes with that close reason

and she seems to be quite open to the idea. But she has this important set of questions as a framing for such a device:

But it's important to understand the answerers' motivations. Do they disagree with the questions being close-worthy? Are they unaware that the question should be closed? Do they have thoughts on how they can improve the question - it's possible that the best case scenario for the asker and the answerers is that the questions get improved first and answered - so it's not so much that they can't answer the question but that they should work with the asker to make it a good question before offering a (possibly problematic) answer.

My question here

Catija's last comment raises an important point. If we have a problem, we should understand it as well as possible before fumbling for solutions. So:

• How many off-topic homework-like questions get answered before they get closed?
• Can we reliably measure at what stage in the closing process this happens?
• Who is answering these questions? How much experience do they have with the site, in terms of both time and reputation?
• Are they unaware that there is a site policy that strongly discourages such answers? Or are they aware of it but they disagree with it? Or do they disagree that it applies to that particular question?

I think that we could probably achieve quite a lot using a pop-up modal, or a pop-in banner, when at-risk users (i.e. the population in question here) attempt to answer at-risk questions (i.e. questions with the homework tag or with a homework-like close-vote pending), but if we want that communication to be effective then we do need to understand who we're talking to.

• Caveat: Attempting to measure any of these metrics via SEDE will have to use special care to include deleted posts since many HW questions and answer get deleted sooner or later
– ACuriousMind Mod
May 5, 2021 at 11:22
• @ACuriousMind Indeed, that's the main thing I'm worried about when thinking about the quantitative side of these questions (but also in discovering and getting to those answers in order to perform any qualitative analyses at scale). May 5, 2021 at 13:03
• I have encountered users who actively try to fight the site policies. Even beyond homework questions, there are users who don't really care about policies and just answer questions if they want to answer them. I think there is a good subset of regular users who purposefully ignore policies; I'm not sure how to qualify that though. A survey? Haha May 5, 2021 at 17:37
• "we do have a problem with off-topic homework problems getting answers before they can be closed by the close-vote review queue": on the contrary, I'd argue that this site tends to close also good homework questions, giving the impression that many users vote to close without carefully reading the question. This is one example from the questions that I answered, where you can see that there is a conceptual question, but there are others too (too many, I think). May 5, 2021 at 19:01
• Note also that I do vote to close homework dumps, but I also think that homework questions are good to teach certain difficult concepts, or techniques that are not so well known (e.g. when it comes to solving circuits, many users tend to comment "Use Kirchhoff's laws", but there are many more techniques that can, and should be used to solve circuits). May 5, 2021 at 19:05
• @MassimoOrtolano "I also think that homework questions are good to teach certain difficult concepts, or techniques that are not so well known" I completely agree with this. However, this point is a little dicey, as it could be used to justify answering really any question despite whether it is on-topic or not. May 5, 2021 at 19:11
• "Who is answering these questions?" I have no stats to back this up, but I suspect that there are many people that are interested in physics that are hardly experts on the topic (I include myself in this group) and when they finally see a question that they actually can answer, they jump on it. May 6, 2021 at 1:10
• Two of our 100k+ users have the gold badge for the homework-tag. May 7, 2021 at 0:34
• @ZeroTheHero That almost qualifies as a stand-alone answer. Stellar observation. May 12, 2021 at 14:05
• @ZeroTheHero I was going to say "well, not all h/w is off-topic, so those users probably answered mostly still-open questions". But the data belies that $-$ about 30% of the h/w answers from those users are to questions that are currently closed. Then again, both the high rep and high number of answers in that sector correlate with long time on the site, so it should be taken with care. May 12, 2021 at 15:02
• @EmilioPisanty yeah I hear ya: taken with great care. Not sure myself what this means. May 12, 2021 at 15:21
• In retrospective thought, answering well written questions, does help the site and it's users, to have a better understanding of physics problems, and maybe that will reduce questions asked. (Consider the case when you google something, and you get stackexchange on the first results, and the same problem isn't solved, because it might be a homework question, I find that a bit too gatekeepy) May 23, 2021 at 7:13

Many of the questions that get closed on our site consist purely of the text (or an image) of a homework assignment, with no attempt by the OP to explain what concept relating to the assignment they need help with, or any thoughts on how they've tried to attack the problem. Some of us call such questions "homework dumps". On the Mathematics stack, they call them PSQs, Problem Statement Questions.

One problem with PSQs is that we generally don't know why the OP needs help. We don't know what misunderstanding or gap in their knowledge prevents them from doing the homework assignment themself. Occasionally, the OP reveals the relevant information in response to comments, but it can be hard work coming up with the right questions to extract that information, and sometimes the OP responds with impatience or even downright hostility in reaction to such "interrogation". Meanwhile, eager answerers attempt to guess what the OP's knowledge gap is, and write an answer on that basis, or simply write a solution to the problem that may be technically correct (or not) but which may not be a good match to the OP's level of schooling.

With the typical PSQ, it's unlikely that the OP will transform it into a question that's suitable for our site, no matter how much assistance we give them.

OTOH, I like to give people the benefit of the doubt, so I often post a comment on PSQs with this link to our policy on homework-like questions, and request that the OP ask a conceptual question that will help them solve their homework problem, but I must admit that it's very rare for the OP to reply with a satisfactory response.

If I get the feeling that the OP is genuinely interested in learning the physics behind their PSQ, I'm happy to assist them into getting their question into an answerable state. But if I get the impression that they just want the answer so they can hand in their homework, then I close-vote (and possibly downvote) and move on.

In summary, I agree with Catija that we probably don't need to make the closing process faster. AFAIK, an OP is very unlikely to substantially improve their question after it's closed, and I think it is pretty rare for closed questions to be reopened, especially if they have a negative score. It would be interesting to see actual statistics on that, though.

We do need to make it easier for new OPs to learn that PSQs are off-topic. And we do need to discourage answerers from answering them. (It'd also be nice if people would do a dupe search before answering questions that are highly likely to be duplicates...)

Coincidentally, the Mathematics stack is currently trying to improve question quality and to discourage people (especially those with high rep) from answering questions that ought to be closed. See here for details.

Their situation is a little different to ours, though. They do not have a close reason analogous to our homework-like reason for handling PSQs, and the community there is somewhat divided on whether such questions should be answered.

• Can you expand on your "In summary..." paragraph? What is your link between the speed of closure and the willingness of the OP to edit a question/likelihood of reopening a question after it is closed? Are you saying we should do a better job at giving time for the OP edit the question, and thus we need to do better at not answering the questions before that happens? May 6, 2021 at 1:25
• @BioPhysicist In theory, if a question is bad but has the potential to be salvaged, we close it to prevent premature answers from being posted before the question has been edited into an acceptable state. However, in practice, it's unlikely for closed questions to get the necessary improvements, and even if they do, they may not get sufficient reopen votes. So increasing closure speed is unlikely to be helpful for such questions. May 6, 2021 at 7:25
• Last year, the close vote dialogs were changed across the network. Those changes were first discussed & tested on SO, see meta.stackoverflow.com/q/394871/4014959 From that post, "Today on Stack Overflow, roughly 20% of questions are edited after they are closed and just 3% of closed questions are ever reopened. We’d like to see more questions improved upon so that they have a better chance of being reopened and answered". May 6, 2021 at 7:27
• Thanks for linking to what Math calls the Enforcement of Quality Standards. I recently posted an account of my experiences with it that explores the consequences of the current enforcement approach. Just as I hope those consequences will improve for future targets of the policy on MSE, I hope if Physics SE ever has similar plans they'll learn from MSE's experience too.
– J.G.
May 25, 2021 at 20:25

At the risk of seeming a bit hard-nosed; there is always the option of downvoting answers to blatant homework/examination questions. I certainly have done for egregious examples in the past and will be doing so more liberally in future, especially as we move into "exam season".

You should also flag a complete answer to a closed homework question for the attention of moderators: "If there are any complete answers, please flag them for moderator attention.", but this doesn't really address the issue of such questions being answered before they can be closed.

Personally I would prefer to see there being consensus over whether a question is a homework problem or not before downvoting answers that may indeed have correct physics in them. However, after the recent "Armored Chicken" furore, it is quite clear that there is support for using downvotes in this way even if it is just your opinion that the question should be closed. Folks, we need to talk about the surface-area-of-a-chicken question

OK, here's a first stab at the data, using the Data Explorer: Users who answer closed homework questions.

This query returns the total number of answers to h/w-tagged questions per user, and the number where the question is currently closed (but not deleted!); it also returns the total score in each category, and the corresponding ratios. The selection can be trimmed by the max age of the question in days, and by setting a minimal total number of h/w questions answered.

As to how best to probe that data, or what it tells us... I'm not sure yet. But I can link to it here and hopefully we can see if there's anything to be learned from it. If additional selectors would be useful, let me know, and I'll add them in if SEDE supports them.

(Finally, the usual caveats $$-$$ this query suffers from selection bias, since it does not include any deleted posts. It should include all upvoted answers to subsequently-deleted h/w questions, since those will survive the roomba, but any closed homework questions without upvoted answers have a $${\sim}9$$ day horizon to make it into the data before they're deleted.)

• Thanks. Still informative despite some of the restrictions. May 12, 2021 at 18:11
• Something to consider with that data could be the time the user has been active on PSE. I answered way more homework questions when I first started than I do now just because I am more familiar with the policies. I am sure my "current percentage" is not as high as my overall percentage. May 12, 2021 at 18:22
• @BioPhysicist Indeed. That's why I added the answer age as a key parameter. I toyed mostly with 30-60 days (with minimum of 5ish total answers in the tag) which seemed to produce more robust data, but I can't quite pin down what contributes to that impression. May 12, 2021 at 18:51
• I think I am getting at something slightly different. The age of the answer doesn't correspond to the "age" of the user at the time they made that answer. I haven't critically thought about why this matters too much or what to do about it, but just in case I wanted to bring it up. May 12, 2021 at 18:56
• Even many of the top users are guilty of the horrific offense of answering homework questions. Is it possible we're just being too uptight about this issue? May 19, 2021 at 18:27
• @ThePhoton I think it is something to be uptight about if the goal is for PSE to not turn into a homework-help site. If the policy is pretty loose then anyone wanting to post an off-topic homework question will have many examples of "well they got an answer, why can't I?" as well as other users being able to say "well they were allowed to answer, why can't I?". Of course, if we want PSE to have a homework-help element to it then of course we are being too strict, but I think the consensus is to not let it be that way at the moment. May 19, 2021 at 18:34
• What do you think of closed questions that get a lot of upvotes, or answers to closed questions with a lot of upvotes. There are competing interests here, with the community voting with up/down votes and high-rep users trying to enforce the homework policy. May 21, 2021 at 14:06

As quite a new user, here are some comments:

1. It didn't seem obvious at first that we aren't meant to answer homework questions, sometimes they have been edited by a gold medal holder and seemed to be legitimate questions. A message saying we shouldn't give detailed answers came after a while, so that helps. Perhaps that message should be there on joining if the community decides homework questions aren't desirable on this site.

2. It also doesn't seem clear what to do if we see a clearly homework type question. There is the 'flag', but on clicking it, it doesn't have a 'report homework question' option - just a few others that don't really seem appropriate to use to flag a homework question. The vote to close option isn't always there, perhaps it's for members of a certain reputation.

3. Lastly, rather than discouraging the (probably young) future users of the site, maybe members should be reminded regularly to downvote answers to homework questions that are too detailed. New members could also we warned by the message in 1) that answers to homework type questions that are too detailed are likely to result in downvotes.

Downvoting seems the ideal tool to deal with this, but to create and keep a culture of downvoting too detailed answers of homework questions, there may need to be regular reminders or perhaps a small reminder box on the screen for all to see.

• On point 2, you can flag to close a question for the homework reason. May 12, 2021 at 19:45
• @ BioPhysicist No doubt you are right, but maybe could be clearer, for example on this recent question physics.stackexchange.com/questions/635846/… after it there is 'Share Cite Edit Follow Flag', on clicking 'Flag' there are 6 options. Did you mean to use the 'In need of moderator attention' one, and mention homework there? Maybe there should be a 7th to choose, about homework questions (- that is if the community really wants to keep them all away from the site) , or did you mean another option that I've missed? May 12, 2021 at 20:08
• "Needs improvement" takes you to closure reasons. From there you select "community specific reason" where you will find the homework reason. It is a bit of clicking, but there you go. It becomes easier when you can just directly vote to close, as it takes out the first step. May 12, 2021 at 22:19

As a physicist, before trying to answer a question, I am used to start by asking myself if there are hidden hypotheses in the way it is formulated.

In the case of this specific question, I see that the discussion is started as if there would be a clear cut between homework&homework-like (HMWK) questions and the others. Of course, this is not the way things go. It is true that there are crystalline clear examples of HMWK as well as non-HMWK questions. But there is a quite wide space in between. In particular, there are questions that were not originally tagged as homework by the authors but the tag has been added by others. In some cases with a good reason, in other cases, the reason was less obvious.

My personal feeling is that there is a significant number of cases of perfectly admissible questions that were closed as HMWK. This could be another way of looking at the phenomenon. I know that a serious investigation about how well-founded is my feeling would imply much more work than simply querying a database. However, at least a statistical sampling could be done. I think that we could find that the questions closed as HMWK that received more answers are those with the less evident characterization as homework. If this would be true, it would have an impact on the present discussion.

I don't have a direct answer to the questions asked, although I think they are good questions. Two ideas come to mind by way of experimental methods for approaching them:

1. Physics was not chosen for the 3-vote close experiment recently launched, but we could potentially have a DIY version if we asked our moderators, temporarily, to cast their votes early in the cycle rather than late. I just did my 20 reviews for the day and I noticed that many of the questionable questions had already been edited by a mod anyway, so perhaps this wouldn't be a big burden? If they are already seeing the questions almost as they are posted and are taking action on them anyway, we could have a short experiment of 1-vote closures to see how that impacts the quantity and quality of questions and answers. It does go against the site's philosophy of moderation a bit, but as a limited experiment maybe it's ok? If we did have 1-vote closure and we still have a problem, that would at least strongly indicate that the speed of the review process is not the driving factor.

2. With respect to the last group of questions regarding knowledge and intent of the answerers, perhaps instead of getting a modal that's meant to educate we could get one that asks some subset of the questions above. So don't just say, "hey, make sure you're answering a question that's within the homework policy" ask the person about to answer where they fall on the question's fit within the policy. So something like checkboxes for "Do you know that this site has a policy regarding homework-like questions and answers? Did you think this question was within the policy? Do you disagree with the policy and are answering anyway?" Presumably this would require help from SE to implement and track.

• For your first point, that feels dangerous. Not all users will know that test would be going on, so it will give the appearance that PSE mods are just really strict. May 10, 2021 at 19:14
• @BioPhysicist, I see your point, but I'm not sure that's a critical issue for a short run. Could be mitigated by leaving a comment at close. It also leads to another question, which I took as an assumption: How many people asking such questions never participate again? (I suspect a lot of these are people looking for one-time help on their homework that do not and would not anyway post again unless they have another homework problem.) May 12, 2021 at 16:45
• Yeah, it's a tricky thing to handle here. May 12, 2021 at 16:46
• Some mods have commented that they do not like to unilaterally close questions, and prefer to see at least some close votes by regular community members before they invoke their mod powers. Of course, that may not apply if the question is blatantly off-topic. May 13, 2021 at 2:25

Why not hold all questions for a week or two before posting them? That would eliminate the motivation for submitting homework problems. (Concepts can wait.)

• While this would solve the problem form me (tho 2 days, not 2 weeks), it’s not gonna happen: the basic tenet of the site is to grow the user base, and this would go against this. In a way it’s simpler from a corporate point of view to let users sign up and their question be closed rather than have them wait for an extended period. Maybe 4 hrs might work as a deterrent to impatient users in a real rush (read: exam questions) but I doubt your idea will be implemented. May 15, 2021 at 14:41

I have been asked, as an ostensibly "pro homework" contributor, to participate in these discussions.

I apologize If the following has already been addressed regarding the threshold for closing H&E questions, but it seems to me that a single vote to close should be the threshold for one that simply states a problem and nothing else.