Getting a better understanding of how and why off-topic homework-like questions get answered before they can be closed

Question

Background

This is a follow-up to What happened with the experiment to lower the vote-to-close threshold?. (See that thread, and the initial proposal here, for background.) The experiment is currently running on a set of (other) Stack Exchange sites that need it more, and need it for other reasons. The SE Community team seems somewhat interested in running it for our use-case, but they don't seem convinced that the case is strong enough that it would do much of a difference.

In the meantime, the Community Manager in charge of the experiment, Catija, suggested (in the experiment announcement at MSE) that

there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer

and this is something we should consider carefully.

I would argue (contrary to this recent thread) that we do have a problem with off-topic homework problems getting answers before they can be closed by the close-vote review queue, which is currently too slow to deal with this. One way to handle this is to make the queue faster (by requiring fewer votes per closure), but that may not be the only way. As Catija suggests,

Maybe you need better guidance to prevent the questions from being asked such as a customized Ask modal or sidebar. Maybe better education for the people answering about when not to answer...

The modal in question currently looks like this, but it can be customized to look like e.g. this (at Code Review). This is something we should definitely look into, but I don't think this problem lies on the side of the askers $-$ I would argue that our problem (off-topic questions getting answers before they can be closed) is largely a problem of user education, i.e., we are not as effective as we should be at communicating site policies to new and relatively-new users.

As a way to address that education problem, I suggested to Catija that we could install

something like a modal, or ideally a warning box that appears above the answer text editor box (even better, something that pops in once the user clicks on the editor box), which reminds the user what kinds of h/w questions are on-topic and which ones shouldn't be answered, and which appears, say, on questions with the h/w tag or which have one or more pending close votes with that close reason

and she seems to be quite open to the idea. But she has this important set of questions as a framing for such a device:

But it's important to understand the answerers' motivations. Do they disagree with the questions being close-worthy? Are they unaware that the question should be closed? Do they have thoughts on how they can improve the question - it's possible that the best case scenario for the asker and the answerers is that the questions get improved first and answered - so it's not so much that they can't answer the question but that they should work with the asker to make it a good question before offering a (possibly problematic) answer.

---

My question here

Catija's last comment raises an important point. If we have a problem, we should understand it as well as possible before fumbling for solutions. So:

• How many off-topic homework-like questions get answered before they get closed?
• Can we reliably measure at what stage in the closing process this happens?
• Who is answering these questions? How much experience do they have with the site, in terms of both time and reputation?
• Are they unaware that there is a site policy that strongly discourages such answers? Or are they aware of it but they disagree with it? Or do they disagree that it applies to that particular question?

I think that we could probably achieve quite a lot using a pop-up modal, or a pop-in banner, when at-risk users (i.e. the population in question here) attempt to answer at-risk questions (i.e. questions with the homework tag or with a homework-like close-vote pending), but if we want that communication to be effective then we do need to understand who we're talking to.

Answer 1

Many of the questions that get closed on our site consist purely of the text (or an image) of a homework assignment, with no attempt by the OP to explain what concept relating to the assignment they need help with, or any thoughts on how they've tried to attack the problem. Some of us call such questions "homework dumps". On the Mathematics stack, they call them PSQs, Problem Statement Questions.

One problem with PSQs is that we generally don't know why the OP needs help. We don't know what misunderstanding or gap in their knowledge prevents them from doing the homework assignment themself. Occasionally, the OP reveals the relevant information in response to comments, but it can be hard work coming up with the right questions to extract that information, and sometimes the OP responds with impatience or even downright hostility in reaction to such "interrogation". Meanwhile, eager answerers attempt to guess what the OP's knowledge gap is, and write an answer on that basis, or simply write a solution to the problem that may be technically correct (or not) but which may not be a good match to the OP's level of schooling.

With the typical PSQ, it's unlikely that the OP will transform it into a question that's suitable for our site, no matter how much assistance we give them.

OTOH, I like to give people the benefit of the doubt, so I often post a comment on PSQs with this link to our policy on homework-like questions, and request that the OP ask a conceptual question that will help them solve their homework problem, but I must admit that it's very rare for the OP to reply with a satisfactory response.

If I get the feeling that the OP is genuinely interested in learning the physics behind their PSQ, I'm happy to assist them into getting their question into an answerable state. But if I get the impression that they just want the answer so they can hand in their homework, then I close-vote (and possibly downvote) and move on.

In summary, I agree with Catija that we probably don't need to make the closing process faster. AFAIK, an OP is very unlikely to substantially improve their question after it's closed, and I think it is pretty rare for closed questions to be reopened, especially if they have a negative score. It would be interesting to see actual statistics on that, though.

We do need to make it easier for new OPs to learn that PSQs are off-topic. And we do need to discourage answerers from answering them. (It'd also be nice if people would do a dupe search before answering questions that are highly likely to be duplicates...)

---

Coincidentally, the Mathematics stack is currently trying to improve question quality and to discourage people (especially those with high rep) from answering questions that ought to be closed. See here for details.

Their situation is a little different to ours, though. They do not have a close reason analogous to our homework-like reason for handling PSQs, and the community there is somewhat divided on whether such questions should be answered.

Answer 2

At the risk of seeming a bit hard-nosed; there is always the option of downvoting answers to blatant homework/examination questions. I certainly have done for egregious examples in the past and will be doing so more liberally in future, especially as we move into "exam season".

You should also flag a complete answer to a closed homework question for the attention of moderators: "If there are any complete answers, please flag them for moderator attention.", but this doesn't really address the issue of such questions being answered before they can be closed.

Personally I would prefer to see there being consensus over whether a question is a homework problem or not before downvoting answers that may indeed have correct physics in them. However, after the recent "Armored Chicken" furore, it is quite clear that there is support for using downvotes in this way even if it is just your opinion that the question should be closed. Folks, we need to talk about the surface-area-of-a-chicken question

Answer 3

OK, here's a first stab at the data, using the Data Explorer: Users who answer closed homework questions.

This query returns the total number of answers to h/w-tagged questions per user, and the number where the question is currently closed (but not deleted!); it also returns the total score in each category, and the corresponding ratios. The selection can be trimmed by the max age of the question in days, and by setting a minimal total number of h/w questions answered.

As to how best to probe that data, or what it tells us... I'm not sure yet. But I can link to it here and hopefully we can see if there's anything to be learned from it. If additional selectors would be useful, let me know, and I'll add them in if SEDE supports them.

(Finally, the usual caveats $-$ this query suffers from selection bias, since it does not include any deleted posts. It should include all upvoted answers to subsequently-deleted h/w questions, since those will survive the roomba, but any closed homework questions without upvoted answers have a ${\sim}9$ day horizon to make it into the data before they're deleted.)

Answer 4

As quite a new user, here are some comments:

1. It didn't seem obvious at first that we aren't meant to answer homework questions, sometimes they have been edited by a gold medal holder and seemed to be legitimate questions. A message saying we shouldn't give detailed answers came after a while, so that helps. Perhaps that message should be there on joining if the community decides homework questions aren't desirable on this site.

2. It also doesn't seem clear what to do if we see a clearly homework type question. There is the 'flag', but on clicking it, it doesn't have a 'report homework question' option - just a few others that don't really seem appropriate to use to flag a homework question. The vote to close option isn't always there, perhaps it's for members of a certain reputation.

3. Lastly, rather than discouraging the (probably young) future users of the site, maybe members should be reminded regularly to downvote answers to homework questions that are too detailed. New members could also we warned by the message in 1) that answers to homework type questions that are too detailed are likely to result in downvotes.

Downvoting seems the ideal tool to deal with this, but to create and keep a culture of downvoting too detailed answers of homework questions, there may need to be regular reminders or perhaps a small reminder box on the screen for all to see.

Answer 5

As a physicist, before trying to answer a question, I am used to start by asking myself if there are hidden hypotheses in the way it is formulated.

In the case of this specific question, I see that the discussion is started as if there would be a clear cut between homework&homework-like (HMWK) questions and the others. Of course, this is not the way things go. It is true that there are crystalline clear examples of HMWK as well as non-HMWK questions. But there is a quite wide space in between. In particular, there are questions that were not originally tagged as homework by the authors but the tag has been added by others. In some cases with a good reason, in other cases, the reason was less obvious.

My personal feeling is that there is a significant number of cases of perfectly admissible questions that were closed as HMWK. This could be another way of looking at the phenomenon. I know that a serious investigation about how well-founded is my feeling would imply much more work than simply querying a database. However, at least a statistical sampling could be done. I think that we could find that the questions closed as HMWK that received more answers are those with the less evident characterization as homework. If this would be true, it would have an impact on the present discussion.

Answer 6

I don't have a direct answer to the questions asked, although I think they are good questions. Two ideas come to mind by way of experimental methods for approaching them:

1. Physics was not chosen for the 3-vote close experiment recently launched, but we could potentially have a DIY version if we asked our moderators, temporarily, to cast their votes early in the cycle rather than late. I just did my 20 reviews for the day and I noticed that many of the questionable questions had already been edited by a mod anyway, so perhaps this wouldn't be a big burden? If they are already seeing the questions almost as they are posted and are taking action on them anyway, we could have a short experiment of 1-vote closures to see how that impacts the quantity and quality of questions and answers. It does go against the site's philosophy of moderation a bit, but as a limited experiment maybe it's ok? If we did have 1-vote closure and we still have a problem, that would at least strongly indicate that the speed of the review process is not the driving factor.

2. With respect to the last group of questions regarding knowledge and intent of the answerers, perhaps instead of getting a modal that's meant to educate we could get one that asks some subset of the questions above. So don't just say, "hey, make sure you're answering a question that's within the homework policy" ask the person about to answer where they fall on the question's fit within the policy. So something like checkboxes for "Do you know that this site has a policy regarding homework-like questions and answers? Did you think this question was within the policy? Do you disagree with the policy and are answering anyway?" Presumably this would require help from SE to implement and track.

Answer 7

Why not hold all questions for a week or two before posting them? That would eliminate the motivation for submitting homework problems. (Concepts can wait.)

Answer 8

I have been asked, as an ostensibly "pro homework" contributor, to participate in these discussions.

I apologize If the following has already been addressed regarding the threshold for closing H&E questions, but it seems to me that a single vote to close should be the threshold for one that simply states a problem and nothing else.

Thanks for your consideration.