This is a follow-up to What happened with the experiment to lower the vote-to-close threshold?. (See that thread, and the initial proposal here, for background.) The experiment is currently running on a set of (other) Stack Exchange sites that need it more, and need it for other reasons. The SE Community team seems somewhat interested in running it for our use-case, but they don't seem convinced that the case is strong enough that it would do much of a difference.
In the meantime, the Community Manager in charge of the experiment, Catija, suggested (in the experiment announcement at MSE) that
there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer
and this is something we should consider carefully.
I would argue (contrary to this recent thread) that we do have a problem with off-topic homework problems getting answers before they can be closed by the close-vote review queue, which is currently too slow to deal with this. One way to handle this is to make the queue faster (by requiring fewer votes per closure), but that may not be the only way. As Catija suggests,
Maybe you need better guidance to prevent the questions from being asked such as a customized Ask modal or sidebar. Maybe better education for the people answering about when not to answer...
The modal in question currently looks like this, but it can be customized to look like e.g. this (at Code Review). This is something we should definitely look into, but I don't think this problem lies on the side of the askers $-$ I would argue that our problem (off-topic questions getting answers before they can be closed) is largely a problem of user education, i.e., we are not as effective as we should be at communicating site policies to new and relatively-new users.
As a way to address that education problem, I suggested to Catija that we could install
something like a modal, or ideally a warning box that appears above the answer text editor box (even better, something that pops in once the user clicks on the editor box), which reminds the user what kinds of h/w questions are on-topic and which ones shouldn't be answered, and which appears, say, on questions with the h/w tag or which have one or more pending close votes with that close reason
and she seems to be quite open to the idea. But she has this important set of questions as a framing for such a device:
But it's important to understand the answerers' motivations. Do they disagree with the questions being close-worthy? Are they unaware that the question should be closed? Do they have thoughts on how they can improve the question - it's possible that the best case scenario for the asker and the answerers is that the questions get improved first and answered - so it's not so much that they can't answer the question but that they should work with the asker to make it a good question before offering a (possibly problematic) answer.
My question here
Catija's last comment raises an important point. If we have a problem, we should understand it as well as possible before fumbling for solutions. So:
- How many off-topic homework-like questions get answered before they get closed?
- Can we reliably measure at what stage in the closing process this happens?
- Who is answering these questions? How much experience do they have with the site, in terms of both time and reputation?
- Are they unaware that there is a site policy that strongly discourages such answers? Or are they aware of it but they disagree with it? Or do they disagree that it applies to that particular question?
I think that we could probably achieve quite a lot using a pop-up modal, or a pop-in banner, when at-risk users (i.e. the population in question here) attempt to answer at-risk questions (i.e. questions with the homework tag or with a homework-like close-vote pending), but if we want that communication to be effective then we do need to understand who we're talking to.