Abstract: Analyzing the data from the last replace-the-homework-policy question was inconclusive. So back to the drawing board, or really back to our starting point: what kinds of questions do we really want to see closed? In particular, what are better ways to characterize our actual reasons for closing questions where we currently use the "homework-like" close reason?
Important note, because I guess this hasn't been clear in the past: the purpose of this series of posts is to come up with a brand new set of guidelines for question closure. We are not reformulating or tweaking or improving our current homework policy. We are replacing it entirely - that means we are getting rid of it and developing a new set of policies and close reasons, which will presumably make no mention of "homework" or "homework-like". Once this process is done, there will be no homework policy and no "homework-like" close reason.
As some/many/most people are aware, we are in the midst of a long-term project to replace our homework policy with a new set of guidelines that better reflect what people actually consider on topic and off topic. In the first phase of this project, we collected some example questions and voted on whether they should be on or off topic, to help clarify the new guidelines we want to come up with. I also collected some opinions on various qualities of these questions in a separate survey. The goal was to try to find some correlation between the topicality of the questions, as represented by the score, and one or a few of the attributes, and that would tell us what are the main factors people consider while voting to close.
As it turns out... that doesn't work very well. I spent a long time tinkering with the data and basically what I found is that people's opinions vary widely, making it hard to draw meaningful conclusions. The first thing I checked was whether any individual factor from the survey correlated well to people's impression of whether a question was on topic. There were some correlations with physical context and interest, and to a lesser extent level and effort, and an anticorrelation with tediousness (this means that within our data set, people are more likely to consider questions which concern tedious calculations on topic), but none of these are very strong.
I also looked at whether a combination of factors could provide a good discriminator between on-topic and off-topic questions, in a couple different ways. First, a partial least squares regression to test whether some linear combination of ratings would be able to accurately predict the scores of the questions, but that didn't work at all.
The other approach, which I think best mirrors how the actual close-voting logic works, is to find a set of factors such that the on-topic questions rate highly on all factors but each off-topic question rates low on one or more of them. This plot shows the mean minus one standard deviation1 of the distribution of ratings for each of the highest-score (most clearly on-topic) sample questions:
and this is the equivalent, showing mean plus one standard deviation, for the lowest-score (most clearly off topic) sample questions:
I was looking for a group of one or a few rating factors such that the corresponding columns in the top plot are all red or white, and the columns in the bottom plot between them contain one blue cell for each question. Unfortunately, there is no clear candidate. I even had the computer go through all possible combinations of rating factors to test them out, and all the combinations in which the on-topic questions rank high have large uncertainties in the off-topic questions, and vice-versa. The one factor that does keep popping up is "check-my-work-ness", which may be an indicator that people agree that questions just asking us to check work are off topic and that we do okay at identifying which questions those are.
Anyway, the point seems to be that none of the factors in the survey are a particularly good proxy for how we decide whether questions are on or off topic. (And remember, we're mostly talking about questions which we currently close or might consider closing using the homework-like reason.) So before we proceed, I want to throw the question back to the community in a revisit of the question where I first tried to collect possible close reasons. In light of all the discussion and analysis we've done since then, when we choose to vote to close questions as homework-like, what reasons do we actually have in mind? Or, how could we do more research to work this out?
1The idea is to identify clearly on-topic questions as those for which something approximating the 68% confidence interval of the score is entirely positive, indicating a high score and clear agreement on that score. It's basically the same idea underlying e.g. the reddit scoring algorithm.