In December 2019, the moderator team posted the question Should we test lowering the vote to close and reopen threshold?, which was positively received, and there was a clear agreement and intention to go forward with the experiment.

What is the status of this experiment? Are still waiting for the full data-taking protocol to be finalized? Is everything ready to go and we just need to push the button? Was it swallowed up by The Monster (a.k.a. the year 2020)?

I don't mean to berate anyone -- just a quick prod to see if we can get this back on track =).

  • 1
    $\begingroup$ You have an answer which explains the situation from the perspective of the volunteer mods. I've added the magic tag status-review to invite SE staff to offer any updates which may be shareable from their end (or to confirm that we've understood things mostly right). $\endgroup$ – rob Mod Dec 10 '20 at 23:18
  • 2
    $\begingroup$ An update today: meta.stackexchange.com/a/358188/280545 $\endgroup$ – rob Mod Dec 17 '20 at 17:43
  • 3
    $\begingroup$ You may have seen that we conducted a test on 13 network sites — please stay tuned for the results of that step and next steps on this. $\endgroup$ – JNat ModStaff Jul 9 at 10:25

Ah Dec. 2019... those were simpler times eh?

So what happened was by Jan. 2020 we were on the list and working to get a more complete dataset defined to extract, and the Community Manager (CM) who was going to write the scripts to do that analysis was let go from the company. So that kind of set things back on getting any sort of custom data defined.

Then the rest of 2020 was happening and more changes to the CM team and so on, and so the project was put on hold network-wide for awhile. While it was on hold, a large number of sites said they wanted to lower the threshold, so now there's a backlog of sites that would like to do it.

My understanding is that the lowering of vote thresholds on sites will resume sometime soon, possibly even this month. But rather than a FIFO queue, it's going to be prioritized on getting sites that are really struggling under 5-vote-closures done first and then moving on to other sites that aren't having a hard time but would still like to do it.

My guess is that we're going to be somewhere low on the urgency list because we do have a really good group of people who are voting to close things and most stuff does get to 5 votes, although it often takes longer than we like it to. It also probably means we won't have a lot of customized analysis done. It just isn't in the cards.

While I'm not sure where we are in the priority queue, I am keeping tabs on it because we'd still like to see how it goes!

  • 2
    $\begingroup$ Simpler times indeed! Thanks for the update. $\endgroup$ – Emilio Pisanty Dec 10 '20 at 19:36
  • 2
    $\begingroup$ It is a bit of a bummer that we've been bumped down and won't get a customized analysis, but I suspect it has a silver lining in that the data-analysis team will have a rather better handle on how the process looks like by the time they get to us. $\endgroup$ – Emilio Pisanty Dec 10 '20 at 19:38
  • 5
    $\begingroup$ @EmilioPisanty No problem, I'm glad there's still interest in it from somebody other than me! I think once some sites start getting switched over, I'll bug somebody to figure out where in the list we actually are. Until the process starts for the urgent sites though, there's not much use in knowing our rank and I don't want to be too annoying! I'll keep everybody updated when I find out more info. $\endgroup$ – tpg2114 Dec 10 '20 at 20:09

Y'all may have seen the update I wrote as it was linked in the comments but I wanted to give y'all a specific update because the use case y'all have for this is very different from our goals of the test.

Our focus is on doing two things:

  1. Increasing the percentage of review tasks that are completed.
  2. Reducing the percentage of review tasks that are completed by moderators who cast first or second votes to close.

Let's look at your stats on those two things:

In a 90 day period starting 30 days ago and reaching back 120 days, you had 3092 questions get flagged or a first vote to close. Of those, 70% were closed, 8% were left open and the remainder 22% were incomplete. In the grand scheme of things, those are pretty decent stats. It's not 100% but some sites are doing far worse.

Over the same period, moderators closed 887 questions and the community closed 2396. Now... this adds up to 3283, which isn't 3092, which is likely because the queries aren't perfectly aligned in how they define voting or flagging to close but if we look at the numbers, that means mods are closing (at some point in the process) about 27% of the questions. I haven't looked at the breakdown on which vote the mods are closing - but, while a somewhat high number, this also isn't a burdensome percentage, compared to some sites.

So, I'll need a bit more info before I can judge whether y'all fall into the groups where we're looking for improvements. You do have a large volume of posts that are going through review, which is an important point - y'all have the seventh largest number of posts during this time period. Many of the sites with worse percentages aren't very big and don't have a ton of reviews to do. But, of the top ten sites, you have the second highest completion rate, with only SO in Portuguese doing better (84%) and Math a close third at 76%.

That said, based on a query I have, I'm not sure that this test will do what y'all are hoping. This query looks at all of the questions closed in a period of time and lists how many were closed with each number of votes and then how many of those questions were answered before being closed. It then calculates, of those that were answered, how many votes to close were there when closure occurred.

For the most recent 90 days (not the same period as above), here's what I get:

# of Close Votes when closed Closed Qs Answered Average close votes at first answer
1 498 114 0
2 286 97 1
3 158 63 1.314
4 113 53 1.814
5 1544 869 1.678

The really the important rows here - assuming this is correct - when you have five vote closures on questions, the average number of votes to close that exist before the question is answered is 1.67 or 1.8 with four vote closures. That means that many questions - more than half - are answered before the question gets three close votes.

I'd have to look at a distribution to see where all of those answers hit but I'm not a SQL master, so I'll have to find someone to write the query for me. The data above is from a query that Shog wrote and Nicolas refined since Shog's version was somehow incorrect.

The takeaway here is... we can test this but I think that to get this answering before closure thing in check, it's going to require more than just changing the number of votes to close - y'all will have to really start thinking about why these questions are getting answers and understanding where the disconnect is -

  • Do you have people who disagree about whether questions should be closed? If so, changing this could drastically increase close-reopen wars since it becomes far easier to close and reopen posts. Maybe better understanding and consensus could help with this.
  • Do you have people who aren't informed about when questions shouldn't be answered because they're out of scope? If so, how can you help inform them to try and prevent answers?
  • Something else? Y'all know your community far better than I - why do you think you get so many quick answers to close-worthy questions? What do you think you can do to change this?

Let me know what you think.

  • 4
    $\begingroup$ If I accumulate the rows in your table (i.e. there were 498+286 questions closed with two or fewer votes), I see that 29% of three-or-fewer-votes closed questions got answers, while 46% of five-votes-closed questions got answers. One interpretation is that switching to three close votes would eliminate about a third (= 1 - (29/46)) of our answers to off-topic questions. That sounds to me like a big gain for a small cost. $\endgroup$ – rob Mod Dec 22 '20 at 21:50
  • $\begingroup$ You're not allowing for the fact that 50% of the 869 received answers when they had only 1.68 votes to close... so, even assuming that all of the other 50% got answers when they had three or four votes (which would be conservative, considering 1.68 is less than two) you're looking at ~435 or 28%, not 46%. $\endgroup$ – Catija ModStaff Dec 22 '20 at 22:03
  • 4
    $\begingroup$ I suspect that our off-topic homework questions are much more likely to get responses from newer users who don’t know the ropes yet. An interesting query would be to divide the group of answered-but-closed questions depending on whether the answerer has or lacks the “close questions” privilege. If we have lots of high-rep users who are answering off-topic questions before they are closed, that’s a lack of consensus in the community; if we have new people grabbing low-hanging fruit before it’s closed, that’s a communication issue. $\endgroup$ – rob Mod Dec 22 '20 at 22:03
  • $\begingroup$ Re “you’re not allowing”: I think I am, by accumulating. A fifth (22%) of the one-close-voted questions get an answer; a quarter (26%) of the two-close-voted questions get an answer; half (46%) of the five-close-voted questions get an answer. That’s consistent with the mean number of close votes at the first answer being between one and two. $\endgroup$ – rob Mod Dec 22 '20 at 22:11
  • 1
    $\begingroup$ Oh, and before I forget: thanks for the update, and thanks for the engagement with us! $\endgroup$ – rob Mod Dec 22 '20 at 22:12
  • $\begingroup$ So, I'm not a mathematician but... I just don't know that you can extrapolate that way? There's way more that get closed with five, so they're likely questions that are more disputable whether they're close-worthy since mods tend to single vote close stuff that's blatant more than stuff that's debatable. You're also not including the 40% that get an answer before getting closed with only three votes. $\endgroup$ – Catija ModStaff Dec 22 '20 at 22:16
  • $\begingroup$ I don’t know that I can either, but it’s hard to fit subtlety and finesse into a comment. Perhaps another query to run would be this: of the questions that were closed with four or five votes, how many had answers posted after the third vote? Those are the answers that would have been blocked by a lower close-vote threshold. $\endgroup$ – rob Mod Dec 22 '20 at 22:22
  • $\begingroup$ I'm also starting to question the average column... the average being exactly 1 for the 2 vote closures seems questionable... if you think about it, it'd be a combination of 0 votes and 1 votes... so shouldn't it be less than 1? The only way it could be exactly 1 is if every answer came when there was one vote and none came when there were 0 votes. $\endgroup$ – Catija ModStaff Dec 22 '20 at 22:22
  • $\begingroup$ Oooh, you’re right, that’s quite improbable. $\endgroup$ – rob Mod Dec 22 '20 at 22:23
  • 1
    $\begingroup$ I've also looked at it for 180 and 360 days and I'm still getting 1, which means, yeah, something's off. I'll see if Nicolas can look into it. $\endgroup$ – Catija ModStaff Dec 22 '20 at 22:24
  • 1
    $\begingroup$ @Catija I support the previous comment to the effect that many of the questions with VTC that get answers are answered by less experienced users - not necessarily less rep but less time on the site. The last 120 days are not representative for me as I have basically given up on VTC after months of quickly burning through my 20 votes and seeing no improvement in the size of review queue. We sorely need the 3-VTC rule. $\endgroup$ – ZeroTheHero Dec 23 '20 at 2:32

Update, May 2021

As recently announced on Meta Stack Exchange at Testing three-vote close and reopen on 13 network sites, SE has just launched into this experiment, on sites where either (a) questions are ageing out of the review queue before they can be closed, or (b) moderators are carrying out most of the closures.

Regarding us (and potentially other sites in similar situations?), the announcement has this to say:

I know that some sites want to test this because they'd like to see questions closed more quickly, before they get answered - I understand this instinct and I'm not saying that we won't consider testing whether this improves the situation on those sites in the future but, right now, speed to closure isn't something we're focused on and there may be more effective solutions to preventing answers to close-worthy questions than closing the question before someone writes an answer.

I'm not sure what that last bit really means, but I've asked, so let's see what the team says.

  • $\begingroup$ Is the experiment still running? I don't know where to look to find a final decision. $\endgroup$ – uhoh 14 hours ago

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .