Y'all may have seen the update I wrote as it was linked in the comments but I wanted to give y'all a specific update because the use case y'all have for this is very different from our goals of the test.
Our focus is on doing two things:
- Increasing the percentage of review tasks that are completed.
- Reducing the percentage of review tasks that are completed by moderators who cast first or second votes to close.
Let's look at your stats on those two things:
In a 90 day period starting 30 days ago and reaching back 120 days, you had 3092 questions get flagged or a first vote to close. Of those, 70% were closed, 8% were left open and the remainder 22% were incomplete. In the grand scheme of things, those are pretty decent stats. It's not 100% but some sites are doing far worse.
Over the same period, moderators closed 887 questions and the community closed 2396. Now... this adds up to 3283, which isn't 3092, which is likely because the queries aren't perfectly aligned in how they define voting or flagging to close but if we look at the numbers, that means mods are closing (at some point in the process) about 27% of the questions. I haven't looked at the breakdown on which vote the mods are closing - but, while a somewhat high number, this also isn't a burdensome percentage, compared to some sites.
So, I'll need a bit more info before I can judge whether y'all fall into the groups where we're looking for improvements. You do have a large volume of posts that are going through review, which is an important point - y'all have the seventh largest number of posts during this time period. Many of the sites with worse percentages aren't very big and don't have a ton of reviews to do. But, of the top ten sites, you have the second highest completion rate, with only SO in Portuguese doing better (84%) and Math a close third at 76%.
That said, based on a query I have, I'm not sure that this test will do what y'all are hoping. This query looks at all of the questions closed in a period of time and lists how many were closed with each number of votes and then how many of those questions were answered before being closed. It then calculates, of those that were answered, how many votes to close were there when closure occurred.
For the most recent 90 days (not the same period as above), here's what I get:
|# of Close Votes when closed
||Average close votes at first answer
The really the important rows here - assuming this is correct - when you have five vote closures on questions, the average number of votes to close that exist before the question is answered is 1.67 or 1.8 with four vote closures. That means that many questions - more than half - are answered before the question gets three close votes.
I'd have to look at a distribution to see where all of those answers hit but I'm not a SQL master, so I'll have to find someone to write the query for me. The data above is from a query that Shog wrote and Nicolas refined since Shog's version was somehow incorrect.
The takeaway here is... we can test this but I think that to get this answering before closure thing in check, it's going to require more than just changing the number of votes to close - y'all will have to really start thinking about why these questions are getting answers and understanding where the disconnect is -
- Do you have people who disagree about whether questions should be closed? If so, changing this could drastically increase close-reopen wars since it becomes far easier to close and reopen posts. Maybe better understanding and consensus could help with this.
- Do you have people who aren't informed about when questions shouldn't be answered because they're out of scope? If so, how can you help inform them to try and prevent answers?
- Something else? Y'all know your community far better than I - why do you think you get so many quick answers to close-worthy questions? What do you think you can do to change this?
Let me know what you think.