After discussion about this was started two and a half years ago, I'm here to share that I'm enabling a test of three-vote close on Physics today. Firstly, I'd like to thank you for your patience. There's been a lot of discussion about this and I mentioned my primary concerns about making this change here the last time I posted, about six months ago.
At the time, I wasn't certain that my data was quite correct and I've gotten a chance to review it. I will say that, while there was clearly something wrong with the data, in the end, the results were the same - essentially, if your primary hope is that three-vote close will prevent lots of close-worthy questions from being answered... and the answer is mostly - it'll prevent some but not the majority.
The chart below needs a bit of explanation - it graphs the number of answered questions closed per month by users - moderators are excluded since this change would only impact user closures - grouped by whether the answer was posted before or after the third vote to close - that is, whether the outcome ("got answered") would have been changed if three-vote closure had been live on the site.
OutcomeNotChanged_NoModClosures
- the upper (yellow) line indicates the number of questions per month that were closed when the first answer came in before the third vote to close. Each month, this was around 250-350 until July 2021, and it's since dropped to around 100-150 per month - this change coincides with your last moderator election.OutcomeChanged_NoModClosures
- the lower (blue) line indicates the number of questions per month that were closed when the first answer came in after the third vote to close. Each month, this was around 30-40 until July 2021, and it's since dropped to around 20-30.
Looking at recent months, here are the numbers - as with the chart above, this only covers closed questions that have answers, not all closed questions:
Month | Outcome not changed | Outcome changed | Total | % not changed | % changed |
---|---|---|---|---|---|
2022-01-01 | 156 | 41 | 197 | 79.2% | 20.8% |
2022-02-01 | 123 | 30 | 153 | 80.4% | 19.6% |
2022-03-01 | 111 | 16 | 127 | 87.4% | 12.6% |
2022-04-01 | 132 | 31 | 163 | 81.0% | 19.0% |
2022-05-01 | 133 | 20 | 153 | 86.9% | 13.3% |
These numbers would have been lower the last time I checked, since it was prior to (or very soon after) your last election. It's not uncommon on medium sized sites like this for an election to have big impact on reviewing since the person who gets elected is frequently an active reviewer - as such, I wanted to check in on the numbers for the balance between community closure and moderator closure.
As I somewhat expected, while users had been closing more without moderator action prior to the election, since, it's pretty much 50-50.
There also seems to be a drop in overall closures, so I wanted to check whether the number of questions completing reviews had dropped.
Fortunately, it seems like the percentage completed hasn't changed or it's improved (it's more likely to be between 80-90% competed than 70-80% completed) - so if there's a reduction in questions closed it's either due to there being fewer nominated for closure initially or there are fewer close-worthy questions asked.
One thing that I do notice is that you have a relatively high percentage of "leave open" reviews, which can indicate disagreement between the first flagger and the reviewers or it can mean that you get a lot of edits that improve questions to the point they're salvageable before getting closed. While this is generally under 10%, this is high compared to many other sites that I've reviewed.
One thing that the moderators also mentioned as a concern is just how full the close review queue is at times. This can seem like a huge load that will be difficult to overcome and I can understand that. It's much nicer to see only a handful of questions in review rather than 100 or more!
For all of these reasons, I've enabled three vote close and reopen as of this posting.
Please use answers here to bring up any concerns or issues you run into. Because there's so much more close activity here, we'll be checking in after about 6 months to make sure things aren't going off the rails but if you're noticing issues, please bring them to my attention before then - our main concern is close-reopen warring.