Testing 3-vote close and reopen

Question

After discussion about this was started two and a half years ago, I'm here to share that I'm enabling a test of three-vote close on Physics today. Firstly, I'd like to thank you for your patience. There's been a lot of discussion about this and I mentioned my primary concerns about making this change here the last time I posted, about six months ago.

At the time, I wasn't certain that my data was quite correct and I've gotten a chance to review it. I will say that, while there was clearly something wrong with the data, in the end, the results were the same - essentially, if your primary hope is that three-vote close will prevent lots of close-worthy questions from being answered... and the answer is mostly - it'll prevent some but not the majority.

The chart below needs a bit of explanation - it graphs the number of answered questions closed per month by users - moderators are excluded since this change would only impact user closures - grouped by whether the answer was posted before or after the third vote to close - that is, whether the outcome ("got answered") would have been changed if three-vote closure had been live on the site.

• OutcomeNotChanged_NoModClosures - the upper (yellow) line indicates the number of questions per month that were closed when the first answer came in before the third vote to close. Each month, this was around 250-350 until July 2021, and it's since dropped to around 100-150 per month - this change coincides with your last moderator election.
• OutcomeChanged_NoModClosures - the lower (blue) line indicates the number of questions per month that were closed when the first answer came in after the third vote to close. Each month, this was around 30-40 until July 2021, and it's since dropped to around 20-30.

Looking at recent months, here are the numbers - as with the chart above, this only covers closed questions that have answers, not all closed questions:

Month	Outcome not changed	Outcome changed	Total	% not changed	% changed
2022-01-01	156	41	197	79.2%	20.8%
2022-02-01	123	30	153	80.4%	19.6%
2022-03-01	111	16	127	87.4%	12.6%
2022-04-01	132	31	163	81.0%	19.0%
2022-05-01	133	20	153	86.9%	13.3%

These numbers would have been lower the last time I checked, since it was prior to (or very soon after) your last election. It's not uncommon on medium sized sites like this for an election to have big impact on reviewing since the person who gets elected is frequently an active reviewer - as such, I wanted to check in on the numbers for the balance between community closure and moderator closure.

As I somewhat expected, while users had been closing more without moderator action prior to the election, since, it's pretty much 50-50.

There also seems to be a drop in overall closures, so I wanted to check whether the number of questions completing reviews had dropped.

Fortunately, it seems like the percentage completed hasn't changed or it's improved (it's more likely to be between 80-90% competed than 70-80% completed) - so if there's a reduction in questions closed it's either due to there being fewer nominated for closure initially or there are fewer close-worthy questions asked.

One thing that I do notice is that you have a relatively high percentage of "leave open" reviews, which can indicate disagreement between the first flagger and the reviewers or it can mean that you get a lot of edits that improve questions to the point they're salvageable before getting closed. While this is generally under 10%, this is high compared to many other sites that I've reviewed.

One thing that the moderators also mentioned as a concern is just how full the close review queue is at times. This can seem like a huge load that will be difficult to overcome and I can understand that. It's much nicer to see only a handful of questions in review rather than 100 or more!

For all of these reasons, I've enabled three vote close and reopen as of this posting.

Please use answers here to bring up any concerns or issues you run into. Because there's so much more close activity here, we'll be checking in after about 6 months to make sure things aren't going off the rails but if you're noticing issues, please bring them to my attention before then - our main concern is close-reopen warring.

Answer 1

An observation. In the two weeks before this announcement, we had a relatively constant 100-ish questions in the close queue. Since the change, the close queue has completely emptied; there are currently seven questions in it.

Thinking about your statistics: our close community is small, like fewer than a dozen regular reviewers over the last month, and probably 90% of the votes in the close queue cast by about six people. You mention that we have a lot of “leave open” reviews, but if I look at the advanced review statistics, I say, “oh, it’s those two users, who have an educator’s patience and respond thoughtfully to questions that sometimes are initially dismissed by the community.” Looking at aggregate statistics can be misleading if the data set being aggregated is very small.

Irrespective of the outcome of our three-vote experiment, I think there would be value in coming up with ways to encourage more distinct users to be part of the regular-reviewer population. In the old days we might have “gamified” this by adding a badge. For instance, a user might get a bronze badge for reaching the “the queue has been cleared!” message where they’ve reviewed everything that’s available, and a silver or gold badge if they can get to the end of a queue on some respectable number of different (perhaps consecutive) days. You approve the only pending suggested edit in the queue? Bronze badge. You do that every day for a month? Silver badge.

I haven’t noticed any changes in the available badges for ages, and “gamification” isn’t trendy in the way it used to be. But I have noticed other recent innovations in “hinting” across SE. For instance, if I visit a site where I’m a lower-rep user and I upvote a nice answer, sometimes I get a little modal dialog that says, “hey, you haven’t voted on any questions in a while! also upvote questions!” A message like that to potential queue reviewers might be possible using similar logic. If someone has a habit of casting good close votes but not of visiting the queues, and the length of the queue is “large” compared to the number of frequent reviewers, perhaps casting a close vote could pop up a similar dialog which says, “hey, you haven’t visited the review queues for a while! consider clicking here and reviewing just this one question.”

Aside from my spitballing about getting a more diverse community of reviewers: I’m really pleased that the close queue is manageable again, hooray about that.

Answer 2

I'll add what I have noticed in my own behavior so far.

1. I have started getting more into the close queue more than I had been recently. My activity had dropped off, partially due to the business of life, but also because voting for question closure seemed somewhat futile. Questions that clearly needed to be closed would sit in the queue for a while, and in that time they would be answered anyway. At that point, closure is basically just an indicator of site policy rather than an actual preventative measure and a tool to use to get users to improve the quality of their question (thus improving the quality of PSE).

2. I have been getting into the reopen queue more as well. Knowing that questions can be closed so easily makes me want to check the reopen queue to make sure there aren't questions that were closed too hastily.

3. I am trying to leave more helpful comments on questions in both closure/reopen queues rather than just voting and moving along. Regarding point 1, now that questions are actually being closed before receiving answers, a comment on how to improve the question is much more helpful, as users whose questions are quickly closed will now have more of a reason to actually improve their posts rather than just get their answer and move on.

All of this to say, so far, at the very least, I think this change has the potential to make the site much better. More users leaving helpful comments guiding users on how to reopen their question, as well as more users being active in the queues themselves (as Rob points out), should give the 3-vote closure the best chance at being an improvement and not a detriment.