15
$\begingroup$

After discussion about this was started two and a half years ago, I'm here to share that I'm enabling a test of three-vote close on Physics today. Firstly, I'd like to thank you for your patience. There's been a lot of discussion about this and I mentioned my primary concerns about making this change here the last time I posted, about six months ago.

At the time, I wasn't certain that my data was quite correct and I've gotten a chance to review it. I will say that, while there was clearly something wrong with the data, in the end, the results were the same - essentially, if your primary hope is that three-vote close will prevent lots of close-worthy questions from being answered... and the answer is mostly - it'll prevent some but not the majority.

The chart below needs a bit of explanation - it graphs the number of answered questions closed per month by users - moderators are excluded since this change would only impact user closures - grouped by whether the answer was posted before or after the third vote to close - that is, whether the outcome ("got answered") would have been changed if three-vote closure had been live on the site.

  • OutcomeNotChanged_NoModClosures - the upper (yellow) line indicates the number of questions per month that were closed when the first answer came in before the third vote to close. Each month, this was around 250-350 until July 2021, and it's since dropped to around 100-150 per month - this change coincides with your last moderator election.
  • OutcomeChanged_NoModClosures - the lower (blue) line indicates the number of questions per month that were closed when the first answer came in after the third vote to close. Each month, this was around 30-40 until July 2021, and it's since dropped to around 20-30.

Graph as described in the paragraph and bullets above.

Looking at recent months, here are the numbers - as with the chart above, this only covers closed questions that have answers, not all closed questions:

Month Outcome not changed Outcome changed Total % not changed % changed
2022-01-01 156 41 197 79.2% 20.8%
2022-02-01 123 30 153 80.4% 19.6%
2022-03-01 111 16 127 87.4% 12.6%
2022-04-01 132 31 163 81.0% 19.0%
2022-05-01 133 20 153 86.9% 13.3%

These numbers would have been lower the last time I checked, since it was prior to (or very soon after) your last election. It's not uncommon on medium sized sites like this for an election to have big impact on reviewing since the person who gets elected is frequently an active reviewer - as such, I wanted to check in on the numbers for the balance between community closure and moderator closure.

As I somewhat expected, while users had been closing more without moderator action prior to the election, since, it's pretty much 50-50.

Graph of questions closed and reopened per month with lines for "all closed/reopened" "moderator closed/reopened", and "community closed/reopened". Prior to August 2021, users did more of the closures than moderators. After that point, they did about equal amounts.

There also seems to be a drop in overall closures, so I wanted to check whether the number of questions completing reviews had dropped.

Graph of questions being reviewed per month based on their outcome - close, leave open or nothing. The percentage of completed reviews (closed and leave open combined) is relatively flat, between 70% and 90% with a slight increase over time.

Fortunately, it seems like the percentage completed hasn't changed or it's improved (it's more likely to be between 80-90% competed than 70-80% completed) - so if there's a reduction in questions closed it's either due to there being fewer nominated for closure initially or there are fewer close-worthy questions asked.

One thing that I do notice is that you have a relatively high percentage of "leave open" reviews, which can indicate disagreement between the first flagger and the reviewers or it can mean that you get a lot of edits that improve questions to the point they're salvageable before getting closed. While this is generally under 10%, this is high compared to many other sites that I've reviewed.

One thing that the moderators also mentioned as a concern is just how full the close review queue is at times. This can seem like a huge load that will be difficult to overcome and I can understand that. It's much nicer to see only a handful of questions in review rather than 100 or more!

For all of these reasons, I've enabled three vote close and reopen as of this posting.

Please use answers here to bring up any concerns or issues you run into. Because there's so much more close activity here, we'll be checking in after about 6 months to make sure things aren't going off the rails but if you're noticing issues, please bring them to my attention before then - our main concern is close-reopen warring.

$\endgroup$
11
  • 1
    $\begingroup$ Does that mean that now, 3 votes are enough to migrate a question away from physics, while still 5 votes are required to migrate it back? -- I always felt that migration is a very asymmetric and thus partially unfair situation (in essense, if a few people here don't like some type of questions, they can get it migrated away, as long as it also has some other home -- even if the majority of people here think that it is also very suitable here; and after the migration, the possible majority who thinks it fits here here has no say on that any more). -- However, now this seems much worse, ... $\endgroup$ Jul 9, 2022 at 9:14
  • $\begingroup$ ... because 3 people can migrate it away, while it takes 5 people from the other community to get it back. $\endgroup$ Jul 9, 2022 at 9:14
  • $\begingroup$ Concrete case: math.stackexchange.com/questions/4489051/…, where the most relevant tag [quantum-information] has about 3000 questions on physics and <200 on mathematics. $\endgroup$ Jul 9, 2022 at 9:15
  • $\begingroup$ @NorbertSchuch Ideally people won't be migrating (or closing) questions because they don't belong on the site, not because they simply don't like the question. $\endgroup$ Jul 9, 2022 at 11:02
  • $\begingroup$ Well, the linked question is clearly about quantum information, and quantum information (judging by the number of tagged questions) certainly has a much better home here than on math. In addition, there were some comments (now deleted, I guess) from one of the close voters indicating a dislike of the mathematical flavor of the question. So my feeling is: Yes, there are people who think that certain mathematical/formal questions in physics physics don't belong here, and who would vote to migrate because of that, while this is - as far as I can tell - not at all the consensus here. $\endgroup$ Jul 9, 2022 at 15:42
  • $\begingroup$ @NorbertSchuch Note that Math is the only migration path with this issue, because migration to other sites requires moderator intervention. If you feel strongly that this particular question is a better fit at Physics (or even at Quantum Computing), I suggest you flag it for attention from a Math moderator. If we get lots of such flags, that is useful information for deciding whether our three-vote close threshold becomes permanent. $\endgroup$
    – rob Mod
    Jul 9, 2022 at 16:40
  • $\begingroup$ @rob Indeed, the pre-defined migration paths suck in that regard ... (not that I don't understand the logic, but it also means that people here have no recourse). Personally I am very happy to have QI questions here (mostly as the level of question here is higher than at qc.se due to different site policies, and the problem that this cannot be filtered due to the meta tag policy, and I do feel they fit perfectly for this site. $\endgroup$ Jul 9, 2022 at 17:00
  • $\begingroup$ (Though this might be an old problem: I remember Joe Fitzsimons' motivation to start Theoretical Physics SE was partly driven by the fact that back then, he felt that all other communities (Physics, TC/CS, Math) felt the topic did not belong there. But Physics.SE was different at that time, I believe.) $\endgroup$ Jul 9, 2022 at 17:01
  • $\begingroup$ @NorbertSchuch But you do have a recourse: ping a moderator. The reason we’re here is because the software can’t (and shouldn’t) do everything automatically. $\endgroup$
    – rob Mod
    Jul 9, 2022 at 17:30
  • $\begingroup$ @rob Before or after migration? $\endgroup$ Jul 9, 2022 at 18:52
  • 1
    $\begingroup$ Oh. It worked :) (Well, it is closed as off-topic -- but this makes lots of sense, because this way the community here can vote about it.) $\endgroup$ Jul 9, 2022 at 18:54

2 Answers 2

6
$\begingroup$

An observation. In the two weeks before this announcement, we had a relatively constant 100-ish questions in the close queue. Since the change, the close queue has completely emptied; there are currently seven questions in it.

Thinking about your statistics: our close community is small, like fewer than a dozen regular reviewers over the last month, and probably 90% of the votes in the close queue cast by about six people. You mention that we have a lot of “leave open” reviews, but if I look at the advanced review statistics, I say, “oh, it’s those two users, who have an educator’s patience and respond thoughtfully to questions that sometimes are initially dismissed by the community.” Looking at aggregate statistics can be misleading if the data set being aggregated is very small.

Irrespective of the outcome of our three-vote experiment, I think there would be value in coming up with ways to encourage more distinct users to be part of the regular-reviewer population. In the old days we might have “gamified” this by adding a badge. For instance, a user might get a bronze badge for reaching the “the queue has been cleared!” message where they’ve reviewed everything that’s available, and a silver or gold badge if they can get to the end of a queue on some respectable number of different (perhaps consecutive) days. You approve the only pending suggested edit in the queue? Bronze badge. You do that every day for a month? Silver badge.

I haven’t noticed any changes in the available badges for ages, and “gamification” isn’t trendy in the way it used to be. But I have noticed other recent innovations in “hinting” across SE. For instance, if I visit a site where I’m a lower-rep user and I upvote a nice answer, sometimes I get a little modal dialog that says, “hey, you haven’t voted on any questions in a while! also upvote questions!” A message like that to potential queue reviewers might be possible using similar logic. If someone has a habit of casting good close votes but not of visiting the queues, and the length of the queue is “large” compared to the number of frequent reviewers, perhaps casting a close vote could pop up a similar dialog which says, “hey, you haven’t visited the review queues for a while! consider clicking here and reviewing just this one question.”

Aside from my spitballing about getting a more diverse community of reviewers: I’m really pleased that the close queue is manageable again, hooray about that.

$\endgroup$
3
  • 3
    $\begingroup$ AFAI’m concerned, I must admit this is the first time in a looooong while that I do not feel overwhelmed by the number of posts to review for one reason or another. $\endgroup$ Jul 18, 2022 at 1:31
  • $\begingroup$ I really like the "gamification" concept as I'm forced to admit I quite like "hunting badges". I'd love to hear others opinion's on this, but I also think reputation incentives could be interesting in these sorts of "bureaucratic work" on the site. For example, I've noticed that some (many, tbf) tags have poor descriptions, and I can't help but think that a lot of users don't see the point of writing detailed descriptions with zero rewards in exchange (or perhaps +2 rep, which I think is what the edit yields). $\endgroup$ Jul 27, 2022 at 2:04
  • $\begingroup$ @rob I am one of the "it’s those two users, who have an educator’s patience " , but is not the only reason I answer questions that are, or would be, candidates for closure. It is the rule that "comments must not be used as answers" and they may be deleted. Just did that in my answer here physics.stackexchange.com/questions/720812/… $\endgroup$
    – anna v
    Jul 31, 2022 at 4:32
5
$\begingroup$

I'll add what I have noticed in my own behavior so far.

  1. I have started getting more into the close queue more than I had been recently. My activity had dropped off, partially due to the business of life, but also because voting for question closure seemed somewhat futile. Questions that clearly needed to be closed would sit in the queue for a while, and in that time they would be answered anyway. At that point, closure is basically just an indicator of site policy rather than an actual preventative measure and a tool to use to get users to improve the quality of their question (thus improving the quality of PSE).

  2. I have been getting into the reopen queue more as well. Knowing that questions can be closed so easily makes me want to check the reopen queue to make sure there aren't questions that were closed too hastily.

  3. I am trying to leave more helpful comments on questions in both closure/reopen queues rather than just voting and moving along. Regarding point 1, now that questions are actually being closed before receiving answers, a comment on how to improve the question is much more helpful, as users whose questions are quickly closed will now have more of a reason to actually improve their posts rather than just get their answer and move on.

All of this to say, so far, at the very least, I think this change has the potential to make the site much better. More users leaving helpful comments guiding users on how to reopen their question, as well as more users being active in the queues themselves (as Rob points out), should give the 3-vote closure the best chance at being an improvement and not a detriment.

$\endgroup$
2
  • 3
    $\begingroup$ I agree - previously going into the Close queue was like cleaning out the Augean stables, now it is more like a light dusting. $\endgroup$
    – Jon Custer
    Jul 20, 2022 at 14:38
  • 3
    $\begingroup$ In undergrad I learned a little about “queuing theory.” I only remember (a) assume arrivals are Poisson-distributed, and (b) just adding one cashier can clear out the line at the grocery checkout. $\endgroup$
    – rob Mod
    Jul 20, 2022 at 20:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .