Should we test lowering the vote to close and reopen threshold?

Update

Based on the votes, we will be giving this experiment a try. We are working with the CM team now to figure out what metrics we can measure for comparisons before and during the test and we will update everybody as soon as we come up with a plan and a date to start the test.

Stay tuned!

In early August, Stack Overflow (the company) announced an experiment to lower the number of votes required to close or to reopen a question to 3 votes on Stack Overflow (the website). The trial period was 30 days, and at the end (plus time for data crunching), the results and an in-depth analysis of the experiment were posted. It was a resounding success on Stack Overflow (the website).

Shortly over a week ago, Stack Overflow (the company) announced the change to vote counts would become permanent on Stack Overflow (the website).

The Community Manager team has indicated that other sites are welcome to try out a 30 day test of the same thing if the community agrees. Several smaller sites and beta sites have started the process to undergo the test, motivated by the limited number of active curators on those sites.

Physics.SE is not a small site, but we're also not the size of Stack Overflow. We have a dedicated core group of curators (thank you!!!), but the group isn't big enough to get quick and effective action on closing or reopening questions that warrant it. A completely not scientific, observation-biased look at the close and reopen queues indicates many questions quickly get 3 or 4 close or reopen votes and then languish.

We would like to propose that Physics.SE undergo a 30 day trial of the lowered close and reopen vote counts, so it will only take 3 votes to close or reopen a question. At the end of the 30 days, we can evaluate how the test went and help Stack Overflow (the company) decide if it works on sites of our scale.

I'll leave the discussion here open for 3 weeks (we'll take a final look at the status on 3 January 2020) and if there is enough of a consensus around evaluating it, we can let the Community Management team know and they can enable the new vote count requirements for 30 days.

Is the community interested in experimenting with a 3 vote requirement for opening and closing questions?

• I'm guessing only up votes on the answers? – BioPhysicist Dec 13 '19 at 3:02
• @AaronStevens Let's use votes on the answers to help establish consensus within the discussion. – rob Dec 13 '19 at 3:38
• @rob Yes, I don't think that was ever in question :) I suppose each user gets two possible votes here then (one up and one down vote). – BioPhysicist Dec 13 '19 at 5:37
• @AaronStevens I'm not so worried about it -- we can see the +/- afterall, and we can decide if we want to use other sites' contest rules (only + counts) or something else. Ideally, it's overwhelmingly clear which one the community wants and we don't need to worry about it. – tpg2114 Dec 13 '19 at 14:30
• What would be the metrics used in this study? What criteria would be used to decide whether it was a success or not? – Emilio Pisanty Dec 13 '19 at 20:43
• @EmilioPisanty We'd have to work with the CM's to figure out how much analysis they will do, but I am particularly interested in the efficacy numbers Shog computed for the Stack Overflow test (the table appears about halfway through). I'm not sure if we can get additional metrics, but it would be nice to see how many questions that end up closed get an answer before closure when it takes 5 votes vs 3 votes, since that's a big issue with HW questions. – tpg2114 Dec 13 '19 at 22:11
• They also have measures of close/reopen wars, since this would make it easier to reopen questions also. So we can see if the community frequently disagrees on closures -- an interesting metric would be how many 5 votes to close get 3 votes to reopen now, because that would trigger a reopen during the test. And of course, the queue backlog should also go down considerably as well during the test period if it is more effective. – tpg2114 Dec 13 '19 at 22:14
• So is this status-complete or not? I mean there is significant difference in the support of and against the question. – user249968 Jan 2 '20 at 2:55
• @Kyubey As stated in the question, we will wait until 3 January 2020. Then if the consensus is to try it, we will reach out to the CM team. I'll update the community as those things progress. – tpg2114 Jan 2 '20 at 2:58
• @tpg2114 I don't see any real harm in the experiment, but I'm not convinced at all by the test suite that you've linked to (which I only managed to get to recently). The problems with closure on SO are very different to anything we have here. The focus of the post you linked to is the "effectiveness" of closure votes, i.e., on whether close-votes age away before being acted on (closure / Leave Open) by the queue or otherwise. Is that really a concern here? Do any close-votes get to age away on this site, before the question makes it out of the queue? That'd be deeply surprising to me. – Emilio Pisanty Jan 4 '20 at 2:53
• I do see a different set of issues, though, particularly in the speed of closure and whether it is able to prevent answers getting posted before the closure, especially for homework posts. But that needs a dedicated test suite, and I'm not sure who can build it and whether it can be performed using public SEDE data. (My gut says it probably won't be, as it requires non-public vote timing information.) – Emilio Pisanty Jan 4 '20 at 2:56
• @EmilioPisanty Just in the past hour or so, I've been talking with Shog about what metrics we can get to assess these things -- he has given me data on "answered before closure" as a function of number of votes when closed. We also have current efficacy measurements for each closure reason (at a broad level -- all off-topic are lumped, for example). I've asked for the possibility of some extra information like time spent at each vote count, not sure if we can get that yet or not. If you have other suggestions, let me know and we can see what we can get. – tpg2114 Jan 4 '20 at 3:13
• Before we launch the test, I'd like to have an idea of what metrics we can actually quantify and how we will use them, rather than trying to p-hack our way through data at the end. So I plan on posting a hypothesis/experimental approach post prior to starting the test -- although, I also don't want to influence people's behavior during the test, so I'm not sure the best way to approach that yet... – tpg2114 Jan 4 '20 at 3:13
• @tpg2114 Thanks for the update. I agree with both the p-hack and the blinding concerns. One possible solution is to agree on both overall goals and a detailed protocol with Shog9 (and hopefully also with SE's data scientists), and then post the former but seal the latter (e.g. by posting here and immediately deleting it). Very few of us are proficient enough at statistics that we'd be able to meaningfully engage with the detailed protocol, I think, and I for one would be better reassured that there's a professional statistician handling the implementation of the goals than having them public. – Emilio Pisanty Jan 4 '20 at 3:23
• @ZeroTheHero Given the downsizing of the CM staff recently, this is on hold at the moment. It also means we may not be able to get customized metrics that we were discussing with one of the departed staff members. The CM who is leading the project is aware we want to test it, and many other sites have also volunteered, but we don't know when it will happen. – tpg2114 Jan 25 '20 at 17:19

YES

Physics.SE should undergo a 30 day test with 3 votes required to close and reopen questions.

• I'm interested in why this answer got such a high score so fast. Do we have a population of users who think that the current closure system is broken? If so, how, and what are its main perceived problems? – Emilio Pisanty Dec 13 '19 at 20:46
• @EmilioPisanty the fact that we can't close OT "Do My HW For Me" fast enough is one. As TPG points out in the post (in possibly biased observations), many times posts just hover at 3 votes for a while before being closed, meanwhile OP gets some idiot to so their HW and we get the credit of being a HW help site. – Kyle Kanos Dec 13 '19 at 21:55
• I also am not sure that is categorize the want for faster/easier close/reopens as thinking the system is broken. Honestly, its just not optimal with the same 4-6 users closing questions. Maybe if the community weren't so lazy, 5 would work. But sadly it seems the majority of capable users have zero interest in reviewing – Kyle Kanos Dec 13 '19 at 21:58
• @EmilioPisanty I up voted, not because I think the current system is horrible, but I still think the test could be interesting/useful. If it's better then let's change it. If not, that's fine too. – BioPhysicist Dec 13 '19 at 22:52
• Given the number of highly suspicious questions of late speeding the closure process can only be beneficial. The OP can always clarify through edits. – ZeroTheHero Dec 13 '19 at 23:32
• I tend to agree the problem is lack of motivation (and I do VTC myself). If I see the title of "yet another do my HW question" there's no value to me of voting to close it, compared with just ignoring it. And whatever others might think, I don't personally count "making Physics.SE a more awesome Q&A site" (or even "getting more rep") any motivation at all. – alephzero Dec 14 '19 at 1:26
• .. To be accurate, I did used to VTC, until the script kiddies who maintain SE recently decided unilaterally to screw up support for my default browser, which means voting and commenting is now even more of a pain. Human nature 101: if you don't make it easy for people to do the right things, they won't do them. – alephzero Dec 14 '19 at 1:29
• @alephzero What browser? What changes make voting and commenting harder? – Kyle Kanos Dec 14 '19 at 12:16

This should be an easy answer.

Yes.

We should trial this. Stack Overflow tried it and were really happy with the results. And that's just what it is, a "trial". Rather than speculating on what the ups or downs of this could be, we have the opportunity to experimentally determine the true situation. If it doesn't work well, we don't adopt it permanently. If it works, we accept it. There's very little risk.

Aren't we supposed to be scientists? Why are we standing around talking about what could happen when we can collect real data and test the theory? I don't recall a lot of instances in science when they published a paper saying "there's a 50/50 chance that this theory would work out, so there's no point in anybody running the easy experiment to check the predictions. We'll just assume it's wrong".

Save any arguing for the decision on whether to adopt it permanently and just run the trial now. Isn't that why we do experiments after all? Because it's a lot easier to make a definitive case about something when you've got experimental data backing you up. So whatever side you're on, you want to trial it to prove to the other side that you're right.

• I agree. This was what I was trying to say on my first comment on @akhmeteli's answer – BioPhysicist Dec 17 '19 at 21:44
• OK, then let's do the science. Properly. With well-defined metrics and a well-defined control, and agreed-upon criteria for success and failure. Only then does running the experiment mean that anybody can be 'proved right' by the data. (For full disclosure: I'm guilty of having proposing uncontrolled trials here previously, and they were much less usable because of that.) – Emilio Pisanty Dec 18 '19 at 8:09
• @EmilioPisanty Love you. <3 Love all science – Jim Dec 18 '19 at 12:51

NO

Physics.SE should not undergo a 30 day test with 3 votes required to close and reopen questions.

Another method is The Silver Hammer$$^™$$.

We've had this trouble previously: A guide to moderating Physics Stack Exchange yourself: close voting. Currently Gold Tag has a Dupe Hammer, extending that to silver adds a whole bunch of people. Not just anyone but people whom are knowledgeable about the subject.

There is no reason why this couldn't be combined with a lowering to 3 or 4.

Previous Silver Hammer discussions:

• This really would only work if the questions closed were strictly duplicates, which isn't always the case. Duplicates account for about 17% of all closures (cf. 10k rep mod tools, which only works for those with >10k rep); the HW reason, meanwhile, accounts for about 40%. – Kyle Kanos Dec 21 '19 at 12:03
• @KyleKanos - Your idea (interpretation) was already discussed in The Tavern. The non sequitur is more a derailment than a helpful use of comments. – Rob Dec 21 '19 at 12:34
• How is pointing out that lowering the requirements for dupe-hammer would only impact duplicates a non sequitur?! – Kyle Kanos Dec 21 '19 at 12:51
• Or are you trying to say that my use of statistics to prove my point the non sequitur? Because those definitely help the point that most questions in the close queue are not proposed duplicates so allowing for silver dupe-hammer powers won't help the more than 80% of the cases we see regularly. – Kyle Kanos Dec 21 '19 at 16:40
• I don't see how this helps. SE is obviously uninterested in implementing any kind of "silver hammer" at this stage. Bringing it up now is just a waste of discussion energy -- which makes it all the more rich that you're criticising others for what you claim is precisely that (even though Kyle's comments were no such thing). – Emilio Pisanty Dec 24 '19 at 2:47

I don't think we should test closing with fewer votes. The outcome for thousands of users of Physics SE would be too sensitive to actions of an individual trigger-happy user. If somebody hates a question, (s)he can also flag it. If a moderator does not close the question after that, maybe the question should not be closed, in the first place.

• To be clear -- the test won't enable an individual to do anything they could or couldn't do before, so we won't see people able to single-handedly close questions unless they have a gold badge in that tag, which is how it works now. And the moderators don't typically act of flags asking for closure while the question is still in the review queue -- we prefer to let the community take action as much as possible. Not sure if that helps ease some of your concerns or not... – tpg2114 Dec 14 '19 at 16:02
• But the proposed test isn't one vote required to close. And there is a daily limit to how many close votes a user can use. Plus, you are forgetting about reopening questions. Your answer is based on speculation, which is what a test would be great for. If what you fear is actually true, then I'm sure it wouldn't be applied. I think your answer actually shows why a test would be useful. – BioPhysicist Dec 14 '19 at 16:05
• Also if you can vote to close a question then you can't also raise a close question flag – BioPhysicist Dec 14 '19 at 16:06
• @tpg2114 : I did not say that an individual user would be able to close a question if fewer votes are enough, I said the outcome would be more sensitive to actions of such a user. If elected moderators are reluctant to close flagged questions in a hurry, maybe we, thousands of other users, should also manifest such wisdom and be reluctant to have questions closed with just 3 votes:-) – akhmeteli Dec 14 '19 at 16:39
• @AaronStevens : And who will assess the results of the test and decide if this measure should be permanent? I am afraid that will be the same trigger-happy enthusiasts of express closing. – akhmeteli Dec 14 '19 at 16:47
• I'm sure @tpg2114 Can answer that better, but I would assume a meta post would be made after the trial showing the acquired data and allowing a final vote to be made. I don't understand where your fear of single users having all the power here is coming from. Really the only people with that power are mods, and they use their power very responsibly. – BioPhysicist Dec 14 '19 at 17:03
• @AaronStevens : Please look how it was done at Stack Overflow (see the link to the results and an in-depth analysis in the above question). Before the experiment, the initial vote to close lead to closing in 36% of the cases; during the experiment, the initial vote to close lead to closing in 55% of the cases. In the link, they call it "a resounding success". I call it dangerous sensitivity to an opinion of just one trigger-happy user. – akhmeteli Dec 14 '19 at 17:20
• If you can prove that one user caused the increase in percentage then I'll believe you – BioPhysicist Dec 14 '19 at 17:24
• @AaronStevens : I was not trying to say that it was one and the same user in all cases, I was saying the first vote to close became too influential. So for each closed question, it was just one user (not the same user for all questions) that pretty much determined the fate of the question. – akhmeteli Dec 14 '19 at 17:27
• So do you think that there are many questions currently posted that have 3-4 votes to close that have come from said trigger happy users? I think the fact that the close vote queue is usually fairly full shows there aren't many users like this. – BioPhysicist Dec 14 '19 at 17:31
• @AaronStevens : I cannot offer my own statistics, but, judging by the statistics from the link, most questions getting the first vote to close will be closed under the new rule. I think this is too much, you may think this is actually great:-) – akhmeteli Dec 14 '19 at 17:40
• @akhmeteli While close efficacy went up, so did reopen efficacy -- remember, the test will also change reopen requirements to 3 votes. So for questions that are wrongly closed, it should be much easier to undo that and reopen the question. Of course, we won't know how the balance changes until we test it, and that's why we do a 30 day test and evaluate before diving in full-time. – tpg2114 Dec 14 '19 at 17:44
• It seems to me that what you are suggesting - refer bad questions to a mod who can close it - is in fact worse than having a multivote system. – ZeroTheHero Dec 15 '19 at 18:29
• Well... We have to agree to disagree. I would much rather a question NOT be closed by a moderator. IMO there are not enough users consistently voting (or not) to close - witness the size of the VTC queue - to keep the threshold at 5. Ideally of the queue was systematically pared down, 5 would be better than 3 but alas this is just not happening. – ZeroTheHero Dec 15 '19 at 21:44
• First reopen votes are often edits to the question - that puts the question in review. If closed questions don't get edited, they're unlikely to get reopened. But, y'all actually have a decent track record of handling the reopen queue... during a 60 day period starting 90 days ago, 316 closed questions were edited, putting them in review, and 80% of those reviews were completed (9% reopened, 72% leave closed, 20% aged out). – Catija Dec 19 '19 at 23:12