8
$\begingroup$

This question was stimulated by comments in the question Can I ask to have a block or left alone by a user?.

@user104372 commented in the aforementioned question: '.. opening downvote caused a cascade of unjustified downvotes...'. I believe s/he makes a good point here and the same is true of close votes.

I have asked more questions then most on this site1 and have always had the feeling that the way closed votes are used on this site is unfair.

When a high rep user (>2999 reps) reads a question I believe they should be making an independent decision, without preconceptions, about the quality of question2. And should only vote to close if and only if they would have done so without the knowledge of the actions of others.

With the current system this is hard to achieve since the high users are told the whether or not others have voted to close the question3 (e.g. close (3) and via the review queues). To me it feels that one close vote almost always leads to the eventual closer of a question.

On the contrary when it comes to 'reopen votes' it appears such 'a cascade' rarely happens.

What are people's views on this? Is there any statistical evidence either way?

1. http://data.stackexchange.com/physics/query/71706/who-asked-the-most-questions

2. I will be the first to admit that some a lot of my questions are poor and do/did deserve closing.

3. https://physics.stackexchange.com/help/privileges/close-questions

$\endgroup$
8
  • 2
    $\begingroup$ Related posts on the mother meta: meta.stackexchange.com/questions/138976/… and meta.stackexchange.com/questions/187800/… $\endgroup$
    – dmckee --- ex-moderator kitten Mod
    Commented Sep 26, 2016 at 19:17
  • 11
    $\begingroup$ How would one distinguish "Most questions with a close vote get closed because people are influenced by the first close vote" and "Most questions with a close vote get closed because they should be closed and the voters would have decided so without knowledge of other votes" in the statistical data? To me it seems as if both scenarios would leave exactly the same trail of evidence - most questions with an initial close vote get closed. $\endgroup$
    – ACuriousMind Mod
    Commented Sep 26, 2016 at 19:17
  • 2
    $\begingroup$ @ACuriousMind I am not a statistician but something like a goodness of fit test with a possion distribution may work. The possion distribution assumes the independence of events an assumption that would hold in the second case but not the first. $\endgroup$
    – Quantum spaghettification
    Commented Sep 27, 2016 at 6:10
  • 5
    $\begingroup$ The problem I see is that close votes can't be assumed to be random to begin with. I have no idea what a "good" proportion of questions voted to close to questions actually closed would be, but I'm pretty sure it should be higher than 50%, since I assume that people have reasons to cast that first close vote. Finding that the distribution of closed/left open questions looks as if the reviewers behave randomly would be a very puzzling result. $\endgroup$
    – ACuriousMind Mod
    Commented Sep 27, 2016 at 12:01
  • 2
    $\begingroup$ @ACuriousMind A agree with everything you said in your last comment. But for a given question the number of close votes for a given number of views should be a random variable following a poission distribution with a mean that depends on the specific question. $\endgroup$
    – Quantum spaghettification
    Commented Sep 28, 2016 at 16:22
  • 3
    $\begingroup$ For what it's worth, I do believe there are some users that are much more likely to be the 4th or 5th close vote on a question than the 1st. It's easier to follow others, since at that point, you can't be called out as 'wrong'. $\endgroup$
    – knzhou
    Commented Sep 28, 2016 at 16:35
  • 2
    $\begingroup$ For what it's worth, I called this out once before; saying that it may help to make everyone more impartial if they couldn't immediately see the number and reasons of close votes. However, after a small amount of debate on the validity of my claim, it was quickly pointed out to me that this change would have to be network-wide, which we can't do anything about here. I can't recall if a mother mod addressed my issue or not, so it could be worth asking this on mother meta, but I'm willing to bet the mother mods can give an excellent reason why it isn't implemented $\endgroup$
    – Jim
    Commented Sep 29, 2016 at 12:14
  • $\begingroup$ @Jim A search on Meta Stack Exchange should point the way to any previous suggestions along those lines. $\endgroup$
    – David Z Mod
    Commented Sep 30, 2016 at 5:43

1 Answer 1

Reset to default
3
$\begingroup$

At the risk of you going off on a rail and misinterpreting it, here is the data:

(Most of the data you might want, going beyond this, is also public, in the SEDE. Given that you already know it exist, I don't think it's appropriate to raise this sort of allegation without at least some evidence of having tried to dig into the data.)

The data shows that a strong majority of close-vote reviews end up with question closure. Most reopen reviews end up with a leave-closed outcome.

Here it is important to note that this sort of dumb counting, without actually looking at the posts and whether they warranted closure/reopening or not. Since you do not have access to the queues it is understandable that you do not have a gauge on the typical content there, but the links above give plenty of information of what actually makes it to the queue. In particular, the second conclusion - few questions are reopened in the queue - bears a close match to the experience of reviewing that queue: for example, a significant fraction of questions on the queue are minor edits that barely begin addressing the problems with the post. The fraction that even approaches reopen-ability is rather low.

There is, moreover, one crucial thing missing from the public data: you cannot know the frame of mind of the people who reviewed posts because you cannot know the frame of mind of people, period, and it is not appropriate to raise accusations that could only be proven if you had psychic powers. You stop short of making the explicit accusation that reviewers are not "making an independent decision, without preconceptions", but the implication is pretty clear. This accusation is of this latter kind: you could only prove it by reading minds.

In particular, and to double down on a previous comment, I don't think that any amount of public data will support your hypothesis. In particular, consider the following possible explanations for the data:

  • Reviewers are by and large a fickle bunch and they tend to just close-vote anything in the review queue because it's there

versus

  • Close-vote reviewers and 3k+ users at large have a generally robust consensus of the type of question that is on- and off-topic for the site, which naturally results in consistent reviews, and tends to put only close-worthy questions in the queue to begin with.

(These are, of course, not the only possible explanations.) Both scenarios explain the data, and neither can really be disproved without public data. The lesson I infer from that is that the question was flawed to begin with, but then I suspect I have about a snowball's chance in the door to hell at convincing you of that.

On the other hand, I do want to stress that raising this sort of accusation without data, where you know the data is there, is pretty inflammatory and unconstructive.

$\endgroup$
13
  • $\begingroup$ It's possible in principle to detect 'nonindependent' close voting. To do this, you can calculate the chance of a close vote as a function of the number of close votes already there, for each user. Of course, there should be a positive correlation even if everybody is impartial, since bad questions get more close votes, so you can control by comparing with a known impartial user, like ACM. $\endgroup$
    – knzhou
    Commented Sep 29, 2016 at 2:02
  • 1
    $\begingroup$ Honestly, I'm quite sure this happens. I've definitely seen several QFT questions voted to closure, where the process was (correctly) started by ACM, but finished by several users that don't know QFT. The result seems fine most of the time, but it feels wrong to me. $\endgroup$
    – knzhou
    Commented Sep 29, 2016 at 2:03
  • $\begingroup$ @knzhou There probably is a fair bit that someone with sizable statistics chops can say in that regard. (On the other hand, even if the individual reviews are in principle public, that data will be hard to mine and pretty messy (e.g. many reviews are resolved out of the queue), and it won't include the number of out-of-queue close votes at time of review.) However, if one goes in with one's mind made up that the effect exists, then we know how that goes. $\endgroup$
    – Emilio Pisanty
    Commented Sep 29, 2016 at 2:18
  • 3
    $\begingroup$ @EmilioPisanty Just to point out that I have never accused anyone of anything, and think high rep uses do a great job. I have simply made an observation, as others have before me and thought it should be opened for discussion. My question is about weather it happens or not. I am under the opinion it does and am simply interested in others opinions. Also I am very willing to change my mind about this situation, why would I ask a question asking for others views on it and evidence either way if I wasn't? $\endgroup$
    – Quantum spaghettification
    Commented Sep 29, 2016 at 4:08
  • 2
    $\begingroup$ Emilio, your search is of no [great] help: knowing the percentage of question closed says nothing about the issue raised here and in other election posts/comments, and many readers don't even know what is SEDE. Also, it is not true the reviewers have a wide consensus.... etc. There have been few habitual reviewers: only 20 listed here](physics.stackexchange.com/review/close/stats) and if you run a more useful search, I bet you'll be surprised on the amazing differences. There is 1) the steward-badge hunter, random voting all items in the queue in a split second, 2)... $\endgroup$
    – user104372
    Commented Sep 29, 2016 at 6:31
  • 2
    $\begingroup$ ...2) the record hunter (how can 10 001 be random?) 3) the one who always VTC because he is strict or 4) to be onthe safe side 5) the one who mostly votes leaveopen because he is lenient or 6) to make it up...and lastly 7) the conscientious reviewers who actually reads and tries to understand OP. I imagine you agree that certainly JohnRennie is competent, balanced and trustworthy, what would help is to know his own ratio of close/leave open votes, and then compare the other rewievers's ratio. That is an intelligent search . Too strict-too fast, VTC - happy= not moderate= no good mod! $\endgroup$
    – user104372
    Commented Sep 29, 2016 at 6:40
  • 4
    $\begingroup$ @Quantumspaghettification, if you don't get data from Emilio, you/anybody can easily acquire vital info: 1) go to any profile and click "all actions-reviews" 2) then click on any "leave closed"/ "close" . So you can get a clear idea of one/more reviewers at a time. Also, if you click on all the reviews in a row(3-days old or more), you can find out the time spent on each item (6/min= about 7 sec/item) $\endgroup$
    – user104372
    Commented Sep 29, 2016 at 7:22
  • 2
    $\begingroup$ @EmilioPisanty to me the post read more like "there might be subconscious bias". In the UK, we wrote exams without our names but with a student number exactly for that reason. For jobs nowadays, applications are often without picture or name. Questioning whether that would change something does not consist questioning the great job that is done on review. But maybe I've interpreted the post with too much good will ... $\endgroup$
    – Sanya
    Commented Sep 29, 2016 at 9:18
  • $\begingroup$ @Sanya The problem is that the OP does not pose any questions that have verifiable answers. $\endgroup$
    – Emilio Pisanty
    Commented Sep 29, 2016 at 10:36
  • $\begingroup$ As for @user104372, if you want me to respond you need to leave the vitriol at the door. $\endgroup$
    – Emilio Pisanty
    Commented Sep 29, 2016 at 10:37
  • 1
    $\begingroup$ Emilio, I did not ask nor expect nor wish you to respond to me. I showed [you] the shortcomings of your searches/post, if you were trying to help OP., and gave him useful advice. If you consider vitriol any grounded, documented criticism, that is only your problem! :) In case you missed it, OP made a verifiable educated guess (here verified), that once a question is on the close queue, most likely it'll be closed, secondly , he made a sensible suggestion that reviewers should vote not-knowing-who-voted-how. Btw, I checked a few pages of your reviews: you are a very balanced reviewer! $\endgroup$
    – user104372
    Commented Sep 29, 2016 at 11:22
  • 1
    $\begingroup$ @EmilioPisanty you said 'the OP does not pose any questions that have verifiable answers.' if this is so, I was not aware of it when I asked the question. A suitable answer would therefore be one where the reasons behind this are explained. I do also feel you have slightly misjudged the tone my question was intended to be written (probably my fault), you seem to think I have jumped to conclusions and am 'accusing people', on the contrary. Yes I have made an observation, but I have never said that observation is right, just that it is worth testing - which is what scientists do. $\endgroup$
    – Quantum spaghettification
    Commented Sep 29, 2016 at 11:33
  • 1
    $\begingroup$ @EmilioPisanty I think this answer could stand to be edited to focus more on the data and less on the way the original question was asked, but it's a good start to research at any rate. $\endgroup$
    – David Z Mod
    Commented Sep 30, 2016 at 5:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .