# Flagging low quality answers: an experiment

Abstract: We are experimenting with expanding the scope of the very low quality (VLQ) flag. For two weeks, August 8-21, moderators will not decline any VLQ flags. We will allow these flags will sit in the queue as long as needed for the community to deal with them. We ask you, the community, to

• use VLQ flags on posts which are "harmfully incorrect" (see below for what that means), which would ordinarily not be a valid use of the VLQ flag
• when reviewing low-quality posts, choose to delete such "harmfully incorrect" posts

The handling of low-quality answers has been a point of contention and friction, on and off, for a fairly long time in this site. For low-quality questions, it happens that every so often someone sees a terrible answer, flags it as low quality, and finds out that it gets declined with the message

flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

which then generally leads to bruised egos and a lot of friction, and also a lot of people unhappy that we don't have a strong enough mechanism to handle the types of posts they wanted to flag. (Examples: 1, 2, 3, 4, 5, 6, 7, 8.)

This thread is an attempt to explain what pressures the system is under, the way the flagging software works under the hood, the way flags are currently handled, and the reasons for this. I will then make two proposals for how the system could be improved (one of which we can implement ourselves, and the second of which would include software tweaks to support that workflow), and I will propose an experiment to see how well it works.

I worked out several of these aspects after going through the bruised-ego, unnecessary-friction cycle in Why was this Very Low Quality flag declined?, and the responses and votes on that post and in a related chat make me hopeful that there's appetite for a change and a rough agreement that it's possible.

## Summary

In short, I claim that answers that are (1) flat-out incorrect, (2) so completely incorrect that no amount of editing can make it useful or constructive, and (3) so completely incorrect that it is actively harmful to the site and its standing, should be removed from the site. They should be flagged as Very Low Quality, and should be deleted via a consensus of six 2k+ users or three 20k+ users.

For the next two weeks, let's experiment with this dynamic and see how well it works.

## The pressures

This site gets its fair share of posts, both questions and answers, that are - shall we say - rather below par. This includes a lot of content that a lot of people would rather we didn't have on the site at all and, in particular, all sorts of content drawing from every depth of crackpottery.

• For example, we get a lot of questions that assume as a starting point some way-off-standard theory and then (some question or other), in such a way that the only possible responses would either be a complete rejection of the hypotheses, or fringe pseudophysics with no place in any serious physics forum.

• We also get a steady smattering of answers that draw on such off-base foundations, and which look something like "time dilation does not actually exist because Einstein was wrong, see my site at for more explanations".

The two types of post above are usually made worse by the fact that in a large fraction of cases the poster will adopt rather belligerent attitudes, refuse to back down on any claim or listen to anyone at all, and very often drag the discussion down to wildly unconstructive mud-slinging, which is unbelievably draining for everyone involved.

• In addition to the above, we also get a steady stream of posts that are just not very good. This ranges from incomprehensible keyboard bashing, to one-liners that only vaguely address the question, one-liner questions with nowhere near enough information to be answerable, 'answers' that are just comments on another post or which ask a separate question, and so on.

(This is, of course, an incomplete list.)

Luckily, of course, the site has a bunch of ways to deal with these things. For off-the-field questions we can close as non-mainstream and we can downvote them (which, once they reach score -4, hides from the active-questions list). For 'answers' that are really a separate question or a comment we can flag as Not An Answer, and it generally gets resolved rather well relatively quickly. For gibberish posts and one-liner nonsensicals we can flag as Very Low Quality and they eventually disappear.

For the very-much-wrong, out-in-the-out-field-of-the-next-field-over answers, we can downvote and... that's it. The way the system works now, there is often precious little that a user can do if she notices a post that is wrong-wrong-wrong. (Think something like this or like this.) As long as it is an attempt to provide some sort of answer, the Not An Answer flag doesn't work, and the Very Low Quality flag currently excludes answers which are generally intelligible.

Our user can then add a comment explaining what is wrong (with the associated risk of getting dragged into an unconstructive discussion), and she can add a single downvote, which often looks like a hopelessly small signal (like, for example, with answers that might get upvoted in a popular question because they 'sound right', or are catchy enough, to enough outsiders, even if any working physicist can see that they're completely incorrect). There isn't even a way to get other experts to notice the post and help fix it.

This, I contend, is a problem.

## What happens to flags once you flag them

The way the system is set up at the moment, there are two moderation queues where a post goes after it is flagged: the Low Quality Posts community moderation review queue, and the moderator queue where all flags end up when they require human intervention to get swiftly handled.

• For questions, the queue looks something like this:

It enables the reviewer to close the question if it is off-topic, edit it if it can be whipped into shape, skip it if the reviewer doesn't know what to do, and decide it looks OK and shouldn't really be in the queue to begin with.

• For answers, the queue looks something like this:

It enables the reviewer to edit the post, help get it off the queue by saying it looks OK, skip it and move elsewhere, and -- depending on the poster's reputation -- recommend deletion of the answer (<20k rep) or a vote to delete it (>20k rep). Deletion votes come with the option of posting a canned message that includes a link to the review page.

Posts with a negative score get deleted when they get six Recommend Deletion votes, or three Delete votes by >20k rep users. (Posts with positive score with that many delete votes get referred to moderators as a disputed deletion, and they probably do deserve that.)

The Low Quality Posts queue is also populated by automatic processes, which generally get marked by text along the lines of 'this answer was automatically flagged because of its length and content', and includes posts which are very short, have lots of links, are by new users, or probably some combination of those. Generally speaking, the algorithm that catches these is pretty well tuned, and it very often does flag up posts that do require some extra attention.

Posts stay in the queue for as long as they need to: until they're kicked off the queue by having more Looks OK votes than flags and (recommended) deletion votes, the queue produces an action on the post (closure, deletion, or edit), or a moderator intervenes.

In addition to this, after fifteen minutes of being raised, Very Low Quality flags also appear on the moderator queue, where moderators can see them and act on them. (However, this time can be changed and indeed it's a one-hour wait in several network sites.)

## How flags are currently handled

The current guidelines for how flags should be raised on this site are at Which flag do I use for an inappropriate post?; in addition, there is network-wide guidance on how to review over on the mother meta at What are the guidelines for reviewing?. In particular, the guidelines for raising a Very Low Quality flag currently read

### Very Low Quality

The "very low quality" flag is for answers which are complete nonsense, such as

• incoherent gibberish
• anything not parseable as English text
• answers posted in another language (although if you can edit in a translation, you're encouraged to do so instead of flagging)

Do not use this flag for

• answers based on non-mainstream physics
• followup questions

In particular, answers which are wrong or based on non-mainstream physics are explicitly excluded. The reason for this is that Very Low Quality flags very quickly make their way to the moderator review queue, and moderators are entirely the wrong people to be handling this sort of flag. Removing an answer because it's wrong, or because its premises are off-mainstream, is a judgement call based on technical aspects of the post.

This is precisely the sort of task that we use community moderation for - a consensus decision based on the judgement of multiple experts who have been on the site long enough to know what works and what doesn't - but if we ask moderators to handle these flags we're asking them to delete users' content based on technical grounds that they might get wrong. To their credit, the site moderators here understand that this is not their job and they repeatedly refuse to delete posts on those grounds.

In the end, this means that when people raise a Very Low Quality flag on a post that's dead-wrong it ends up being declined by a moderator because the system is asking them to do more than they should be doing. (Comments, of course, are another story.)

## What should be done with low-quality answers?

OK, so the story so far is how the system works, why it works this way, and what problems this causes. So, in an ideal world, how should this sort of post be handled?

One view which is held relatively widely (example) goes something like

Technically, we can't delete things which answer the question (even if they're wrong or short/low quality).

I contend that this is wrong: we can and should delete answers that are incorrect beyond saving.

(If you disagree, then as I argued previously: what compels us to host content that is wildly incorrect? Say, things like this or this. Users permit Stack Exchange to display their CC BY-SA-licensed content, and they ask the site to display it, but what compels us to host incorrect content?)

To be a bit more precise, I contend that if an answer is

1. flat-out incorrect,
2. so completely incorrect that no amount of editing can make it useful or constructive, and
3. so completely incorrect that it is actively harmful to the site and its standing,

then it should be deleted. In particular, it should be deleted via consensus-based community moderation, rather than through unilateral moderator action.

Stack Exchange sites shine because when you land on a question you're likely to find a good question with an easily accessible good answer written by someone who knows their stuff, instead of pages of back-and-forth trying to get the asker to provide enough information, endless discussions over technical details, or off-the-wall posts that are just plain wrong. In the words of ACuriousMind

the mission statement of StackExchange is "The best answers rise to the top", not "Even the worst answer is here for your convenience".

Or think about it this way: if [insert rising-star researcher in your field] were to stumble on that post on the site, would they think "huh, this is an interesting place" and decide to participate, or would they give it up as a wash because incorrect content gets such an easy pass?

## Fixing the VLQ flags, part 1: community process

Hopefully I still have everybody on board (and if you disagree with the above section, I do want to hear your views and I hope you're comfortable sharing them), in that there is a problem to be solved that the current mechanism isn't solving. Luckily enough, though, we do have the software tools to build a community process that can help clean up by putting deletion-worthy content in front of the people who should be deleting it (a consensus of experienced users) rather than the people who shouldn't (unilateral action by a moderator).

I propose, therefore, to change the community guidelines for flagging and reviewing:

• For flagging: use the Very Low Quality flag for answers which are complete nonsense, such as

• incoherent gibberish
• anything not parseable as English text
• answers posted in another language

as well as answers that are wrong, or based on non-mainstream physics, that you think are

1. flat-out incorrect,
2. so completely incorrect that no amount of editing can make it useful or constructive, and
3. so completely incorrect that it is actively harmful to the site and its standing.

Do not use this flag for

• followup questions
• For reviewing: vote to delete / recommend deletion on posts that in your judgement satisfy the above criterion. If you're not sure, click skip; if you think the post belongs on the site, click looks OK.

Very Low Quality flags should go to the Low Quality Posts review queue and remain there: that is, mods should not act to decline or dismiss VLQ flags raised over the technical quality of posts (whether they end up in the moderator queue or not), unless the review queue is obviously indecisive about it or is taking overly long to handle a post. I would fix the timescale for this at 2 to 3 days (but this should be easier to decide after the experiment below).

## Back-end changes

The above proposal has the enormous strength that it works completely within the existing Stack Exchange software, which is good because asking for developer time to work on something on this scale requires some pretty strong arguments that it will really help the site.

That said, to help it work better it would really help to keep Very Low Quality flags from being shown to moderators for a rather longer time than they do now - substantially longer than an hour, at the very least. This gives the community moderation space to do its work, and refers to the moderators the stuff that can't be handled that way.

This does have the disadvantage that gibberish and other clear candidates for deletion take a bit longer to clean up than they do in the current system. I think this is an acceptable trade-off, though.

Once the community moderation process is up and running, we should also consider other cosmetic changes (like, for example, a re-write of the text in the flagging prompt), but the way I see it these are secondary to the change in the process itself.

## The experiment

Finally, how do we get from here to there? I propose we experiment by switching to the system described above for two weeks and seeing how well it works. Thus:

• For the next two weeks (Monday 8 August to Sunday 21 August) moderators will still see Very Low Quality flags but they will only act on gibberish and non-parseable text, leaving any calls that depend on a judgement call on the technical merit of a post to the Low Quality Posts review queue.

• During that time, flag and review as per the above guidelines. If you see content that you think is so wrong it's harmful, flag it for removal. If you review content that you think is so wrong it's harmful, recommend its deletion.

After that, we'll have a look at as many statistics as we can get our hands on (some of which do require some data digging by an SE community manager) to see how well it worked.

### Coda

Finally, if you disagree with the above (including the case where you think we shouldn't even run the experiment) then I am interested in why you think so. However, there is one thing I do want to ask people not to fall into, and that is 'answer by Stack Exchange say-so'.

One example of this is here, which sort of goes "well, Stack Exchange put up these vague text messages, so therefore this is how we should run our own community moderation". Another example is this comment on that same question, coming from an outsider to this site and just referring to generic SE guidance. If you do want to just refer to generic guidelines, then I do ask that you provide a suitable argument for why they're also sufficient to help this site deal with the pressures it's under.

The reason for this is that the Stack Exchange software and the core guidelines for community moderation need to satisfy the needs of a huge array of 150+ sites with a wide variety of communities, attitudes, situations, and constraints.

The engine and guidelines need to handle the enormous volume of Stack Overflow, with review queues so long that people resorted to robotic reviewing so much that review audits needed to be introduced. They need to handle The Workplace, which every so often draws in a crowd of outsiders that's pretty sure it can solve everybody's problem. They need to handle the happy atmosphere of TeX.SE, which is just happy to throw upvotes pretty much anywhere. They need to work for movies and SciFi and RPG, which inhabit a very different corner of the internet than we do.

In particular, for many of those sites, the concept of an answer being 'harmfully incorrect' doesn't even make sense. What does a non-mainstream answer look like in Mathematics? Do they get them in Chemistry or Photography? Unless the guidelines you want to quote were built considering an explicit analogue of non-mainstream, harmfully-incorrect answers, then repeating them as-is is simply not helpful.

Final question: isn't it time I piped down? Why yes, yes it is. Over to you.

• I don't think you should do anything that would deter people from posting there thoughts on. I know things could become unmanigeable but this is a forum and you can get responses and questions about topics that might well be missed from a draconium approach to the rules and there could be exceptional insights to the proposed scientific truth such action would limit this this is not a game and there are should be no winners and losers. – 8Mad0Manc8 Aug 4 '16 at 20:59
• I definitely agree with this, with one exception: to remove the moderator problem (community consensus on incorrect answers) I think we should keep the VLQ flag and add a Wrong flag. Well, perhaps not named that, but that covers non-mainstream physics answers and so terribly wrong answers that every physicist (and anyone whose been through 6th grade) reads them and goes, "What...?!" This would include your opposite magnets do not attract example, and, say, this answer (deleted - can someone show that?). – heather Aug 4 '16 at 22:39
• @heather I'm not sure what you're proposing - where would the line be between the two? In any case, if we do ask for a new flag, it will take a very long time for SE to be able to dedicate resources to it; that's why this proposal is built explicitly within the realm of things that are controlled by the community and don't require back-end changes in functionality. (Changes in parameters, and cosmetic changes such as the explanations on flags, are easy to get. Changing the name of a flag, or getting a new one, would require a huge number of internal changes.) – Emilio Pisanty Aug 4 '16 at 22:45
• @EmilioPisanty, I was proposing a new flag for wrong answers aside from the VLQ flag, with the VLQ flag for unintelligibility. Of course, that's a good point about the time bit...okay, I changed my mind, your idea is the best. – heather Aug 4 '16 at 23:02
• @Jim No, that's not how the system currently works - flags based on technical way-outness are generally (rightly) declined at the moment, such as the declined flags of my previous post. – Emilio Pisanty Aug 5 '16 at 17:19
• Sure, but it says right in the meta post about non-mainstream posts that non-mainstream answers are to be flagged for deletion by mods. I do it all the time, they remove non-mainstream answers – Jim Aug 5 '16 at 18:34
• I think this post would be a lot easier to understand if it weren't so long. The most important parts are 1) The statement of the problem, and 2) The proposed community response. Unfortunately, the statement of the problem is buried beneath apocalyptic rhetoric which (I think) overstates the prevalence of bad posts without showing enough examples. Also, the proposed community response should be more easily accessed, i.e. delete most of the text before and after it. The "coda", for example, is un-necessary. – DanielSank Aug 7 '16 at 1:24
• @8Mad0Manc8 This is most certainly not a forum. – DanielSank Aug 9 '16 at 18:53
• It's too bad that bad questions can't also be dealt with. Some of the questions (not many) are actually stupid questions which, if the OP would have thought about the question for a bit, would realize that the question just does NOT make sense. – David White Aug 11 '16 at 17:31
• @DanielSank yes strictly speaking this is not a forum but it is a public place where the public and scientists are able to speak there collective thoughts and scientific "beliefs" or facts. I don't believe the public should be put off with there thoughts by someones professional dismissal of it. The public should be able and encouraged to participate and offer there personal curiosity without being belittled by the usual unobtainable part of society that scientist are. – 8Mad0Manc8 Aug 11 '16 at 23:45
• @EmilioPisanty One question :- did you write all of this alone ? how long did it take ? – A---B Aug 12 '16 at 17:29
• You seem to be saying there is a problem here but where is the evidence that such answers are "actively harmful" to the site? We are scientists here, and professionally demand to see evidence before we make decisions. – sammy gerbil Aug 17 '16 at 0:51
• just add 400 pages and you'll have a book – user46925 Aug 23 '16 at 9:07

I heartily support Emilio's proposal, but to explain why I support it will require a bit of rambling. So please bear with me and I'll try not to make this too painful.

The Physics Stack Exchange has evolved into an educational resource. We hoped it would be a research tool akin to the MathOverflow but for whatever reason that didn't happen. It appears that the culture amongst research physicists just doesn't lend itself to this goal. The Theoretical Physics SE got canned due to lack of traffic and the Physics Overflow is withering on the vine. So an educational tool is what we have ended up with.

But if the above sounds a bit downbeat let me emphasise just how important I think the Physics SE is in its role as an educational tool. Over the five years (good grief!) I've been a member I have explained basic principles of (for example) special relativity to students who have gone on to do PhDs, and it's hugely rewarding to think I might have contributed in some small way to the next generation of scientists. Most of us would, at least in principle, like to make the world a better place and this is one way you can really do it.

But, and it's a big but, if you're going to take on the role of teacher then you have the responsibility of ensuring you teach the truth and only the truth. If a student repeats misinformation that you've given them in an exam, and fails as a result, then you deserve to lie awake at night feeling guilty.

The point I'm working my way round to is that given our role as an educational resource it is important that our answers here are unimpeachable. This is not a place for trotting out your preconceptions and personal biases. Any answer posted here should not raise eyebrows if it appeared in a mainstream physics textbook. Well that's a grand goal, and of course one we are bound to fail at as long as answers are written by humans, but that's the goal to strive for.

And I think we are pretty good at the peer review process. Not perfect of course, but on the whole I think we do a decent job. We spot answers that are confused, misleading or plain wrong and indicate by comments and downvoting that the OP should not trust the answer (not if they want to pass their exams that is). The point of Emilio's proposal is to make this process still better, and that's why I support it.

• You support the proposal... which part? The OP gives at least two major ideas: the community response and the software changes. Which part do you like? – DanielSank Aug 7 '16 at 1:25
• "Any answer posted here should not raise eyebrows if it appeared in a mainstream physics textbook." What does it mean to "raise eyebrows"? – DanielSank Aug 7 '16 at 1:27
• I've moved the genesis of an off-topic comment discussion to chat so it can go on without filling up the comment section here. If it results in a suggested improvement to the post, feel free to come back and post that as a comment. – David Z Aug 8 '16 at 11:27
• a low voted post is not "repeating misinformation" unless people misunderstand basic SE dynamics. its serving as a signal of lukewarm support or what to avoid based on collective wisdom. think "unimpeachable posts/ textbook quality" is entirely not realistic for any internet site esp based on community/ volunteer contributions... its good as a goal in theory but its unachievable in practice – vzn Aug 9 '16 at 17:19

In short, I claim that answers that are (1) flat-out incorrect, (2) so completely incorrect that no amount of editing can make it useful or constructive, and (3) so completely incorrect that it is actively harmful to the site and its standing, should be removed from the site.

Who says they're flat out incorrect? You? What if your answers are flat out incorrect? What if you're asking for is censorship of correct answers so that your incorrect answers have no competition? This proposal would allow physics stack exchange to be hijacked and censored by a "cabal of quacks" working in concert.

So: no. If there are very-low-quality answers, you explain why they're very low quality. You hold them up to your own answers in comparison. You do not censor the competition with some vague insinuations that that's not mainstream. Not when answers like this get twelve downvotes despite the multiple references to Einstein. Not when what's actively harmful to the site are self-appointed "experts" who are sneering and scathing to newcomers and juniors, and whose arrogance is only exceeded by their ignorance.

• I thought you'd say something like this, but I should be clear that it's not meant to address answers like yours (where correctness requires a fair bit more time and expertise to determine conclusively); the relevant examples are linked in the question. Maybe you think those posts should be retained but I honestly don't think a policy of 'absolutely everything needs to stay' is right. That said, I won't argue further - I've said my piece, you've said yours, and if people agree with you then I'll be happy to accept that that is how the community wants to run the site. – Emilio Pisanty Aug 5 '16 at 17:27
• But you have been agitating for censorship in chat. And the moot point is that stack exchange has rules which make it difficult for dishonest posters working in concert to hijack a site. Your proposal sounds fine provided posters are honest. But power corrupts. Giving high-rep posters the power to delete answers is like letting "the expert in the field" referee all papers. (He's the guy who wrote the £150 textbook). And once those answers are deleted, who's to know whether they were right or wrong? I reiterate: no. – John Duffield Aug 6 '16 at 16:31
• I wanted to up-vote this answer, but then I got to the part about Einstein. References to Einstein do not a correct answer make. Appeal to authority is not good science. Here's an example: Einstein says red jellybeans are green. See, I cited Einstein, but what I said is wrong. The only way to know that is to go read all of Einstein's work to see that I misquoted. The reference isn't enough; you have to actually present coherent science yourself without leaning on appeals to authority. – DanielSank Aug 7 '16 at 1:30
• Your example sucks. I present coherent science. It isn't appealing to authority to refer to some peer-reviewed paper. Or to hard scientific evidence, which you airily dismiss on specious grounds too, with similar vague insinuations. Let's not forget, that according to you, users should _"beware that many users of this site find JD's answers to be "nonsense that sounds good". Letting people like you delete answers is like putting the fox in charge of the hen house. – John Duffield Aug 7 '16 at 7:43
• Mr. Duffield, I couldn't agree any more with your opinion than I already do. Is Physics StackExchange to be a place where only "Dr. Physics" types come to consult with each other? Or, is it to be a place that includes far more people? If it's the former, StackExchange won't endure for the long term. The censors just don't get that concept. It may be too lowly for them. – Inquisitive Aug 7 '16 at 17:22
• @Inquisitive : to be fair, physics stack exchange is meant to be for active researchers, academics, and students. IMHO the important issue here is that power corrupts. High-rep users will be able to delete an answer that's better than theirs. And once it's deleted, who's to know? – John Duffield Aug 8 '16 at 9:51
• I agree with the general direction this answer takes, but at the end there, you kinda went off a bit and it sounded like you were just being bitter about an instance of unfair voting against an answer of yours. The validity of your bitterness aside, you'd at least make me a happier person if this post didn't contain veiled emotions and sub-textual grudges. Of course, if I'm reading the situation wrong, then that's entirely my mistake. – Jim Aug 8 '16 at 13:09
• @Jim : the veiled emotion and sub-textual grudges aren't coming from me. They're coming from the OP. See this. He's employing sophistry here. See this comment in chat. That probably won't make you a happier person. Sorry about that. – John Duffield Aug 10 '16 at 12:19
• I share your views on this matter, John. Your example is a good one : your answer proved unpopular but it was not deleted. That is the important point. The rhetoric about the "harmfulness" of incorrect answers is over-hyped and verges on censorship. – sammy gerbil Aug 17 '16 at 0:41
• "And once it's deleted, who's to know?" It's somewhat of an aside, but do keep in mind that posts on Stack Exchange are only soft-deleted. With the exception of comments, any user with sufficient reputation sees deleted posts. (Deleted comments are visible only to moderators and Stack Exchange employees, and require specific action to be displayed, but they remain in the system.) At that point, you can either flag or vote for undeletion of a post. I see you don't have sufficient reputation anywhere on the network to see deleted posts, but believe me, they do show up once you have enough rep. – user Aug 19 '16 at 13:41
• @Michael Kjörling : noted. IMHO part of the issue there is that there's only 77 physics users with >10k rep, and not many are active. – John Duffield Aug 19 '16 at 15:19

I do not see where you have provided a definition of the term "harmfully incorrect." I very much doubt such answers exist. I think those who are advocating the view that some answers are so incorrect that they are positively "harmful" are not acting scientifically, since no evidence of such "harmfulness" has been provided. It seems to me that the "harm" is imaginary.

I disagree with John Rennie when he says that Answers on this site should be "unimpeachable". That is setting too high a standard which even those with the highest academic standing in this community may not consistently reach. It is an elitist view which would require most of the Answers on this site to be deleted, and would deter many users from participating by offering an Answer.

The current peer-review system of commenting on mildly incorrect Answers, and "down-voting to oblivion" those which are "harmfully incorrect," seems perfectly adequate to me.

Science tolerates alternative views, thrives in a climate on honest debate, and is stifled by the censorship of anything unorthodox. The fact that Answers are not mainstream or not wholly correct should not prevent them from being expressed. The advantages of maintaining freedom of speech outweigh any illusory "harmfulness".